Jul 07 06:58:40.322240 ip-10-0-138-97 systemd[1]: Starting Kubernetes Kubelet...
Jul 07 06:58:40.794704 ip-10-0-138-97 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 06:58:40.794704 ip-10-0-138-97 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 07 06:58:40.794704 ip-10-0-138-97 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 06:58:40.794704 ip-10-0-138-97 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 07 06:58:40.794704 ip-10-0-138-97 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 06:58:40.794704 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.773150    2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 07 06:58:40.794704 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777170    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 06:58:40.794704 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777181    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 06:58:40.794704 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777184    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 06:58:40.794704 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777187    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 06:58:40.794704 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777191    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 06:58:40.794704 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777194    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 06:58:40.794704 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777197    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 06:58:40.794704 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777200    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 06:58:40.840032 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777202    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 06:58:40.840032 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777205    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 06:58:40.840032 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777208    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 06:58:40.840032 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777211    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 06:58:40.840032 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777213    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 06:58:40.840032 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777216    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 06:58:40.840032 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777218    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 06:58:40.840032 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777221    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 06:58:40.840032 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777223    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 06:58:40.840032 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777226    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 06:58:40.840032 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777228    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 06:58:40.840032 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777231    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 06:58:40.840032 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777234    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 06:58:40.840032 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777236    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 06:58:40.840032 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777239    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 06:58:40.840032 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777241    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 06:58:40.840032 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777244    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 06:58:40.840032 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777247    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 06:58:40.840032 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777251    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 06:58:40.842038 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777254    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 06:58:40.842038 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777256    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 06:58:40.842038 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777259    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 06:58:40.842038 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777264    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 06:58:40.842038 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777267    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 06:58:40.842038 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777270    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 06:58:40.842038 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777274    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 06:58:40.842038 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777277    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 06:58:40.842038 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777280    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 06:58:40.842038 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777283    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 06:58:40.842038 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777285    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 06:58:40.842038 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777288    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 06:58:40.842038 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777290    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 06:58:40.842038 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777293    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 06:58:40.842038 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777296    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 06:58:40.842038 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777299    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 06:58:40.842038 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777302    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 06:58:40.842038 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777304    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 06:58:40.842038 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777306    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 06:58:40.850889 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777309    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 06:58:40.850889 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777311    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 06:58:40.850889 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777314    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 06:58:40.850889 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777316    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 06:58:40.850889 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777319    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 06:58:40.850889 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777322    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 06:58:40.850889 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777325    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 06:58:40.850889 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777327    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 06:58:40.850889 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777330    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 06:58:40.850889 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777333    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 06:58:40.850889 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777337    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 06:58:40.850889 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777341    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 06:58:40.850889 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777344    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 06:58:40.850889 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777347    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 06:58:40.850889 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777349    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 06:58:40.850889 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777352    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 06:58:40.850889 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777355    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 06:58:40.850889 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777357    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 06:58:40.850889 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777360    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 06:58:40.853083 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777362    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 06:58:40.853083 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777365    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 06:58:40.853083 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777368    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 06:58:40.853083 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777370    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 06:58:40.853083 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777373    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 06:58:40.853083 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777376    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 06:58:40.853083 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777379    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 06:58:40.853083 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777381    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 06:58:40.853083 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777384    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 06:58:40.853083 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777386    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 06:58:40.853083 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777389    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 06:58:40.853083 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777391    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 06:58:40.853083 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777394    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 06:58:40.853083 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777397    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 06:58:40.853083 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777400    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 06:58:40.853083 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777403    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 06:58:40.853083 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777405    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 06:58:40.853083 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777408    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 06:58:40.853083 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777411    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 06:58:40.853083 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777413    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 06:58:40.858864 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777416    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 06:58:40.858864 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777418    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 06:58:40.858864 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777421    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 06:58:40.858864 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777424    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 06:58:40.858864 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777427    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 06:58:40.858864 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777430    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 06:58:40.858864 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777432    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 06:58:40.858864 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777435    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 06:58:40.858864 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777437    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 06:58:40.858864 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777441    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 06:58:40.858864 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777444    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 06:58:40.858864 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777447    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 06:58:40.858864 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777450    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 06:58:40.858864 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.777453    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 06:58:40.858864 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778163    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 06:58:40.858864 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778170    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 06:58:40.858864 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778172    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 06:58:40.858864 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778175    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 06:58:40.858864 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778178    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 06:58:40.858864 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778181    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 06:58:40.900670 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778184    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 06:58:40.900670 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778186    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 06:58:40.900670 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778189    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 06:58:40.900670 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778192    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 06:58:40.900670 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778197    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 06:58:40.900670 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778200    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 06:58:40.900670 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778204    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 06:58:40.900670 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778206    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 06:58:40.900670 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778209    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 06:58:40.900670 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778212    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 06:58:40.900670 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778215    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 06:58:40.900670 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778217    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 06:58:40.900670 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778220    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 06:58:40.900670 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778222    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 06:58:40.900670 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778225    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 06:58:40.900670 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778227    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 06:58:40.900670 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778230    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 06:58:40.900670 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778232    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 06:58:40.900670 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778235    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 06:58:40.860080 ip-10-0-138-97 systemd[1]: Started Kubernetes Kubelet.
Jul 07 06:58:40.904495 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778237    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 06:58:40.904495 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778240    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 06:58:40.904495 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778243    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 06:58:40.904495 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778245    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 06:58:40.904495 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778247    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 06:58:40.904495 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778250    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 06:58:40.904495 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778252    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 06:58:40.904495 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778255    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 06:58:40.904495 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778257    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 06:58:40.904495 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778260    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 06:58:40.904495 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778262    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 06:58:40.904495 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778265    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 06:58:40.904495 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778267    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 06:58:40.904495 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778271    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 06:58:40.904495 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778273    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 06:58:40.904495 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778276    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 06:58:40.904495 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778278    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 06:58:40.904495 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778281    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 06:58:40.904495 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778284    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 06:58:40.904495 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778287    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 06:58:41.067312 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778290    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 06:58:41.067312 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778294    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 06:58:41.067312 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778297    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 06:58:41.067312 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778300    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 06:58:41.067312 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778303    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 06:58:41.067312 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778306    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 06:58:41.067312 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778309    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 06:58:41.067312 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778311    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 06:58:41.067312 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778314    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 06:58:41.067312 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778316    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 06:58:41.067312 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778319    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 06:58:41.067312 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778321    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 06:58:41.067312 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778324    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 06:58:41.067312 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778326    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 06:58:41.067312 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778329    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 06:58:41.067312 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778331    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 06:58:41.067312 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778334    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 06:58:41.067312 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778339    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 06:58:41.067312 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778342    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 06:58:41.069306 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778345    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 06:58:41.069306 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778347    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 06:58:41.069306 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778350    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 06:58:41.069306 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778352    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 06:58:41.069306 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778355    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 06:58:41.069306 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778359    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 06:58:41.069306 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778361    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 06:58:41.069306 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778364    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 06:58:41.069306 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778368    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 06:58:41.069306 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778373    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 06:58:41.069306 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778376    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 06:58:41.069306 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778379    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 06:58:41.069306 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778381    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 06:58:41.069306 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778384    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 06:58:41.069306 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778387    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 06:58:41.069306 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778389    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 06:58:41.069306 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778392    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 06:58:41.069306 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778394    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 06:58:41.069306 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778397    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 06:58:41.069306 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778399    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 06:58:41.070125 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778402    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 06:58:41.070125 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778404    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 06:58:41.070125 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778407    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 06:58:41.070125 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778409    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 06:58:41.070125 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778412    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 06:58:41.070125 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778414    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 06:58:41.070125 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778417    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 06:58:41.070125 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778419    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 06:58:41.070125 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778422    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 06:58:41.070125 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778424    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 06:58:41.070125 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778428    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 06:58:41.070125 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778430    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 06:58:41.070125 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778433    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 06:58:41.070125 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778435    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 06:58:41.070125 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778438    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 06:58:41.070125 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778808    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 06:58:41.070125 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778812    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 06:58:41.070125 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778815    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 06:58:41.070125 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778817    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 06:58:41.070125 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778822    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 06:58:41.070890 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778824    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 06:58:41.070890 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778827    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 06:58:41.070890 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778829    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 06:58:41.070890 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778832    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 06:58:41.070890 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778835    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 06:58:41.070890 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778837    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 06:58:41.070890 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778840    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 06:58:41.070890 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778843    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 06:58:41.070890 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778846    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 06:58:41.070890 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778849    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 06:58:41.070890 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778852    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 06:58:41.070890 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778854    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 06:58:41.070890 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778857    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 06:58:41.070890 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778860    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 06:58:41.070890 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778862    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 06:58:41.070890 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778866    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 06:58:41.070890 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778869    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 06:58:41.070890 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778871    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 06:58:41.070890 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778874    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 06:58:41.070890 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778877    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 06:58:41.073616 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778880    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 06:58:41.073616 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778882    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 06:58:41.073616 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778886    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 06:58:41.073616 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778889    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 06:58:41.073616 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778891    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 06:58:41.073616 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778894    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 06:58:41.073616 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778897    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 06:58:41.073616 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778899    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 06:58:41.073616 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778902    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 06:58:41.073616 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778905    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 06:58:41.073616 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778909    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 06:58:41.073616 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778912    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 06:58:41.073616 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778916    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 06:58:41.073616 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778919    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 06:58:41.073616 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778922    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 06:58:41.073616 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778925    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 06:58:41.073616 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778928    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 06:58:41.073616 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778930    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 06:58:41.073616 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778933    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 06:58:41.073616 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778936    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 06:58:41.074467 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778938    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 06:58:41.074467 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778941    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 06:58:41.074467 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778943    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 06:58:41.074467 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778969    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 06:58:41.074467 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778972    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 06:58:41.074467 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778974    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 06:58:41.074467 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778977    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 06:58:41.074467 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778980    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 06:58:41.074467 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778982    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 06:58:41.074467 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778985    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 06:58:41.074467 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778988    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 06:58:41.074467 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778991    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 06:58:41.074467 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778994    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 06:58:41.074467 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.778997    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 06:58:41.074467 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779000    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 06:58:41.074467 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779003    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 06:58:41.074467 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779006    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 06:58:41.074467 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779009    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 06:58:41.074467 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779011    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 06:58:41.074467 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779014    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 06:58:41.078634 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779016    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 06:58:41.078634 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779019    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 06:58:41.078634 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779021    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 06:58:41.078634 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779024    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 06:58:41.078634 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779026    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 06:58:41.078634 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779029    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 06:58:41.078634 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779032    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 06:58:41.078634 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779035    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 06:58:41.078634 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779038    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 06:58:41.078634 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779040    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 06:58:41.078634 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779043    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 06:58:41.078634 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779046    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 06:58:41.078634 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779049    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 06:58:41.078634 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779051    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 06:58:41.078634 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779054    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 06:58:41.078634 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779056    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 06:58:41.078634 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779059    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 06:58:41.078634 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779062    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 06:58:41.078634 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779065    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 06:58:41.078634 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779068    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 06:58:41.080341 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779091    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 06:58:41.080341 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779095    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 06:58:41.080341 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779098    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 06:58:41.080341 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779101    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 06:58:41.080341 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779104    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 06:58:41.080341 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779106    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 06:58:41.080341 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779110    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 06:58:41.080341 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779113    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 06:58:41.080341 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779118    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 06:58:41.080341 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779121    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 06:58:41.080341 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779124    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 06:58:41.080341 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779127    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 06:58:41.080341 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779130    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 06:58:41.080341 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779132    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 06:58:41.080341 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779207    2579 flags.go:64] FLAG: --address="0.0.0.0"
Jul 07 06:58:41.080341 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779218    2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 07 06:58:41.080341 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779224    2579 flags.go:64] FLAG: --anonymous-auth="true"
Jul 07 06:58:41.080341 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779230    2579 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 07 06:58:41.080341 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779235    2579 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 07 06:58:41.080341 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779238    2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 07 06:58:41.084928 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779242    2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 07 06:58:41.084928 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779246    2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 07 06:58:41.084928 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779249    2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 07 06:58:41.084928 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779252    2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 07 06:58:41.084928 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779255    2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 07 06:58:41.084928 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779258    2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 07 06:58:41.084928 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779261    2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 07 06:58:41.084928 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779264    2579 flags.go:64] FLAG: --cgroup-root=""
Jul 07 06:58:41.084928 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779267    2579 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 07 06:58:41.084928 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779271    2579 flags.go:64] FLAG: --client-ca-file=""
Jul 07 06:58:41.084928 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779274    2579 flags.go:64] FLAG: --cloud-provider="external"
Jul 07 06:58:41.084928 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779277    2579 flags.go:64] FLAG: --cluster-dns="[]"
Jul 07 06:58:41.084928 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779281    2579 flags.go:64] FLAG: --cluster-domain=""
Jul 07 06:58:41.084928 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779284    2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 07 06:58:41.084928 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779287    2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 07 06:58:41.084928 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779291    2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 07 06:58:41.084928 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779294    2579 flags.go:64] FLAG: --container-log-max-files="5"
Jul 07 06:58:41.084928 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779297    2579 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 07 06:58:41.084928 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779302    2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 07 06:58:41.084928 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779305    2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 07 06:58:41.084928 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779308    2579 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 07 06:58:41.084928 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779311    2579 flags.go:64] FLAG: --contention-profiling="false"
Jul 07 06:58:41.084928 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779314    2579 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 07 06:58:41.084928 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779317    2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 07 06:58:41.087041 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779320    2579 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 07 06:58:41.087041 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779323    2579 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 07 06:58:41.087041 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779327    2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 07 06:58:41.087041 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779330    2579 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 07 06:58:41.087041 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779333    2579 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 07 06:58:41.087041 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779337    2579 flags.go:64] FLAG: --enable-load-reader="false"
Jul 07 06:58:41.087041 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779340    2579 flags.go:64] FLAG: --enable-server="true"
Jul 07 06:58:41.087041 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779344    2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 07 06:58:41.087041 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779348    2579 flags.go:64] FLAG: --event-burst="100"
Jul 07 06:58:41.087041 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779351    2579 flags.go:64] FLAG: --event-qps="50"
Jul 07 06:58:41.087041 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779354    2579 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 07 06:58:41.087041 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779357    2579 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 07 06:58:41.087041 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779360    2579 flags.go:64] FLAG: --eviction-hard=""
Jul 07 06:58:41.087041 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779364    2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 07 06:58:41.087041 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779367    2579 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 07 06:58:41.087041 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779370    2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 07 06:58:41.087041 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779373    2579 flags.go:64] FLAG: --eviction-soft=""
Jul 07 06:58:41.087041 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779376    2579 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 07 06:58:41.087041 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779379    2579 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 07 06:58:41.087041 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779381    2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 07 06:58:41.087041 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779384    2579 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 07 06:58:41.087041 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779387    2579 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 07 06:58:41.087041 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779390    2579 flags.go:64] FLAG: --fail-swap-on="true"
Jul 07 06:58:41.087041 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779392    2579 flags.go:64] FLAG: --feature-gates=""
Jul 07 06:58:41.087041 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779396    2579 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 07 06:58:41.088696 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779399    2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 07 06:58:41.088696 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779403    2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 07 06:58:41.088696 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779407    2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 07 06:58:41.088696 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779410    2579 flags.go:64] FLAG: --healthz-port="10248"
Jul 07 06:58:41.088696 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779412    2579 flags.go:64] FLAG: --help="false"
Jul 07 06:58:41.088696 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779415    2579 flags.go:64] FLAG: --hostname-override="ip-10-0-138-97.ec2.internal"
Jul 07 06:58:41.088696 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779419    2579 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 07 06:58:41.088696 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779422    2579 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 07 06:58:41.088696 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779425    2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 07 06:58:41.088696 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779428    2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 07 06:58:41.088696 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779432    2579 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 07 06:58:41.088696 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779435    2579 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 07 06:58:41.088696 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779439    2579 flags.go:64] FLAG: --image-service-endpoint=""
Jul 07 06:58:41.088696 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779442    2579 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 07 06:58:41.088696 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779445    2579 flags.go:64] FLAG: --kube-api-burst="100"
Jul 07 06:58:41.088696 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779448    2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 07 06:58:41.088696 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779451    2579 flags.go:64] FLAG: --kube-api-qps="50"
Jul 07 06:58:41.088696 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779454    2579 flags.go:64] FLAG: --kube-reserved=""
Jul 07 06:58:41.088696 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779457    2579 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 07 06:58:41.088696 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779460    2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 07 06:58:41.088696 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779463    2579 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 07 06:58:41.088696 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779466    2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 07 06:58:41.088696 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779468    2579 flags.go:64] FLAG: --lock-file=""
Jul 07 06:58:41.088696 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779471    2579 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 07 06:58:41.089945 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779474    2579 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 07 06:58:41.089945 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779477    2579 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 07 06:58:41.089945 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779482    2579 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 07 06:58:41.089945 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779485    2579 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 07 06:58:41.089945 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779488    2579 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 07 06:58:41.089945 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779491    2579 flags.go:64] FLAG: --logging-format="text"
Jul 07 06:58:41.089945 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779494    2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 07 06:58:41.089945 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779497    2579 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 07 06:58:41.089945 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779500    2579 flags.go:64] FLAG: --manifest-url=""
Jul 07 06:58:41.089945 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779504    2579 flags.go:64] FLAG: --manifest-url-header=""
Jul 07 06:58:41.089945 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779508    2579 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 07 06:58:41.089945 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779511    2579 flags.go:64] FLAG: --max-open-files="1000000"
Jul 07 06:58:41.089945 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779515    2579 flags.go:64] FLAG: --max-pods="110"
Jul 07 06:58:41.089945 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779517    2579 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 07 06:58:41.089945 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779520    2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 07 06:58:41.089945 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779523    2579 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 07 06:58:41.089945 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779526    2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 07 06:58:41.089945 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779529    2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 07 06:58:41.089945 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779532    2579 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 07 06:58:41.089945 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779534    2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 07 06:58:41.089945 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779543    2579 flags.go:64] FLAG: --node-status-max-images="50"
Jul 07 06:58:41.089945 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779546    2579 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 07 06:58:41.089945 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779549    2579 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 07 06:58:41.089945 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779552    2579 flags.go:64] FLAG: --pod-cidr=""
Jul 07 06:58:41.090650 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779555    2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5"
Jul 07 06:58:41.090650 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779560    2579 flags.go:64] FLAG: --pod-manifest-path=""
Jul 07 06:58:41.090650 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779563    2579 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 07 06:58:41.090650 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779566    2579 flags.go:64] FLAG: --pods-per-core="0"
Jul 07 06:58:41.090650 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779569    2579 flags.go:64] FLAG: --port="10250"
Jul 07 06:58:41.090650 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779572    2579 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 07 06:58:41.090650 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779575    2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-08a952e42755059f8"
Jul 07 06:58:41.090650 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779578    2579 flags.go:64] FLAG: --qos-reserved=""
Jul 07 06:58:41.090650 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779581    2579 flags.go:64] FLAG: --read-only-port="10255"
Jul 07 06:58:41.090650 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779584    2579 flags.go:64] FLAG: --register-node="true"
Jul 07 06:58:41.090650 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779587    2579 flags.go:64] FLAG: --register-with-taints=""
Jul 07 06:58:41.090650 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779591    2579 flags.go:64] FLAG: --registry-burst="10"
Jul 07 06:58:41.090650 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779593    2579 flags.go:64] FLAG: --registry-qps="5"
Jul 07 06:58:41.090650 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779596    2579 flags.go:64] FLAG: --reserved-cpus=""
Jul 07 06:58:41.090650 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779599    2579 flags.go:64] FLAG: --reserved-memory=""
Jul 07 06:58:41.090650 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779602    2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 07 06:58:41.090650 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779605    2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 07 06:58:41.090650 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779608    2579 flags.go:64] FLAG: --rotate-certificates="false"
Jul 07 06:58:41.090650 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779612    2579 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 07 06:58:41.090650 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779615    2579 flags.go:64] FLAG: --runonce="false"
Jul 07 06:58:41.090650 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779618    2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 07 06:58:41.090650 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779621    2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 07 06:58:41.090650 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779623    2579 flags.go:64] FLAG: --seccomp-default="false"
Jul 07 06:58:41.090650 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779626    2579 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 07 06:58:41.090650 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779629    2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 07 06:58:41.091405 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779632    2579 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 07 06:58:41.091405 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779635    2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 07 06:58:41.091405 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779638    2579 flags.go:64] FLAG: --storage-driver-password="root"
Jul 07 06:58:41.091405 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779642    2579 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 07 06:58:41.091405 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779645    2579 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 07 06:58:41.091405 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779648    2579 flags.go:64] FLAG: --storage-driver-user="root"
Jul 07 06:58:41.091405 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779651    2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 07 06:58:41.091405 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779654    2579 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 07 06:58:41.091405 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779657    2579 flags.go:64] FLAG: --system-cgroups=""
Jul 07 06:58:41.091405 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779660    2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 07 06:58:41.091405 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779665    2579 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 07 06:58:41.091405 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779668    2579 flags.go:64] FLAG: --tls-cert-file=""
Jul 07 06:58:41.091405 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779670    2579 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 07 06:58:41.091405 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779680    2579 flags.go:64] FLAG: --tls-min-version=""
Jul 07 06:58:41.091405 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779683    2579 flags.go:64] FLAG: --tls-private-key-file=""
Jul 07 06:58:41.091405 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779686    2579 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 07 06:58:41.091405 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779689    2579 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 07 06:58:41.091405 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779692    2579 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 07 06:58:41.091405 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779695    2579 flags.go:64] FLAG: --v="2"
Jul 07 06:58:41.091405 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779699    2579 flags.go:64] FLAG: --version="false"
Jul 07 06:58:41.091405 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779703    2579 flags.go:64] FLAG: --vmodule=""
Jul 07 06:58:41.091405 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779707    2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 07 06:58:41.091405 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.779710    2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 07 06:58:41.091405 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779827    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 06:58:41.091405 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779831    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 06:58:41.092172 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779836    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 06:58:41.092172 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779839    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 06:58:41.092172 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779842    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 06:58:41.092172 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779845    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 06:58:41.092172 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779848    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 06:58:41.092172 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779851    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 06:58:41.092172 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779854    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 06:58:41.092172 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779856    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 06:58:41.092172 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779860    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 06:58:41.092172 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779864    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 06:58:41.092172 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779869    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 06:58:41.092172 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779872    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 06:58:41.092172 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779875    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 06:58:41.092172 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779877    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 06:58:41.092172 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779880    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 06:58:41.092172 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779882    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 06:58:41.092172 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779885    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 06:58:41.092172 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779887    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 06:58:41.092172 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779890    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 06:58:41.092931 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779893    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 06:58:41.092931 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779895    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 06:58:41.092931 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779898    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 06:58:41.092931 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779900    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 06:58:41.092931 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779918    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 06:58:41.092931 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779922    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 06:58:41.092931 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779925    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 06:58:41.092931 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779928    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 06:58:41.092931 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779931    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 06:58:41.092931 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779933    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 06:58:41.092931 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779936    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 06:58:41.092931 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779938    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 06:58:41.092931 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779941    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 06:58:41.092931 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779944    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 06:58:41.092931 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779947    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 06:58:41.092931 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779950    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 06:58:41.092931 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779970    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 06:58:41.092931 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779973    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 06:58:41.092931 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779976    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 06:58:41.092931 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779979    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 06:58:41.093718 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779982    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 06:58:41.093718 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779984    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 06:58:41.093718 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779987    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 06:58:41.093718 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779991    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 06:58:41.093718 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779993    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 06:58:41.093718 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779996    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 06:58:41.093718 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.779998    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 06:58:41.093718 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.780001    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 06:58:41.093718 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.780003    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 06:58:41.093718 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.780007    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 06:58:41.093718 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.780015    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 06:58:41.093718 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.780018    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 06:58:41.093718 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.780021    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 06:58:41.093718 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.780023    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 06:58:41.093718 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.780026    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 06:58:41.093718 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.780029    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 06:58:41.093718 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.780031    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 06:58:41.093718 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.780034    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 06:58:41.093718 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.780036    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 06:58:41.093718 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.780039    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 06:58:41.094471 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.780042    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 06:58:41.094471 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.780044    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 06:58:41.094471 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.780047    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 06:58:41.094471 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.780049    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 06:58:41.094471 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.780052    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 06:58:41.094471 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.780056    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 06:58:41.094471 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.780060    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 06:58:41.094471 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.780064    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 06:58:41.094471 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.780067    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 06:58:41.094471 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.780071    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 06:58:41.094471 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.780073    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 06:58:41.094471 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.780076    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 06:58:41.094471 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.780079    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 06:58:41.094471 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.780082    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 06:58:41.094471 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.780085    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 06:58:41.094471 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.780088    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 06:58:41.094471 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.780091    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 06:58:41.094471 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.780093    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 06:58:41.094471 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.780096    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 06:58:41.095259 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.780098    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 06:58:41.095259 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.780101    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 06:58:41.095259 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.780103    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 06:58:41.095259 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.780109    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 06:58:41.095259 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.780113    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 06:58:41.095259 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.780115    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 06:58:41.095259 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.780118    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 06:58:41.095259 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.780120    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 06:58:41.095259 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.780123    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 06:58:41.095259 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.780126    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 06:58:41.095259 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.780129    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 06:58:41.095259 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.780131    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 06:58:41.095259 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.780135    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 06:58:41.095259 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.780138    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 06:58:41.095259 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.780141    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 06:58:41.095259 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.780144    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 06:58:41.095259 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.780146    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 06:58:41.095259 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.780149    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 06:58:41.095259 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.780153    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 06:58:41.096020 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.780161    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 06:58:41.096020 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.787546    2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 07 06:58:41.096020 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.787564    2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 07 06:58:41.096020 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.787582    2579 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 07 06:58:41.096020 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.787588    2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 07 06:58:41.096020 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787650    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 06:58:41.096020 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787655    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 06:58:41.096020 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787658    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 06:58:41.096020 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787661    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 06:58:41.096020 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787664    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 06:58:41.096020 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787666    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 06:58:41.096020 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787669    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 06:58:41.096020 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787672    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 06:58:41.096020 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787674    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 06:58:41.096020 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787677    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 06:58:41.098345 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787679    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 06:58:41.098345 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787682    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 06:58:41.098345 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787685    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 06:58:41.098345 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787688    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 06:58:41.098345 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787690    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 06:58:41.098345 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787692    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 06:58:41.098345 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787695    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 06:58:41.098345 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787698    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 06:58:41.098345 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787703    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 06:58:41.098345 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787707    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 06:58:41.098345 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787711    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 06:58:41.098345 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787714    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 06:58:41.098345 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787717    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 06:58:41.098345 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787720    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 06:58:41.098345 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787723    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 06:58:41.098345 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787726    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 06:58:41.098345 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787729    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 06:58:41.098345 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787731    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 06:58:41.098345 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787734    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 06:58:41.099123 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787743    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 06:58:41.099123 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787745    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 06:58:41.099123 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787748    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 06:58:41.099123 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787752    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 06:58:41.099123 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787755    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 06:58:41.099123 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787757    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 06:58:41.099123 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787760    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 06:58:41.099123 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787762    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 06:58:41.099123 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787765    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 06:58:41.099123 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787768    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 06:58:41.099123 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787770    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 06:58:41.099123 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787773    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 06:58:41.099123 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787776    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 06:58:41.099123 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787778    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 06:58:41.099123 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787780    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 06:58:41.099123 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787784    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 06:58:41.099123 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787788    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 06:58:41.099123 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787791    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 06:58:41.099123 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787794    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 06:58:41.101413 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787796    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 06:58:41.101413 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787799    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 06:58:41.101413 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787803    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 06:58:41.101413 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787806    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 06:58:41.101413 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787809    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 06:58:41.101413 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787811    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 06:58:41.101413 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787814    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 06:58:41.101413 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787817    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 06:58:41.101413 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787821    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 06:58:41.101413 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787824    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 06:58:41.101413 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787826    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 06:58:41.101413 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787829    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 06:58:41.101413 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787832    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 06:58:41.101413 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787835    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 06:58:41.101413 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787843    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 06:58:41.101413 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787846    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 06:58:41.101413 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787850    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 06:58:41.101413 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787852    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 06:58:41.101413 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787855    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 06:58:41.102188 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787857    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 06:58:41.102188 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787860    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 06:58:41.102188 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787862    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 06:58:41.102188 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787865    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 06:58:41.102188 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787867    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 06:58:41.102188 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787870    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 06:58:41.102188 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787873    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 06:58:41.102188 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787875    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 06:58:41.102188 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787878    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 06:58:41.102188 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787880    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 06:58:41.102188 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787882    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 06:58:41.102188 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787885    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 06:58:41.102188 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787888    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 06:58:41.102188 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787890    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 06:58:41.102188 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787892    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 06:58:41.102188 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787897    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 06:58:41.102188 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787899    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 06:58:41.102188 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787901    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 06:58:41.102188 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787904    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 06:58:41.102188 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787907    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 06:58:41.103731 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787909    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 06:58:41.103731 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787912    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 06:58:41.103731 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787914    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 06:58:41.103731 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787917    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 06:58:41.103731 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787919    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 06:58:41.103731 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787921    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 06:58:41.103731 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787924    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 06:58:41.103731 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787927    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 06:58:41.103731 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787935    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 06:58:41.103731 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787939    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 06:58:41.103731 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787941    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 06:58:41.103731 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.787944    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 06:58:41.103731 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.787948    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 06:58:41.103731 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788117    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 06:58:41.103731 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788122    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 06:58:41.104480 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788126    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 06:58:41.104480 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788128    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 06:58:41.104480 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788131    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 06:58:41.104480 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788134    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 06:58:41.104480 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788137    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 06:58:41.104480 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788139    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 06:58:41.104480 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788142    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 06:58:41.104480 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788144    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 06:58:41.104480 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788147    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 06:58:41.104480 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788149    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 06:58:41.104480 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788152    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 06:58:41.104480 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788154    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 06:58:41.104480 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788157    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 06:58:41.104480 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788161    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 06:58:41.104480 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788163    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 06:58:41.104480 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788166    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 06:58:41.104480 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788168    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 06:58:41.104480 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788171    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 06:58:41.104480 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788173    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 06:58:41.104480 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788176    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 06:58:41.197413 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788178    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 06:58:41.197413 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788181    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 06:58:41.197413 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788183    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 06:58:41.197413 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788185    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 06:58:41.197413 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788188    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 06:58:41.197413 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788191    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 06:58:41.197413 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788201    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 06:58:41.197413 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788205    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 06:58:41.197413 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788208    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 06:58:41.197413 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788211    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 06:58:41.197413 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788213    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 06:58:41.197413 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788216    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 06:58:41.197413 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788218    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 06:58:41.197413 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788221    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 06:58:41.197413 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788223    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 06:58:41.197413 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788226    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 06:58:41.197413 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788228    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 06:58:41.197413 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788231    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 06:58:41.197413 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788233    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 06:58:41.198117 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788237    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 06:58:41.198117 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788240    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 06:58:41.198117 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788243    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 06:58:41.198117 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788246    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 06:58:41.198117 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788249    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 06:58:41.198117 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788252    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 06:58:41.198117 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788257    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 06:58:41.198117 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788260    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 06:58:41.198117 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788263    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 06:58:41.198117 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788265    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 06:58:41.198117 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788268    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 06:58:41.198117 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788270    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 06:58:41.198117 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788273    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 06:58:41.198117 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788275    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 06:58:41.198117 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788278    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 06:58:41.198117 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788280    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 06:58:41.198117 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788283    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 06:58:41.198117 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788285    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 06:58:41.198117 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788288    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 06:58:41.198858 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788292    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 06:58:41.198858 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788300    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 06:58:41.198858 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788303    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 06:58:41.198858 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788305    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 06:58:41.198858 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788308    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 06:58:41.198858 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788310    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 06:58:41.198858 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788313    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 06:58:41.198858 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788315    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 06:58:41.198858 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788318    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 06:58:41.198858 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788321    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 06:58:41.198858 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788324    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 06:58:41.198858 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788327    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 06:58:41.198858 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788330    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 06:58:41.198858 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788332    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 06:58:41.198858 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788335    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 06:58:41.198858 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788337    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 06:58:41.198858 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788340    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 06:58:41.198858 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788342    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 06:58:41.198858 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788345    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 06:58:41.198858 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788349    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 06:58:41.200758 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788351    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 06:58:41.200758 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788354    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 06:58:41.200758 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788356    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 06:58:41.200758 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788359    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 06:58:41.200758 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788362    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 06:58:41.200758 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788364    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 06:58:41.200758 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788367    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 06:58:41.200758 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788369    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 06:58:41.200758 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788372    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 06:58:41.200758 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788374    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 06:58:41.200758 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788377    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 06:58:41.200758 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788379    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 06:58:41.200758 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788382    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 06:58:41.200758 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788385    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 06:58:41.200758 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788393    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 06:58:41.200758 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788396    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 06:58:41.200758 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788399    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 06:58:41.200758 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788401    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 06:58:41.200758 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:40.788404    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 06:58:41.201518 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.788409    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 06:58:41.201518 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.788525    2579 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 07 06:58:41.201518 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.790770    2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 07 06:58:41.201518 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.792465    2579 server.go:1019] "Starting client certificate rotation"
Jul 07 06:58:41.201518 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.792573    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 07 06:58:41.201518 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.792610    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 07 06:58:41.201518 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.798289    2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 07 06:58:41.201518 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.806920    2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 07 06:58:41.201518 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.819771    2579 log.go:25] "Validated CRI v1 runtime API"
Jul 07 06:58:41.201518 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.822050    2579 log.go:25] "Validated CRI v1 image API"
Jul 07 06:58:41.201518 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.822809    2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 07 06:58:41.201518 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.827643    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 07 06:58:41.201917 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.829507    2579 fs.go:135] Filesystem UUIDs: map[12d47320-f048-4236-99e5-719cf9a6db0a:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 fe898f91-23b9-4407-98ed-8b80e40593ed:/dev/nvme0n1p3]
Jul 07 06:58:41.201917 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.829528    2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 07 06:58:41.203457 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.834675    2579 manager.go:216] Machine: {Timestamp:2026-07-07 06:58:40.833286098 +0000 UTC m=+0.377047570 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3087259 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2215d8d3cdb078419b6ed4483c14f8 SystemUUID:ec2215d8-d3cd-b078-419b-6ed4483c14f8 BootID:e9abbfd5-3347-4b44-892a-558301a43e42 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:b0:bb:a3:f5:13 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:b0:bb:a3:f5:13 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:02:4a:e6:20:c4:ed Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 07 06:58:41.203457 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.834780    2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 07 06:58:41.203457 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.834861    2579 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 07 06:58:41.203457 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.838215    2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 07 06:58:41.203457 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.838244    2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-97.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 07 06:58:41.203457 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.839785    2579 topology_manager.go:138] "Creating topology manager with none policy"
Jul 07 06:58:41.203457 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.839798    2579 container_manager_linux.go:309] "Creating device plugin manager"
Jul 07 06:58:41.203457 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.839808    2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 07 06:58:41.203457 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.840620    2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 07 06:58:41.203457 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.842647    2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 07 06:58:41.203457 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.846285    2579 state_mem.go:36] "Initialized new in-memory state store"
Jul 07 06:58:41.203457 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.846436    2579 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 07 06:58:41.203457 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.846473    2579 kubelet.go:486] "Attempting to sync node with API server"
Jul 07 06:58:41.203457 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.846492    2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 07 06:58:41.203457 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.846505    2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 07 06:58:41.203457 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.846518    2579 kubelet.go:398] "Adding apiserver pod source"
Jul 07 06:58:41.203457 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.846530    2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 07 06:58:41.203457 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.846930    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 07 06:58:41.204232 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.846950    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 07 06:58:41.204232 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.850848    2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 07 06:58:41.204232 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.853634    2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 07 06:58:41.204232 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.853679    2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 07 06:58:41.204232 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.855012    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 07 06:58:41.204232 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.855029    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 07 06:58:41.204232 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.855034    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 07 06:58:41.204232 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.855040    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 07 06:58:41.204232 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.855050    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 07 06:58:41.204232 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.855056    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 07 06:58:41.204232 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.855064    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 07 06:58:41.204232 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.855070    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 07 06:58:41.204232 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.855077    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 07 06:58:41.204232 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.855094    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 07 06:58:41.204232 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.855109    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 07 06:58:41.204232 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.855114    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 07 06:58:41.204232 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.855992    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 07 06:58:41.204232 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.856000    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 07 06:58:41.204232 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.859186    2579 server.go:1268] "Started kubelet"
Jul 07 06:58:41.204232 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.859263    2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 07 06:58:41.204232 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.859344    2579 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 07 06:58:41.204232 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.859265    2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 07 06:58:41.204232 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.864517    2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 07 06:58:41.204232 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.864688    2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-97.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 07 06:58:41.204232 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:40.865579    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-97.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 07 06:58:41.204232 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:40.865848    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 07 06:58:41.204232 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.867255    2579 server.go:310] "Adding debug handlers to kubelet server"
Jul 07 06:58:41.204232 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:40.870592    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-97.ec2.internal.18bfefb0fa91e9dc  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-97.ec2.internal,UID:ip-10-0-138-97.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-97.ec2.internal,},FirstTimestamp:2026-07-07 06:58:40.859154908 +0000 UTC m=+0.402916383,LastTimestamp:2026-07-07 06:58:40.859154908 +0000 UTC m=+0.402916383,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-97.ec2.internal,}"
Jul 07 06:58:41.204232 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:40.874088    2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 07 06:58:41.204232 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.875334    2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 07 06:58:41.204232 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.875361    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 07 06:58:41.204232 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:40.875413    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-97.ec2.internal\" not found"
Jul 07 06:58:41.204232 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.876076    2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 07 06:58:41.204232 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.876104    2579 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 07 06:58:41.204232 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.876126    2579 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 07 06:58:41.204232 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.876181    2579 factory.go:55] Registering systemd factory
Jul 07 06:58:41.204232 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.876196    2579 factory.go:223] Registration of the systemd container factory successfully
Jul 07 06:58:41.204232 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.876196    2579 reconstruct.go:97] "Volume reconstruction finished"
Jul 07 06:58:41.204232 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.876205    2579 reconciler.go:29] "Reconciler: start to sync state"
Jul 07 06:58:41.204232 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.876480    2579 factory.go:153] Registering CRI-O factory
Jul 07 06:58:41.204232 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.876492    2579 factory.go:223] Registration of the crio container factory successfully
Jul 07 06:58:41.206158 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.876544    2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 07 06:58:41.206158 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.876571    2579 factory.go:103] Registering Raw factory
Jul 07 06:58:41.206158 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.876584    2579 manager.go:1196] Started watching for new ooms in manager
Jul 07 06:58:41.206158 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.876942    2579 manager.go:349] Starting recovery of all containers
Jul 07 06:58:41.206158 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:40.883622    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 07 06:58:41.206158 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:40.885375    2579 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-97.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 07 06:58:41.206158 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.888933    2579 manager.go:354] Recovery completed
Jul 07 06:58:41.206158 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.893574    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 06:58:41.206158 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.898161    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-97.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 06:58:41.206158 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.898215    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-97.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 06:58:41.206158 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.898231    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-97.ec2.internal" event="NodeHasSufficientPID"
Jul 07 06:58:41.206158 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.904338    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-67qvb"
Jul 07 06:58:41.206158 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.904347    2579 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 07 06:58:41.206158 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.904364    2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 07 06:58:41.206158 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.904384    2579 state_mem.go:36] "Initialized new in-memory state store"
Jul 07 06:58:41.206158 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.904390    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 06:58:41.206158 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:40.904556    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-97.ec2.internal.18bfefb0fce5a108  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-97.ec2.internal,UID:ip-10-0-138-97.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-97.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-97.ec2.internal,},FirstTimestamp:2026-07-07 06:58:40.89819572 +0000 UTC m=+0.441957196,LastTimestamp:2026-07-07 06:58:40.89819572 +0000 UTC m=+0.441957196,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-97.ec2.internal,}"
Jul 07 06:58:41.206158 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.905519    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-97.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 06:58:41.206158 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.905575    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-97.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 06:58:41.206158 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.905607    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-97.ec2.internal" event="NodeHasSufficientPID"
Jul 07 06:58:41.206158 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.906216    2579 policy_none.go:49] "None policy: Start"
Jul 07 06:58:41.206158 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.906231    2579 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 07 06:58:41.206158 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.906240    2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 07 06:58:41.206158 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.907210    2579 policy_none.go:47] "Start"
Jul 07 06:58:41.206158 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.912147    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-67qvb"
Jul 07 06:58:41.206158 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:40.915803    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-97.ec2.internal.18bfefb0fce60985  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-97.ec2.internal,UID:ip-10-0-138-97.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-138-97.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-138-97.ec2.internal,},FirstTimestamp:2026-07-07 06:58:40.898222469 +0000 UTC m=+0.441983940,LastTimestamp:2026-07-07 06:58:40.898222469 +0000 UTC m=+0.441983940,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-97.ec2.internal,}"
Jul 07 06:58:41.206158 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.943832    2579 manager.go:337] "Starting Device Plugin manager"
Jul 07 06:58:41.206158 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:40.943865    2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 07 06:58:41.206158 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.943880    2579 server.go:87] "Starting device plugin registration server"
Jul 07 06:58:41.206158 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.944219    2579 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 07 06:58:41.206158 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.944231    2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 07 06:58:41.206158 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.944335    2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 07 06:58:41.207303 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.944423    2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 07 06:58:41.207303 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.944432    2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 07 06:58:41.207303 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:40.944861    2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 07 06:58:41.207303 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:40.944912    2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-97.ec2.internal\" not found"
Jul 07 06:58:41.207303 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.956642    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 07 06:58:41.207303 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.957886    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 07 06:58:41.207303 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.957917    2579 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 07 06:58:41.207303 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.957939    2579 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 07 06:58:41.207303 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:40.958026    2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 07 06:58:41.207303 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:40.960489    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 06:58:41.207303 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:41.044572    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 06:58:41.207303 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:41.046103    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-97.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 06:58:41.207303 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:41.046132    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-97.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 06:58:41.207303 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:41.046148    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-97.ec2.internal" event="NodeHasSufficientPID"
Jul 07 06:58:41.207303 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:41.046173    2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-97.ec2.internal"
Jul 07 06:58:41.207303 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:41.058148    2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-97.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-97.ec2.internal"]
Jul 07 06:58:41.207303 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:41.058325    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 06:58:41.207303 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:41.059031    2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-97.ec2.internal"
Jul 07 06:58:41.207303 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:41.059050    2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-97.ec2.internal\": node \"ip-10-0-138-97.ec2.internal\" not found"
Jul 07 06:58:41.207303 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:41.061486    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-97.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 06:58:41.207303 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:41.061507    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-97.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 06:58:41.207303 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:41.061517    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-97.ec2.internal" event="NodeHasSufficientPID"
Jul 07 06:58:41.207303 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:41.063677    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 06:58:41.207303 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:41.063794    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-97.ec2.internal"
Jul 07 06:58:41.207303 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:41.063825    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 06:58:41.207303 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:41.064442    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-97.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 06:58:41.207303 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:41.064472    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-97.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 06:58:41.207303 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:41.064489    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-97.ec2.internal" event="NodeHasSufficientPID"
Jul 07 06:58:41.207303 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:41.064486    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-97.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 06:58:41.207303 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:41.064520    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-97.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 06:58:41.207303 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:41.064533    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-97.ec2.internal" event="NodeHasSufficientPID"
Jul 07 06:58:41.207303 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:41.066259    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-97.ec2.internal"
Jul 07 06:58:41.207303 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:41.066278    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 06:58:41.207303 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:41.066945    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-97.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 06:58:41.207303 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:41.067006    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-97.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 06:58:41.207303 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:41.067022    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-97.ec2.internal" event="NodeHasSufficientPID"
Jul 07 06:58:41.207303 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:41.096664    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-97.ec2.internal\" not found"
Jul 07 06:58:41.207303 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:41.177087    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/407910b7b4a6fcb93fc8f7af29beb88b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-97.ec2.internal\" (UID: \"407910b7b4a6fcb93fc8f7af29beb88b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-97.ec2.internal"
Jul 07 06:58:41.209228 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:41.177114    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/407910b7b4a6fcb93fc8f7af29beb88b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-97.ec2.internal\" (UID: \"407910b7b4a6fcb93fc8f7af29beb88b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-97.ec2.internal"
Jul 07 06:58:41.209228 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:41.177136    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a566f4272b430543cc74511348672fa6-config\") pod \"kube-apiserver-proxy-ip-10-0-138-97.ec2.internal\" (UID: \"a566f4272b430543cc74511348672fa6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-97.ec2.internal"
Jul 07 06:58:41.209228 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:41.197365    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-97.ec2.internal\" not found"
Jul 07 06:58:41.313531 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:41.277243    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/407910b7b4a6fcb93fc8f7af29beb88b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-97.ec2.internal\" (UID: \"407910b7b4a6fcb93fc8f7af29beb88b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-97.ec2.internal"
Jul 07 06:58:41.313531 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:41.277292    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a566f4272b430543cc74511348672fa6-config\") pod \"kube-apiserver-proxy-ip-10-0-138-97.ec2.internal\" (UID: \"a566f4272b430543cc74511348672fa6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-97.ec2.internal"
Jul 07 06:58:41.313531 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:41.277338    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a566f4272b430543cc74511348672fa6-config\") pod \"kube-apiserver-proxy-ip-10-0-138-97.ec2.internal\" (UID: \"a566f4272b430543cc74511348672fa6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-97.ec2.internal"
Jul 07 06:58:41.313531 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:41.277344    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/407910b7b4a6fcb93fc8f7af29beb88b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-97.ec2.internal\" (UID: \"407910b7b4a6fcb93fc8f7af29beb88b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-97.ec2.internal"
Jul 07 06:58:41.313531 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:41.277389    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/407910b7b4a6fcb93fc8f7af29beb88b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-97.ec2.internal\" (UID: \"407910b7b4a6fcb93fc8f7af29beb88b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-97.ec2.internal"
Jul 07 06:58:41.313531 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:41.277434    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/407910b7b4a6fcb93fc8f7af29beb88b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-97.ec2.internal\" (UID: \"407910b7b4a6fcb93fc8f7af29beb88b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-97.ec2.internal"
Jul 07 06:58:41.313531 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:41.298368    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-97.ec2.internal\" not found"
Jul 07 06:58:41.427250 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:41.398981    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-97.ec2.internal\" not found"
Jul 07 06:58:41.539688 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:41.499412    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-97.ec2.internal\" not found"
Jul 07 06:58:41.658107 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:41.599865    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-97.ec2.internal\" not found"
Jul 07 06:58:41.773417 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:41.700407    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-97.ec2.internal\" not found"
Jul 07 06:58:41.882029 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:41.792008    2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 07 06:58:41.882029 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:41.792170    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 07 06:58:41.882029 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:41.801309    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-97.ec2.internal\" not found"
Jul 07 06:58:41.998266 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:41.902084    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-97.ec2.internal\" not found"
Jul 07 06:58:41.998266 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:41.914234    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-06 06:53:41 +0000 UTC" deadline="2028-01-08 01:00:04.181006445 +0000 UTC"
Jul 07 06:58:41.998266 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:41.914262    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13194h1m22.266747767s"
Jul 07 06:58:42.001084 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.001067    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 06:58:42.002257 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:42.002239    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-97.ec2.internal\" not found"
Jul 07 06:58:42.002832 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.002813    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-97.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 06:58:42.002892 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.002853    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-97.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 06:58:42.002892 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.002869    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-97.ec2.internal" event="NodeHasSufficientPID"
Jul 07 06:58:42.012851 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.012831    2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 06:58:42.012976 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.012962    2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 06:58:42.031913 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.031887    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-97.ec2.internal"
Jul 07 06:58:42.036695 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.036678    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-97.ec2.internal"
Jul 07 06:58:42.047872 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.047845    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 07 06:58:42.050896 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.050877    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 07 06:58:42.054147 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.054131    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-97.ec2.internal"
Jul 07 06:58:42.056636 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.056623    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-97.ec2.internal"
Jul 07 06:58:42.076059 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.076037    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-97.ec2.internal"
Jul 07 06:58:42.090364 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.090048    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 07 06:58:42.090364 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:42.090106    2579 kubelet.go:3250] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-97.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-97.ec2.internal"
Jul 07 06:58:42.090364 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.090122    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-97.ec2.internal"
Jul 07 06:58:42.101427 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.101408    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 07 06:58:42.101477 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:42.101455    2579 kubelet.go:3250] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-138-97.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-97.ec2.internal"
Jul 07 06:58:42.195488 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.195378    2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 06:58:42.789167 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:42.789120    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod407910b7b4a6fcb93fc8f7af29beb88b.slice/crio-d69f103bfc0dce7c336fa1c80c3e6bcd44deb8061ed573235a40137a2ca722b9 WatchSource:0}: Error finding container d69f103bfc0dce7c336fa1c80c3e6bcd44deb8061ed573235a40137a2ca722b9: Status 404 returned error can't find the container with id d69f103bfc0dce7c336fa1c80c3e6bcd44deb8061ed573235a40137a2ca722b9
Jul 07 06:58:42.789587 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:42.789564    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda566f4272b430543cc74511348672fa6.slice/crio-ede00cdcb64801043703f8c3b44d784ceed59be9c9ec2481446f899433a886f7 WatchSource:0}: Error finding container ede00cdcb64801043703f8c3b44d784ceed59be9c9ec2481446f899433a886f7: Status 404 returned error can't find the container with id ede00cdcb64801043703f8c3b44d784ceed59be9c9ec2481446f899433a886f7
Jul 07 06:58:42.795631 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.795610    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 06:58:42.848533 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.848484    2579 apiserver.go:52] "Watching apiserver"
Jul 07 06:58:42.857725 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.857697    2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 07 06:58:42.860257 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.860231    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-2tp8w","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-97.ec2.internal","openshift-multus/multus-additional-cni-plugins-j5jdf","openshift-multus/network-metrics-daemon-s9stz","openshift-network-operator/iptables-alerter-llqp4","kube-system/konnectivity-agent-z246s","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7r4r7","openshift-dns/node-resolver-r77h8","openshift-image-registry/node-ca-fzzb4","openshift-multus/multus-rw27d","openshift-network-diagnostics/network-check-target-nfjzx","openshift-ovn-kubernetes/ovnkube-node-wvm5w","kube-system/global-pull-secret-syncer-gnlks","kube-system/kube-apiserver-proxy-ip-10-0-138-97.ec2.internal"]
Jul 07 06:58:42.864237 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.864215    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2tp8w"
Jul 07 06:58:42.866498 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.866471    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rw27d"
Jul 07 06:58:42.867321 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.867301    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 07 06:58:42.867321 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.867306    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 07 06:58:42.867536 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.867462    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-h9tgt\""
Jul 07 06:58:42.868779 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.868765    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s9stz"
Jul 07 06:58:42.868845 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:42.868822    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s9stz" podUID="0dba90e8-7ad4-46cb-bc27-718b6f644f5d"
Jul 07 06:58:42.869369 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.869353    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 07 06:58:42.870141 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.870123    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 07 06:58:42.871258 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.870913    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-bv2tj\""
Jul 07 06:58:42.871258 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.870913    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 07 06:58:42.871258 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.871008    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 07 06:58:42.871458 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.871436    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-j5jdf"
Jul 07 06:58:42.873844 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.873824    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-llqp4"
Jul 07 06:58:42.874721 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.874705    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 07 06:58:42.874792 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.874760    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-g9xcn\""
Jul 07 06:58:42.874873 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.874856    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 07 06:58:42.876308 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.876292    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-z246s"
Jul 07 06:58:42.876606 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.876589    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 07 06:58:42.876758 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.876740    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 07 06:58:42.876834 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.876788    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 07 06:58:42.876834 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.876814    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-tgtgq\""
Jul 07 06:58:42.876939 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.876878    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 07 06:58:42.879350 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.879203    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7r4r7"
Jul 07 06:58:42.880381 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.880357    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-74p7f\""
Jul 07 06:58:42.880567 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.880550    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 07 06:58:42.880718 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.880632    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 07 06:58:42.882288 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.882262    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-r77h8"
Jul 07 06:58:42.882288 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.882279    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 07 06:58:42.882950 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.882919    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 07 06:58:42.883022 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.883004    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 07 06:58:42.883324 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.883302    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-ttmv9\""
Jul 07 06:58:42.885523 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.885502    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-fzzb4"
Jul 07 06:58:42.885886 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.885842    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5e1fa63c-3fd7-4883-9aad-5770f7c3b08f-system-cni-dir\") pod \"multus-rw27d\" (UID: \"5e1fa63c-3fd7-4883-9aad-5770f7c3b08f\") " pod="openshift-multus/multus-rw27d"
Jul 07 06:58:42.885886 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.885874    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5e1fa63c-3fd7-4883-9aad-5770f7c3b08f-os-release\") pod \"multus-rw27d\" (UID: \"5e1fa63c-3fd7-4883-9aad-5770f7c3b08f\") " pod="openshift-multus/multus-rw27d"
Jul 07 06:58:42.886051 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.885895    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5e1fa63c-3fd7-4883-9aad-5770f7c3b08f-host-var-lib-cni-bin\") pod \"multus-rw27d\" (UID: \"5e1fa63c-3fd7-4883-9aad-5770f7c3b08f\") " pod="openshift-multus/multus-rw27d"
Jul 07 06:58:42.886051 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.885920    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5e1fa63c-3fd7-4883-9aad-5770f7c3b08f-hostroot\") pod \"multus-rw27d\" (UID: \"5e1fa63c-3fd7-4883-9aad-5770f7c3b08f\") " pod="openshift-multus/multus-rw27d"
Jul 07 06:58:42.886051 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.885955    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 07 06:58:42.886051 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.885971    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/76cc2585-6fae-464f-afcf-f629ed055e11-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-j5jdf\" (UID: \"76cc2585-6fae-464f-afcf-f629ed055e11\") " pod="openshift-multus/multus-additional-cni-plugins-j5jdf"
Jul 07 06:58:42.886051 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.885996    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2gfzc\""
Jul 07 06:58:42.886051 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.886019    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhn8b\" (UniqueName: \"kubernetes.io/projected/76cc2585-6fae-464f-afcf-f629ed055e11-kube-api-access-zhn8b\") pod \"multus-additional-cni-plugins-j5jdf\" (UID: \"76cc2585-6fae-464f-afcf-f629ed055e11\") " pod="openshift-multus/multus-additional-cni-plugins-j5jdf"
Jul 07 06:58:42.886051 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.886054    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/99da3e49-fb00-4c1a-b0ba-cd1be152fb0c-iptables-alerter-script\") pod \"iptables-alerter-llqp4\" (UID: \"99da3e49-fb00-4c1a-b0ba-cd1be152fb0c\") " pod="openshift-network-operator/iptables-alerter-llqp4"
Jul 07 06:58:42.886360 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.886083    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/858752de-a754-4997-9bda-cd13f51366d4-etc-systemd\") pod \"tuned-2tp8w\" (UID: \"858752de-a754-4997-9bda-cd13f51366d4\") " pod="openshift-cluster-node-tuning-operator/tuned-2tp8w"
Jul 07 06:58:42.886360 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.886109    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/858752de-a754-4997-9bda-cd13f51366d4-run\") pod \"tuned-2tp8w\" (UID: \"858752de-a754-4997-9bda-cd13f51366d4\") " pod="openshift-cluster-node-tuning-operator/tuned-2tp8w"
Jul 07 06:58:42.886360 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.886134    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 07 06:58:42.886360 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.886133    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lwwf\" (UniqueName: \"kubernetes.io/projected/858752de-a754-4997-9bda-cd13f51366d4-kube-api-access-5lwwf\") pod \"tuned-2tp8w\" (UID: \"858752de-a754-4997-9bda-cd13f51366d4\") " pod="openshift-cluster-node-tuning-operator/tuned-2tp8w"
Jul 07 06:58:42.886360 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.886176    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/323baf0a-36cc-4f8c-b00b-ccc911176299-etc-selinux\") pod \"aws-ebs-csi-driver-node-7r4r7\" (UID: \"323baf0a-36cc-4f8c-b00b-ccc911176299\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7r4r7"
Jul 07 06:58:42.886360 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.886214    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/76cc2585-6fae-464f-afcf-f629ed055e11-cni-binary-copy\") pod \"multus-additional-cni-plugins-j5jdf\" (UID: \"76cc2585-6fae-464f-afcf-f629ed055e11\") " pod="openshift-multus/multus-additional-cni-plugins-j5jdf"
Jul 07 06:58:42.886360 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.886238    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/76cc2585-6fae-464f-afcf-f629ed055e11-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-j5jdf\" (UID: \"76cc2585-6fae-464f-afcf-f629ed055e11\") " pod="openshift-multus/multus-additional-cni-plugins-j5jdf"
Jul 07 06:58:42.886360 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.886263    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/858752de-a754-4997-9bda-cd13f51366d4-var-lib-kubelet\") pod \"tuned-2tp8w\" (UID: \"858752de-a754-4997-9bda-cd13f51366d4\") " pod="openshift-cluster-node-tuning-operator/tuned-2tp8w"
Jul 07 06:58:42.886360 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.886286    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e1fa63c-3fd7-4883-9aad-5770f7c3b08f-host-var-lib-kubelet\") pod \"multus-rw27d\" (UID: \"5e1fa63c-3fd7-4883-9aad-5770f7c3b08f\") " pod="openshift-multus/multus-rw27d"
Jul 07 06:58:42.886360 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.886304    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/323baf0a-36cc-4f8c-b00b-ccc911176299-sys-fs\") pod \"aws-ebs-csi-driver-node-7r4r7\" (UID: \"323baf0a-36cc-4f8c-b00b-ccc911176299\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7r4r7"
Jul 07 06:58:42.886360 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.886318    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/76cc2585-6fae-464f-afcf-f629ed055e11-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j5jdf\" (UID: \"76cc2585-6fae-464f-afcf-f629ed055e11\") " pod="openshift-multus/multus-additional-cni-plugins-j5jdf"
Jul 07 06:58:42.886360 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.886352    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s28r2\" (UniqueName: \"kubernetes.io/projected/99da3e49-fb00-4c1a-b0ba-cd1be152fb0c-kube-api-access-s28r2\") pod \"iptables-alerter-llqp4\" (UID: \"99da3e49-fb00-4c1a-b0ba-cd1be152fb0c\") " pod="openshift-network-operator/iptables-alerter-llqp4"
Jul 07 06:58:42.886856 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.886377    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/338d4939-ce17-426b-a06b-bb8f40ebe2a1-agent-certs\") pod \"konnectivity-agent-z246s\" (UID: \"338d4939-ce17-426b-a06b-bb8f40ebe2a1\") " pod="kube-system/konnectivity-agent-z246s"
Jul 07 06:58:42.886856 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.886394    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/858752de-a754-4997-9bda-cd13f51366d4-lib-modules\") pod \"tuned-2tp8w\" (UID: \"858752de-a754-4997-9bda-cd13f51366d4\") " pod="openshift-cluster-node-tuning-operator/tuned-2tp8w"
Jul 07 06:58:42.886856 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.886415    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5e1fa63c-3fd7-4883-9aad-5770f7c3b08f-etc-kubernetes\") pod \"multus-rw27d\" (UID: \"5e1fa63c-3fd7-4883-9aad-5770f7c3b08f\") " pod="openshift-multus/multus-rw27d"
Jul 07 06:58:42.886856 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.886443    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/323baf0a-36cc-4f8c-b00b-ccc911176299-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7r4r7\" (UID: \"323baf0a-36cc-4f8c-b00b-ccc911176299\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7r4r7"
Jul 07 06:58:42.886856 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.886464    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/76cc2585-6fae-464f-afcf-f629ed055e11-system-cni-dir\") pod \"multus-additional-cni-plugins-j5jdf\" (UID: \"76cc2585-6fae-464f-afcf-f629ed055e11\") " pod="openshift-multus/multus-additional-cni-plugins-j5jdf"
Jul 07 06:58:42.886856 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.886484    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/99da3e49-fb00-4c1a-b0ba-cd1be152fb0c-host-slash\") pod \"iptables-alerter-llqp4\" (UID: \"99da3e49-fb00-4c1a-b0ba-cd1be152fb0c\") " pod="openshift-network-operator/iptables-alerter-llqp4"
Jul 07 06:58:42.886856 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.886504    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/338d4939-ce17-426b-a06b-bb8f40ebe2a1-konnectivity-ca\") pod \"konnectivity-agent-z246s\" (UID: \"338d4939-ce17-426b-a06b-bb8f40ebe2a1\") " pod="kube-system/konnectivity-agent-z246s"
Jul 07 06:58:42.886856 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.886523    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/858752de-a754-4997-9bda-cd13f51366d4-etc-tuned\") pod \"tuned-2tp8w\" (UID: \"858752de-a754-4997-9bda-cd13f51366d4\") " pod="openshift-cluster-node-tuning-operator/tuned-2tp8w"
Jul 07 06:58:42.886856 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.886539    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5e1fa63c-3fd7-4883-9aad-5770f7c3b08f-multus-socket-dir-parent\") pod \"multus-rw27d\" (UID: \"5e1fa63c-3fd7-4883-9aad-5770f7c3b08f\") " pod="openshift-multus/multus-rw27d"
Jul 07 06:58:42.886856 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.886583    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5e1fa63c-3fd7-4883-9aad-5770f7c3b08f-multus-daemon-config\") pod \"multus-rw27d\" (UID: \"5e1fa63c-3fd7-4883-9aad-5770f7c3b08f\") " pod="openshift-multus/multus-rw27d"
Jul 07 06:58:42.886856 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.886632    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/323baf0a-36cc-4f8c-b00b-ccc911176299-registration-dir\") pod \"aws-ebs-csi-driver-node-7r4r7\" (UID: \"323baf0a-36cc-4f8c-b00b-ccc911176299\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7r4r7"
Jul 07 06:58:42.886856 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.886671    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/76cc2585-6fae-464f-afcf-f629ed055e11-cnibin\") pod \"multus-additional-cni-plugins-j5jdf\" (UID: \"76cc2585-6fae-464f-afcf-f629ed055e11\") " pod="openshift-multus/multus-additional-cni-plugins-j5jdf"
Jul 07 06:58:42.886856 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.886715    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/858752de-a754-4997-9bda-cd13f51366d4-etc-modprobe-d\") pod \"tuned-2tp8w\" (UID: \"858752de-a754-4997-9bda-cd13f51366d4\") " pod="openshift-cluster-node-tuning-operator/tuned-2tp8w"
Jul 07 06:58:42.886856 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.886739    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/858752de-a754-4997-9bda-cd13f51366d4-sys\") pod \"tuned-2tp8w\" (UID: \"858752de-a754-4997-9bda-cd13f51366d4\") " pod="openshift-cluster-node-tuning-operator/tuned-2tp8w"
Jul 07 06:58:42.886856 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.886763    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5e1fa63c-3fd7-4883-9aad-5770f7c3b08f-cni-binary-copy\") pod \"multus-rw27d\" (UID: \"5e1fa63c-3fd7-4883-9aad-5770f7c3b08f\") " pod="openshift-multus/multus-rw27d"
Jul 07 06:58:42.886856 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.886820    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5e1fa63c-3fd7-4883-9aad-5770f7c3b08f-host-run-k8s-cni-cncf-io\") pod \"multus-rw27d\" (UID: \"5e1fa63c-3fd7-4883-9aad-5770f7c3b08f\") " pod="openshift-multus/multus-rw27d"
Jul 07 06:58:42.887416 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.886850    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5e1fa63c-3fd7-4883-9aad-5770f7c3b08f-multus-conf-dir\") pod \"multus-rw27d\" (UID: \"5e1fa63c-3fd7-4883-9aad-5770f7c3b08f\") " pod="openshift-multus/multus-rw27d"
Jul 07 06:58:42.887416 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.886879    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0dba90e8-7ad4-46cb-bc27-718b6f644f5d-metrics-certs\") pod \"network-metrics-daemon-s9stz\" (UID: \"0dba90e8-7ad4-46cb-bc27-718b6f644f5d\") " pod="openshift-multus/network-metrics-daemon-s9stz"
Jul 07 06:58:42.887416 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.886908    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/323baf0a-36cc-4f8c-b00b-ccc911176299-socket-dir\") pod \"aws-ebs-csi-driver-node-7r4r7\" (UID: \"323baf0a-36cc-4f8c-b00b-ccc911176299\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7r4r7"
Jul 07 06:58:42.887416 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.886943    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/858752de-a754-4997-9bda-cd13f51366d4-etc-kubernetes\") pod \"tuned-2tp8w\" (UID: \"858752de-a754-4997-9bda-cd13f51366d4\") " pod="openshift-cluster-node-tuning-operator/tuned-2tp8w"
Jul 07 06:58:42.887416 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.886966    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/858752de-a754-4997-9bda-cd13f51366d4-etc-sysctl-d\") pod \"tuned-2tp8w\" (UID: \"858752de-a754-4997-9bda-cd13f51366d4\") " pod="openshift-cluster-node-tuning-operator/tuned-2tp8w"
Jul 07 06:58:42.887416 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.887005    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/858752de-a754-4997-9bda-cd13f51366d4-tmp\") pod \"tuned-2tp8w\" (UID: \"858752de-a754-4997-9bda-cd13f51366d4\") " pod="openshift-cluster-node-tuning-operator/tuned-2tp8w"
Jul 07 06:58:42.887416 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.887021    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5e1fa63c-3fd7-4883-9aad-5770f7c3b08f-multus-cni-dir\") pod \"multus-rw27d\" (UID: \"5e1fa63c-3fd7-4883-9aad-5770f7c3b08f\") " pod="openshift-multus/multus-rw27d"
Jul 07 06:58:42.887416 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.887038    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5e1fa63c-3fd7-4883-9aad-5770f7c3b08f-cnibin\") pod \"multus-rw27d\" (UID: \"5e1fa63c-3fd7-4883-9aad-5770f7c3b08f\") " pod="openshift-multus/multus-rw27d"
Jul 07 06:58:42.887416 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.887060    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/323baf0a-36cc-4f8c-b00b-ccc911176299-device-dir\") pod \"aws-ebs-csi-driver-node-7r4r7\" (UID: \"323baf0a-36cc-4f8c-b00b-ccc911176299\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7r4r7"
Jul 07 06:58:42.887416 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.887079    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5tlq\" (UniqueName: \"kubernetes.io/projected/323baf0a-36cc-4f8c-b00b-ccc911176299-kube-api-access-q5tlq\") pod \"aws-ebs-csi-driver-node-7r4r7\" (UID: \"323baf0a-36cc-4f8c-b00b-ccc911176299\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7r4r7"
Jul 07 06:58:42.887416 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.887094    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/76cc2585-6fae-464f-afcf-f629ed055e11-os-release\") pod \"multus-additional-cni-plugins-j5jdf\" (UID: \"76cc2585-6fae-464f-afcf-f629ed055e11\") " pod="openshift-multus/multus-additional-cni-plugins-j5jdf"
Jul 07 06:58:42.887416 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.887111    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/858752de-a754-4997-9bda-cd13f51366d4-etc-sysconfig\") pod \"tuned-2tp8w\" (UID: \"858752de-a754-4997-9bda-cd13f51366d4\") " pod="openshift-cluster-node-tuning-operator/tuned-2tp8w"
Jul 07 06:58:42.887416 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.887141    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/858752de-a754-4997-9bda-cd13f51366d4-host\") pod \"tuned-2tp8w\" (UID: \"858752de-a754-4997-9bda-cd13f51366d4\") " pod="openshift-cluster-node-tuning-operator/tuned-2tp8w"
Jul 07 06:58:42.887416 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.887170    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5e1fa63c-3fd7-4883-9aad-5770f7c3b08f-host-run-netns\") pod \"multus-rw27d\" (UID: \"5e1fa63c-3fd7-4883-9aad-5770f7c3b08f\") " pod="openshift-multus/multus-rw27d"
Jul 07 06:58:42.887416 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.887221    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5e1fa63c-3fd7-4883-9aad-5770f7c3b08f-host-var-lib-cni-multus\") pod \"multus-rw27d\" (UID: \"5e1fa63c-3fd7-4883-9aad-5770f7c3b08f\") " pod="openshift-multus/multus-rw27d"
Jul 07 06:58:42.887416 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.887257    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5e1fa63c-3fd7-4883-9aad-5770f7c3b08f-host-run-multus-certs\") pod \"multus-rw27d\" (UID: \"5e1fa63c-3fd7-4883-9aad-5770f7c3b08f\") " pod="openshift-multus/multus-rw27d"
Jul 07 06:58:42.887416 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.887306    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5bw9\" (UniqueName: \"kubernetes.io/projected/5e1fa63c-3fd7-4883-9aad-5770f7c3b08f-kube-api-access-r5bw9\") pod \"multus-rw27d\" (UID: \"5e1fa63c-3fd7-4883-9aad-5770f7c3b08f\") " pod="openshift-multus/multus-rw27d"
Jul 07 06:58:42.887914 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.887332    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2bss\" (UniqueName: \"kubernetes.io/projected/0dba90e8-7ad4-46cb-bc27-718b6f644f5d-kube-api-access-b2bss\") pod \"network-metrics-daemon-s9stz\" (UID: \"0dba90e8-7ad4-46cb-bc27-718b6f644f5d\") " pod="openshift-multus/network-metrics-daemon-s9stz"
Jul 07 06:58:42.887914 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.887355    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/858752de-a754-4997-9bda-cd13f51366d4-etc-sysctl-conf\") pod \"tuned-2tp8w\" (UID: \"858752de-a754-4997-9bda-cd13f51366d4\") " pod="openshift-cluster-node-tuning-operator/tuned-2tp8w"
Jul 07 06:58:42.887982 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.887940    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nfjzx"
Jul 07 06:58:42.888057 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:42.888035    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nfjzx" podUID="4e917e03-b190-40ac-8c93-cd4d794fa127"
Jul 07 06:58:42.888171 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.888155    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 07 06:58:42.888362 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.888343    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 07 06:58:42.888626 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.888608    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-h92m2\""
Jul 07 06:58:42.888711 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.888674    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 07 06:58:42.890664 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.890650    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wvm5w"
Jul 07 06:58:42.891237 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.891220    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 07 06:58:42.893291 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.893273    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 07 06:58:42.893403 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.893378    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gnlks"
Jul 07 06:58:42.893507 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:42.893447    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gnlks" podUID="575f1248-fecc-48d9-9c6a-db334596c8b6"
Jul 07 06:58:42.894047 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.893994    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 07 06:58:42.894047 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.894039    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 07 06:58:42.894237 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.894068    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-mkjsf\""
Jul 07 06:58:42.894237 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.894040    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 07 06:58:42.894237 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.894151    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 07 06:58:42.896275 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.896229    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 07 06:58:42.927331 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.927314    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-r2dvm"
Jul 07 06:58:42.939666 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.939639    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-r2dvm"
Jul 07 06:58:42.962533 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.962491    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-97.ec2.internal" event={"ID":"407910b7b4a6fcb93fc8f7af29beb88b","Type":"ContainerStarted","Data":"d69f103bfc0dce7c336fa1c80c3e6bcd44deb8061ed573235a40137a2ca722b9"}
Jul 07 06:58:42.963667 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.963646    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-97.ec2.internal" event={"ID":"a566f4272b430543cc74511348672fa6","Type":"ContainerStarted","Data":"ede00cdcb64801043703f8c3b44d784ceed59be9c9ec2481446f899433a886f7"}
Jul 07 06:58:42.976871 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.976847    2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 07 06:58:42.988080 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.988054    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/575f1248-fecc-48d9-9c6a-db334596c8b6-original-pull-secret\") pod \"global-pull-secret-syncer-gnlks\" (UID: \"575f1248-fecc-48d9-9c6a-db334596c8b6\") " pod="kube-system/global-pull-secret-syncer-gnlks"
Jul 07 06:58:42.988187 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.988086    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e1fa63c-3fd7-4883-9aad-5770f7c3b08f-host-var-lib-kubelet\") pod \"multus-rw27d\" (UID: \"5e1fa63c-3fd7-4883-9aad-5770f7c3b08f\") " pod="openshift-multus/multus-rw27d"
Jul 07 06:58:42.988187 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.988107    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7c1bef18-765e-4290-8c9c-80c985fcb304-host\") pod \"node-ca-fzzb4\" (UID: \"7c1bef18-765e-4290-8c9c-80c985fcb304\") " pod="openshift-image-registry/node-ca-fzzb4"
Jul 07 06:58:42.988187 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.988126    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/858752de-a754-4997-9bda-cd13f51366d4-lib-modules\") pod \"tuned-2tp8w\" (UID: \"858752de-a754-4997-9bda-cd13f51366d4\") " pod="openshift-cluster-node-tuning-operator/tuned-2tp8w"
Jul 07 06:58:42.988187 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.988149    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5e1fa63c-3fd7-4883-9aad-5770f7c3b08f-etc-kubernetes\") pod \"multus-rw27d\" (UID: \"5e1fa63c-3fd7-4883-9aad-5770f7c3b08f\") " pod="openshift-multus/multus-rw27d"
Jul 07 06:58:42.988322 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.988190    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e1fa63c-3fd7-4883-9aad-5770f7c3b08f-host-var-lib-kubelet\") pod \"multus-rw27d\" (UID: \"5e1fa63c-3fd7-4883-9aad-5770f7c3b08f\") " pod="openshift-multus/multus-rw27d"
Jul 07 06:58:42.988322 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.988198    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5e1fa63c-3fd7-4883-9aad-5770f7c3b08f-etc-kubernetes\") pod \"multus-rw27d\" (UID: \"5e1fa63c-3fd7-4883-9aad-5770f7c3b08f\") " pod="openshift-multus/multus-rw27d"
Jul 07 06:58:42.988322 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.988196    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/323baf0a-36cc-4f8c-b00b-ccc911176299-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7r4r7\" (UID: \"323baf0a-36cc-4f8c-b00b-ccc911176299\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7r4r7"
Jul 07 06:58:42.988322 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.988233    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85470f35-7e80-42cf-8e5d-d83e5276ca5e-host-run-ovn-kubernetes\") pod \"ovnkube-node-wvm5w\" (UID: \"85470f35-7e80-42cf-8e5d-d83e5276ca5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvm5w"
Jul 07 06:58:42.988322 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.988252    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/323baf0a-36cc-4f8c-b00b-ccc911176299-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7r4r7\" (UID: \"323baf0a-36cc-4f8c-b00b-ccc911176299\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7r4r7"
Jul 07 06:58:42.988322 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.988251    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85470f35-7e80-42cf-8e5d-d83e5276ca5e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wvm5w\" (UID: \"85470f35-7e80-42cf-8e5d-d83e5276ca5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvm5w"
Jul 07 06:58:42.988322 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.988275    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/858752de-a754-4997-9bda-cd13f51366d4-lib-modules\") pod \"tuned-2tp8w\" (UID: \"858752de-a754-4997-9bda-cd13f51366d4\") " pod="openshift-cluster-node-tuning-operator/tuned-2tp8w"
Jul 07 06:58:42.988322 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.988295    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5e1fa63c-3fd7-4883-9aad-5770f7c3b08f-multus-socket-dir-parent\") pod \"multus-rw27d\" (UID: \"5e1fa63c-3fd7-4883-9aad-5770f7c3b08f\") " pod="openshift-multus/multus-rw27d"
Jul 07 06:58:42.988322 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.988315    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5e1fa63c-3fd7-4883-9aad-5770f7c3b08f-multus-daemon-config\") pod \"multus-rw27d\" (UID: \"5e1fa63c-3fd7-4883-9aad-5770f7c3b08f\") " pod="openshift-multus/multus-rw27d"
Jul 07 06:58:42.988697 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.988337    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/323baf0a-36cc-4f8c-b00b-ccc911176299-registration-dir\") pod \"aws-ebs-csi-driver-node-7r4r7\" (UID: \"323baf0a-36cc-4f8c-b00b-ccc911176299\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7r4r7"
Jul 07 06:58:42.988697 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.988353    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/76cc2585-6fae-464f-afcf-f629ed055e11-cnibin\") pod \"multus-additional-cni-plugins-j5jdf\" (UID: \"76cc2585-6fae-464f-afcf-f629ed055e11\") " pod="openshift-multus/multus-additional-cni-plugins-j5jdf"
Jul 07 06:58:42.988697 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.988371    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7c1bef18-765e-4290-8c9c-80c985fcb304-serviceca\") pod \"node-ca-fzzb4\" (UID: \"7c1bef18-765e-4290-8c9c-80c985fcb304\") " pod="openshift-image-registry/node-ca-fzzb4"
Jul 07 06:58:42.988697 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.988389    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv9bp\" (UniqueName: \"kubernetes.io/projected/4e917e03-b190-40ac-8c93-cd4d794fa127-kube-api-access-nv9bp\") pod \"network-check-target-nfjzx\" (UID: \"4e917e03-b190-40ac-8c93-cd4d794fa127\") " pod="openshift-network-diagnostics/network-check-target-nfjzx"
Jul 07 06:58:42.988697 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.988392    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5e1fa63c-3fd7-4883-9aad-5770f7c3b08f-multus-socket-dir-parent\") pod \"multus-rw27d\" (UID: \"5e1fa63c-3fd7-4883-9aad-5770f7c3b08f\") " pod="openshift-multus/multus-rw27d"
Jul 07 06:58:42.988697 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.988410    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/85470f35-7e80-42cf-8e5d-d83e5276ca5e-env-overrides\") pod \"ovnkube-node-wvm5w\" (UID: \"85470f35-7e80-42cf-8e5d-d83e5276ca5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvm5w"
Jul 07 06:58:42.988697 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.988431    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/99da3e49-fb00-4c1a-b0ba-cd1be152fb0c-host-slash\") pod \"iptables-alerter-llqp4\" (UID: \"99da3e49-fb00-4c1a-b0ba-cd1be152fb0c\") " pod="openshift-network-operator/iptables-alerter-llqp4"
Jul 07 06:58:42.988697 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.988441    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/76cc2585-6fae-464f-afcf-f629ed055e11-cnibin\") pod \"multus-additional-cni-plugins-j5jdf\" (UID: \"76cc2585-6fae-464f-afcf-f629ed055e11\") " pod="openshift-multus/multus-additional-cni-plugins-j5jdf"
Jul 07 06:58:42.988697 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.988449    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/338d4939-ce17-426b-a06b-bb8f40ebe2a1-konnectivity-ca\") pod \"konnectivity-agent-z246s\" (UID: \"338d4939-ce17-426b-a06b-bb8f40ebe2a1\") " pod="kube-system/konnectivity-agent-z246s"
Jul 07 06:58:42.988697 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.988471    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/858752de-a754-4997-9bda-cd13f51366d4-sys\") pod \"tuned-2tp8w\" (UID: \"858752de-a754-4997-9bda-cd13f51366d4\") " pod="openshift-cluster-node-tuning-operator/tuned-2tp8w"
Jul 07 06:58:42.988697 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.988492    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5e1fa63c-3fd7-4883-9aad-5770f7c3b08f-host-run-k8s-cni-cncf-io\") pod \"multus-rw27d\" (UID: \"5e1fa63c-3fd7-4883-9aad-5770f7c3b08f\") " pod="openshift-multus/multus-rw27d"
Jul 07 06:58:42.988697 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.988514    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5e1fa63c-3fd7-4883-9aad-5770f7c3b08f-multus-conf-dir\") pod \"multus-rw27d\" (UID: \"5e1fa63c-3fd7-4883-9aad-5770f7c3b08f\") " pod="openshift-multus/multus-rw27d"
Jul 07 06:58:42.988697 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.988470    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/323baf0a-36cc-4f8c-b00b-ccc911176299-registration-dir\") pod \"aws-ebs-csi-driver-node-7r4r7\" (UID: \"323baf0a-36cc-4f8c-b00b-ccc911176299\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7r4r7"
Jul 07 06:58:42.988697 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.988506    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/99da3e49-fb00-4c1a-b0ba-cd1be152fb0c-host-slash\") pod \"iptables-alerter-llqp4\" (UID: \"99da3e49-fb00-4c1a-b0ba-cd1be152fb0c\") " pod="openshift-network-operator/iptables-alerter-llqp4"
Jul 07 06:58:42.988697 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.988543    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/323baf0a-36cc-4f8c-b00b-ccc911176299-device-dir\") pod \"aws-ebs-csi-driver-node-7r4r7\" (UID: \"323baf0a-36cc-4f8c-b00b-ccc911176299\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7r4r7"
Jul 07 06:58:42.988697 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.988562    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/858752de-a754-4997-9bda-cd13f51366d4-sys\") pod \"tuned-2tp8w\" (UID: \"858752de-a754-4997-9bda-cd13f51366d4\") " pod="openshift-cluster-node-tuning-operator/tuned-2tp8w"
Jul 07 06:58:42.988697 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.988578    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5e1fa63c-3fd7-4883-9aad-5770f7c3b08f-multus-conf-dir\") pod \"multus-rw27d\" (UID: \"5e1fa63c-3fd7-4883-9aad-5770f7c3b08f\") " pod="openshift-multus/multus-rw27d"
Jul 07 06:58:42.989537 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.988568    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/85470f35-7e80-42cf-8e5d-d83e5276ca5e-ovn-node-metrics-cert\") pod \"ovnkube-node-wvm5w\" (UID: \"85470f35-7e80-42cf-8e5d-d83e5276ca5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvm5w"
Jul 07 06:58:42.989537 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.988568    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5e1fa63c-3fd7-4883-9aad-5770f7c3b08f-host-run-k8s-cni-cncf-io\") pod \"multus-rw27d\" (UID: \"5e1fa63c-3fd7-4883-9aad-5770f7c3b08f\") " pod="openshift-multus/multus-rw27d"
Jul 07 06:58:42.989537 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.988611    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/323baf0a-36cc-4f8c-b00b-ccc911176299-device-dir\") pod \"aws-ebs-csi-driver-node-7r4r7\" (UID: \"323baf0a-36cc-4f8c-b00b-ccc911176299\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7r4r7"
Jul 07 06:58:42.989537 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.988620    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/858752de-a754-4997-9bda-cd13f51366d4-etc-sysctl-d\") pod \"tuned-2tp8w\" (UID: \"858752de-a754-4997-9bda-cd13f51366d4\") " pod="openshift-cluster-node-tuning-operator/tuned-2tp8w"
Jul 07 06:58:42.989537 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.988647    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/858752de-a754-4997-9bda-cd13f51366d4-tmp\") pod \"tuned-2tp8w\" (UID: \"858752de-a754-4997-9bda-cd13f51366d4\") " pod="openshift-cluster-node-tuning-operator/tuned-2tp8w"
Jul 07 06:58:42.989537 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.988672    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5e1fa63c-3fd7-4883-9aad-5770f7c3b08f-cnibin\") pod \"multus-rw27d\" (UID: \"5e1fa63c-3fd7-4883-9aad-5770f7c3b08f\") " pod="openshift-multus/multus-rw27d"
Jul 07 06:58:42.989537 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.988699    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5e1fa63c-3fd7-4883-9aad-5770f7c3b08f-host-run-netns\") pod \"multus-rw27d\" (UID: \"5e1fa63c-3fd7-4883-9aad-5770f7c3b08f\") " pod="openshift-multus/multus-rw27d"
Jul 07 06:58:42.989537 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.988724    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5e1fa63c-3fd7-4883-9aad-5770f7c3b08f-host-var-lib-cni-multus\") pod \"multus-rw27d\" (UID: \"5e1fa63c-3fd7-4883-9aad-5770f7c3b08f\") " pod="openshift-multus/multus-rw27d"
Jul 07 06:58:42.989537 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.988746    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/858752de-a754-4997-9bda-cd13f51366d4-etc-sysctl-d\") pod \"tuned-2tp8w\" (UID: \"858752de-a754-4997-9bda-cd13f51366d4\") " pod="openshift-cluster-node-tuning-operator/tuned-2tp8w"
Jul 07 06:58:42.989537 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.988751    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/76cc2585-6fae-464f-afcf-f629ed055e11-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-j5jdf\" (UID: \"76cc2585-6fae-464f-afcf-f629ed055e11\") " pod="openshift-multus/multus-additional-cni-plugins-j5jdf"
Jul 07 06:58:42.989537 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.988766    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5e1fa63c-3fd7-4883-9aad-5770f7c3b08f-cnibin\") pod \"multus-rw27d\" (UID: \"5e1fa63c-3fd7-4883-9aad-5770f7c3b08f\") " pod="openshift-multus/multus-rw27d"
Jul 07 06:58:42.989537 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.988778    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s28r2\" (UniqueName: \"kubernetes.io/projected/99da3e49-fb00-4c1a-b0ba-cd1be152fb0c-kube-api-access-s28r2\") pod \"iptables-alerter-llqp4\" (UID: \"99da3e49-fb00-4c1a-b0ba-cd1be152fb0c\") " pod="openshift-network-operator/iptables-alerter-llqp4"
Jul 07 06:58:42.989537 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.988787    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5e1fa63c-3fd7-4883-9aad-5770f7c3b08f-host-run-netns\") pod \"multus-rw27d\" (UID: \"5e1fa63c-3fd7-4883-9aad-5770f7c3b08f\") " pod="openshift-multus/multus-rw27d"
Jul 07 06:58:42.989537 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.988795    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5e1fa63c-3fd7-4883-9aad-5770f7c3b08f-host-var-lib-cni-multus\") pod \"multus-rw27d\" (UID: \"5e1fa63c-3fd7-4883-9aad-5770f7c3b08f\") " pod="openshift-multus/multus-rw27d"
Jul 07 06:58:42.989537 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.988807    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/85470f35-7e80-42cf-8e5d-d83e5276ca5e-run-systemd\") pod \"ovnkube-node-wvm5w\" (UID: \"85470f35-7e80-42cf-8e5d-d83e5276ca5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvm5w"
Jul 07 06:58:42.989537 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.988834    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/858752de-a754-4997-9bda-cd13f51366d4-etc-sysctl-conf\") pod \"tuned-2tp8w\" (UID: \"858752de-a754-4997-9bda-cd13f51366d4\") " pod="openshift-cluster-node-tuning-operator/tuned-2tp8w"
Jul 07 06:58:42.989537 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.988859    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5e1fa63c-3fd7-4883-9aad-5770f7c3b08f-host-var-lib-cni-bin\") pod \"multus-rw27d\" (UID: \"5e1fa63c-3fd7-4883-9aad-5770f7c3b08f\") " pod="openshift-multus/multus-rw27d"
Jul 07 06:58:42.989537 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.988885    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5e1fa63c-3fd7-4883-9aad-5770f7c3b08f-hostroot\") pod \"multus-rw27d\" (UID: \"5e1fa63c-3fd7-4883-9aad-5770f7c3b08f\") " pod="openshift-multus/multus-rw27d"
Jul 07 06:58:42.990405 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.988911    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/85470f35-7e80-42cf-8e5d-d83e5276ca5e-run-openvswitch\") pod \"ovnkube-node-wvm5w\" (UID: \"85470f35-7e80-42cf-8e5d-d83e5276ca5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvm5w"
Jul 07 06:58:42.990405 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.988930    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5e1fa63c-3fd7-4883-9aad-5770f7c3b08f-multus-daemon-config\") pod \"multus-rw27d\" (UID: \"5e1fa63c-3fd7-4883-9aad-5770f7c3b08f\") " pod="openshift-multus/multus-rw27d"
Jul 07 06:58:42.990405 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.988938    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/85470f35-7e80-42cf-8e5d-d83e5276ca5e-ovnkube-config\") pod \"ovnkube-node-wvm5w\" (UID: \"85470f35-7e80-42cf-8e5d-d83e5276ca5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvm5w"
Jul 07 06:58:42.990405 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.988964    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/858752de-a754-4997-9bda-cd13f51366d4-etc-systemd\") pod \"tuned-2tp8w\" (UID: \"858752de-a754-4997-9bda-cd13f51366d4\") " pod="openshift-cluster-node-tuning-operator/tuned-2tp8w"
Jul 07 06:58:42.990405 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.988991    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/858752de-a754-4997-9bda-cd13f51366d4-run\") pod \"tuned-2tp8w\" (UID: \"858752de-a754-4997-9bda-cd13f51366d4\") " pod="openshift-cluster-node-tuning-operator/tuned-2tp8w"
Jul 07 06:58:42.990405 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.989036    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/858752de-a754-4997-9bda-cd13f51366d4-etc-sysctl-conf\") pod \"tuned-2tp8w\" (UID: \"858752de-a754-4997-9bda-cd13f51366d4\") " pod="openshift-cluster-node-tuning-operator/tuned-2tp8w"
Jul 07 06:58:42.990405 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.989050    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/338d4939-ce17-426b-a06b-bb8f40ebe2a1-konnectivity-ca\") pod \"konnectivity-agent-z246s\" (UID: \"338d4939-ce17-426b-a06b-bb8f40ebe2a1\") " pod="kube-system/konnectivity-agent-z246s"
Jul 07 06:58:42.990405 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.989070    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5e1fa63c-3fd7-4883-9aad-5770f7c3b08f-host-var-lib-cni-bin\") pod \"multus-rw27d\" (UID: \"5e1fa63c-3fd7-4883-9aad-5770f7c3b08f\") " pod="openshift-multus/multus-rw27d"
Jul 07 06:58:42.990405 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.989037    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lwwf\" (UniqueName: \"kubernetes.io/projected/858752de-a754-4997-9bda-cd13f51366d4-kube-api-access-5lwwf\") pod \"tuned-2tp8w\" (UID: \"858752de-a754-4997-9bda-cd13f51366d4\") " pod="openshift-cluster-node-tuning-operator/tuned-2tp8w"
Jul 07 06:58:42.990405 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.989093    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/858752de-a754-4997-9bda-cd13f51366d4-etc-systemd\") pod \"tuned-2tp8w\" (UID: \"858752de-a754-4997-9bda-cd13f51366d4\") " pod="openshift-cluster-node-tuning-operator/tuned-2tp8w"
Jul 07 06:58:42.990405 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.989033    2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 07 06:58:42.990405 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.989109    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/323baf0a-36cc-4f8c-b00b-ccc911176299-etc-selinux\") pod \"aws-ebs-csi-driver-node-7r4r7\" (UID: \"323baf0a-36cc-4f8c-b00b-ccc911176299\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7r4r7"
Jul 07 06:58:42.990405 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.989136    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/76cc2585-6fae-464f-afcf-f629ed055e11-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-j5jdf\" (UID: \"76cc2585-6fae-464f-afcf-f629ed055e11\") " pod="openshift-multus/multus-additional-cni-plugins-j5jdf"
Jul 07 06:58:42.990405 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.989163    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/85470f35-7e80-42cf-8e5d-d83e5276ca5e-host-slash\") pod \"ovnkube-node-wvm5w\" (UID: \"85470f35-7e80-42cf-8e5d-d83e5276ca5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvm5w"
Jul 07 06:58:42.990405 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.989189    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/858752de-a754-4997-9bda-cd13f51366d4-var-lib-kubelet\") pod \"tuned-2tp8w\" (UID: \"858752de-a754-4997-9bda-cd13f51366d4\") " pod="openshift-cluster-node-tuning-operator/tuned-2tp8w"
Jul 07 06:58:42.990405 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.989203    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/323baf0a-36cc-4f8c-b00b-ccc911176299-etc-selinux\") pod \"aws-ebs-csi-driver-node-7r4r7\" (UID: \"323baf0a-36cc-4f8c-b00b-ccc911176299\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7r4r7"
Jul 07 06:58:42.990405 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.989215    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/323baf0a-36cc-4f8c-b00b-ccc911176299-sys-fs\") pod \"aws-ebs-csi-driver-node-7r4r7\" (UID: \"323baf0a-36cc-4f8c-b00b-ccc911176299\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7r4r7"
Jul 07 06:58:42.991050 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.989243    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/76cc2585-6fae-464f-afcf-f629ed055e11-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j5jdf\" (UID: \"76cc2585-6fae-464f-afcf-f629ed055e11\") " pod="openshift-multus/multus-additional-cni-plugins-j5jdf"
Jul 07 06:58:42.991050 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.989285    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/323baf0a-36cc-4f8c-b00b-ccc911176299-sys-fs\") pod \"aws-ebs-csi-driver-node-7r4r7\" (UID: \"323baf0a-36cc-4f8c-b00b-ccc911176299\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7r4r7"
Jul 07 06:58:42.991050 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.989287    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/858752de-a754-4997-9bda-cd13f51366d4-run\") pod \"tuned-2tp8w\" (UID: \"858752de-a754-4997-9bda-cd13f51366d4\") " pod="openshift-cluster-node-tuning-operator/tuned-2tp8w"
Jul 07 06:58:42.991050 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.989305    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/86018993-6534-4639-9254-a898f62948fe-hosts-file\") pod \"node-resolver-r77h8\" (UID: \"86018993-6534-4639-9254-a898f62948fe\") " pod="openshift-dns/node-resolver-r77h8"
Jul 07 06:58:42.991050 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.989107    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5e1fa63c-3fd7-4883-9aad-5770f7c3b08f-hostroot\") pod \"multus-rw27d\" (UID: \"5e1fa63c-3fd7-4883-9aad-5770f7c3b08f\") " pod="openshift-multus/multus-rw27d"
Jul 07 06:58:42.991050 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.989322    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/858752de-a754-4997-9bda-cd13f51366d4-var-lib-kubelet\") pod \"tuned-2tp8w\" (UID: \"858752de-a754-4997-9bda-cd13f51366d4\") " pod="openshift-cluster-node-tuning-operator/tuned-2tp8w"
Jul 07 06:58:42.991050 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.989344    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/85470f35-7e80-42cf-8e5d-d83e5276ca5e-host-kubelet\") pod \"ovnkube-node-wvm5w\" (UID: \"85470f35-7e80-42cf-8e5d-d83e5276ca5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvm5w"
Jul 07 06:58:42.991050 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.989377    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/76cc2585-6fae-464f-afcf-f629ed055e11-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-j5jdf\" (UID: \"76cc2585-6fae-464f-afcf-f629ed055e11\") " pod="openshift-multus/multus-additional-cni-plugins-j5jdf"
Jul 07 06:58:42.991050 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.989388    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/76cc2585-6fae-464f-afcf-f629ed055e11-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j5jdf\" (UID: \"76cc2585-6fae-464f-afcf-f629ed055e11\") " pod="openshift-multus/multus-additional-cni-plugins-j5jdf"
Jul 07 06:58:42.991050 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.989387    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fgh7\" (UniqueName: \"kubernetes.io/projected/85470f35-7e80-42cf-8e5d-d83e5276ca5e-kube-api-access-5fgh7\") pod \"ovnkube-node-wvm5w\" (UID: \"85470f35-7e80-42cf-8e5d-d83e5276ca5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvm5w"
Jul 07 06:58:42.991050 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.989433    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/76cc2585-6fae-464f-afcf-f629ed055e11-system-cni-dir\") pod \"multus-additional-cni-plugins-j5jdf\" (UID: \"76cc2585-6fae-464f-afcf-f629ed055e11\") " pod="openshift-multus/multus-additional-cni-plugins-j5jdf"
Jul 07 06:58:42.991050 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.989462    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/575f1248-fecc-48d9-9c6a-db334596c8b6-dbus\") pod \"global-pull-secret-syncer-gnlks\" (UID: \"575f1248-fecc-48d9-9c6a-db334596c8b6\") " pod="kube-system/global-pull-secret-syncer-gnlks"
Jul 07 06:58:42.991050 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.989489    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/858752de-a754-4997-9bda-cd13f51366d4-etc-tuned\") pod \"tuned-2tp8w\" (UID: \"858752de-a754-4997-9bda-cd13f51366d4\") " pod="openshift-cluster-node-tuning-operator/tuned-2tp8w"
Jul 07 06:58:42.991050 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.989509    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/76cc2585-6fae-464f-afcf-f629ed055e11-system-cni-dir\") pod \"multus-additional-cni-plugins-j5jdf\" (UID: \"76cc2585-6fae-464f-afcf-f629ed055e11\") " pod="openshift-multus/multus-additional-cni-plugins-j5jdf"
Jul 07 06:58:42.991050 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.989517    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/85470f35-7e80-42cf-8e5d-d83e5276ca5e-etc-openvswitch\") pod \"ovnkube-node-wvm5w\" (UID: \"85470f35-7e80-42cf-8e5d-d83e5276ca5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvm5w"
Jul 07 06:58:42.991050 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.989548    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/85470f35-7e80-42cf-8e5d-d83e5276ca5e-host-cni-netd\") pod \"ovnkube-node-wvm5w\" (UID: \"85470f35-7e80-42cf-8e5d-d83e5276ca5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvm5w"
Jul 07 06:58:42.991050 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.989574    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/858752de-a754-4997-9bda-cd13f51366d4-etc-modprobe-d\") pod \"tuned-2tp8w\" (UID: \"858752de-a754-4997-9bda-cd13f51366d4\") " pod="openshift-cluster-node-tuning-operator/tuned-2tp8w"
Jul 07 06:58:42.991553 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.989608    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5e1fa63c-3fd7-4883-9aad-5770f7c3b08f-cni-binary-copy\") pod \"multus-rw27d\" (UID: \"5e1fa63c-3fd7-4883-9aad-5770f7c3b08f\") " pod="openshift-multus/multus-rw27d"
Jul 07 06:58:42.991553 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.989634    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0dba90e8-7ad4-46cb-bc27-718b6f644f5d-metrics-certs\") pod \"network-metrics-daemon-s9stz\" (UID: \"0dba90e8-7ad4-46cb-bc27-718b6f644f5d\") " pod="openshift-multus/network-metrics-daemon-s9stz"
Jul 07 06:58:42.991553 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.989661    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/323baf0a-36cc-4f8c-b00b-ccc911176299-socket-dir\") pod \"aws-ebs-csi-driver-node-7r4r7\" (UID: \"323baf0a-36cc-4f8c-b00b-ccc911176299\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7r4r7"
Jul 07 06:58:42.991553 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.989688    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/99da3e49-fb00-4c1a-b0ba-cd1be152fb0c-iptables-alerter-script\") pod \"iptables-alerter-llqp4\" (UID: \"99da3e49-fb00-4c1a-b0ba-cd1be152fb0c\") " pod="openshift-network-operator/iptables-alerter-llqp4"
Jul 07 06:58:42.991553 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.989717    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/85470f35-7e80-42cf-8e5d-d83e5276ca5e-log-socket\") pod \"ovnkube-node-wvm5w\" (UID: \"85470f35-7e80-42cf-8e5d-d83e5276ca5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvm5w"
Jul 07 06:58:42.991553 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.989742    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/85470f35-7e80-42cf-8e5d-d83e5276ca5e-host-cni-bin\") pod \"ovnkube-node-wvm5w\" (UID: \"85470f35-7e80-42cf-8e5d-d83e5276ca5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvm5w"
Jul 07 06:58:42.991553 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.989688    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/858752de-a754-4997-9bda-cd13f51366d4-etc-modprobe-d\") pod \"tuned-2tp8w\" (UID: \"858752de-a754-4997-9bda-cd13f51366d4\") " pod="openshift-cluster-node-tuning-operator/tuned-2tp8w"
Jul 07 06:58:42.991553 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:42.989775    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 06:58:42.991553 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.989774    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/858752de-a754-4997-9bda-cd13f51366d4-etc-kubernetes\") pod \"tuned-2tp8w\" (UID: \"858752de-a754-4997-9bda-cd13f51366d4\") " pod="openshift-cluster-node-tuning-operator/tuned-2tp8w"
Jul 07 06:58:42.991553 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.989802    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/323baf0a-36cc-4f8c-b00b-ccc911176299-socket-dir\") pod \"aws-ebs-csi-driver-node-7r4r7\" (UID: \"323baf0a-36cc-4f8c-b00b-ccc911176299\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7r4r7"
Jul 07 06:58:42.991553 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.989804    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/76cc2585-6fae-464f-afcf-f629ed055e11-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-j5jdf\" (UID: \"76cc2585-6fae-464f-afcf-f629ed055e11\") " pod="openshift-multus/multus-additional-cni-plugins-j5jdf"
Jul 07 06:58:42.991553 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.989820    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5e1fa63c-3fd7-4883-9aad-5770f7c3b08f-multus-cni-dir\") pod \"multus-rw27d\" (UID: \"5e1fa63c-3fd7-4883-9aad-5770f7c3b08f\") " pod="openshift-multus/multus-rw27d"
Jul 07 06:58:42.991553 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.989861    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/858752de-a754-4997-9bda-cd13f51366d4-etc-kubernetes\") pod \"tuned-2tp8w\" (UID: \"858752de-a754-4997-9bda-cd13f51366d4\") " pod="openshift-cluster-node-tuning-operator/tuned-2tp8w"
Jul 07 06:58:42.991553 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.989870    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5tlq\" (UniqueName: \"kubernetes.io/projected/323baf0a-36cc-4f8c-b00b-ccc911176299-kube-api-access-q5tlq\") pod \"aws-ebs-csi-driver-node-7r4r7\" (UID: \"323baf0a-36cc-4f8c-b00b-ccc911176299\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7r4r7"
Jul 07 06:58:42.991553 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:42.989896    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0dba90e8-7ad4-46cb-bc27-718b6f644f5d-metrics-certs podName:0dba90e8-7ad4-46cb-bc27-718b6f644f5d nodeName:}" failed. No retries permitted until 2026-07-07 06:58:43.489874298 +0000 UTC m=+3.033635787 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0dba90e8-7ad4-46cb-bc27-718b6f644f5d-metrics-certs") pod "network-metrics-daemon-s9stz" (UID: "0dba90e8-7ad4-46cb-bc27-718b6f644f5d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 06:58:42.991553 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.989943    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5e1fa63c-3fd7-4883-9aad-5770f7c3b08f-multus-cni-dir\") pod \"multus-rw27d\" (UID: \"5e1fa63c-3fd7-4883-9aad-5770f7c3b08f\") " pod="openshift-multus/multus-rw27d"
Jul 07 06:58:42.991553 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.989995    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/76cc2585-6fae-464f-afcf-f629ed055e11-os-release\") pod \"multus-additional-cni-plugins-j5jdf\" (UID: \"76cc2585-6fae-464f-afcf-f629ed055e11\") " pod="openshift-multus/multus-additional-cni-plugins-j5jdf"
Jul 07 06:58:42.992106 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.990017    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/76cc2585-6fae-464f-afcf-f629ed055e11-cni-binary-copy\") pod \"multus-additional-cni-plugins-j5jdf\" (UID: \"76cc2585-6fae-464f-afcf-f629ed055e11\") " pod="openshift-multus/multus-additional-cni-plugins-j5jdf"
Jul 07 06:58:42.992106 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.990039    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/85470f35-7e80-42cf-8e5d-d83e5276ca5e-ovnkube-script-lib\") pod \"ovnkube-node-wvm5w\" (UID: \"85470f35-7e80-42cf-8e5d-d83e5276ca5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvm5w"
Jul 07 06:58:42.992106 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.990066    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/575f1248-fecc-48d9-9c6a-db334596c8b6-kubelet-config\") pod \"global-pull-secret-syncer-gnlks\" (UID: \"575f1248-fecc-48d9-9c6a-db334596c8b6\") " pod="kube-system/global-pull-secret-syncer-gnlks"
Jul 07 06:58:42.992106 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.990092    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/76cc2585-6fae-464f-afcf-f629ed055e11-os-release\") pod \"multus-additional-cni-plugins-j5jdf\" (UID: \"76cc2585-6fae-464f-afcf-f629ed055e11\") " pod="openshift-multus/multus-additional-cni-plugins-j5jdf"
Jul 07 06:58:42.992106 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.990094    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/858752de-a754-4997-9bda-cd13f51366d4-etc-sysconfig\") pod \"tuned-2tp8w\" (UID: \"858752de-a754-4997-9bda-cd13f51366d4\") " pod="openshift-cluster-node-tuning-operator/tuned-2tp8w"
Jul 07 06:58:42.992106 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.990137    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/858752de-a754-4997-9bda-cd13f51366d4-host\") pod \"tuned-2tp8w\" (UID: \"858752de-a754-4997-9bda-cd13f51366d4\") " pod="openshift-cluster-node-tuning-operator/tuned-2tp8w"
Jul 07 06:58:42.992106 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.990133    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/858752de-a754-4997-9bda-cd13f51366d4-etc-sysconfig\") pod \"tuned-2tp8w\" (UID: \"858752de-a754-4997-9bda-cd13f51366d4\") " pod="openshift-cluster-node-tuning-operator/tuned-2tp8w"
Jul 07 06:58:42.992106 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.990174    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5e1fa63c-3fd7-4883-9aad-5770f7c3b08f-host-run-multus-certs\") pod \"multus-rw27d\" (UID: \"5e1fa63c-3fd7-4883-9aad-5770f7c3b08f\") " pod="openshift-multus/multus-rw27d"
Jul 07 06:58:42.992106 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.990216    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/858752de-a754-4997-9bda-cd13f51366d4-host\") pod \"tuned-2tp8w\" (UID: \"858752de-a754-4997-9bda-cd13f51366d4\") " pod="openshift-cluster-node-tuning-operator/tuned-2tp8w"
Jul 07 06:58:42.992106 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.990226    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5e1fa63c-3fd7-4883-9aad-5770f7c3b08f-host-run-multus-certs\") pod \"multus-rw27d\" (UID: \"5e1fa63c-3fd7-4883-9aad-5770f7c3b08f\") " pod="openshift-multus/multus-rw27d"
Jul 07 06:58:42.992106 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.990242    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5bw9\" (UniqueName: \"kubernetes.io/projected/5e1fa63c-3fd7-4883-9aad-5770f7c3b08f-kube-api-access-r5bw9\") pod \"multus-rw27d\" (UID: \"5e1fa63c-3fd7-4883-9aad-5770f7c3b08f\") " pod="openshift-multus/multus-rw27d"
Jul 07 06:58:42.992106 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.990270    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2bss\" (UniqueName: \"kubernetes.io/projected/0dba90e8-7ad4-46cb-bc27-718b6f644f5d-kube-api-access-b2bss\") pod \"network-metrics-daemon-s9stz\" (UID: \"0dba90e8-7ad4-46cb-bc27-718b6f644f5d\") " pod="openshift-multus/network-metrics-daemon-s9stz"
Jul 07 06:58:42.992106 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.990294    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5e1fa63c-3fd7-4883-9aad-5770f7c3b08f-cni-binary-copy\") pod \"multus-rw27d\" (UID: \"5e1fa63c-3fd7-4883-9aad-5770f7c3b08f\") " pod="openshift-multus/multus-rw27d"
Jul 07 06:58:42.992106 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.990297    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhn8b\" (UniqueName: \"kubernetes.io/projected/76cc2585-6fae-464f-afcf-f629ed055e11-kube-api-access-zhn8b\") pod \"multus-additional-cni-plugins-j5jdf\" (UID: \"76cc2585-6fae-464f-afcf-f629ed055e11\") " pod="openshift-multus/multus-additional-cni-plugins-j5jdf"
Jul 07 06:58:42.992106 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.990327    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/99da3e49-fb00-4c1a-b0ba-cd1be152fb0c-iptables-alerter-script\") pod \"iptables-alerter-llqp4\" (UID: \"99da3e49-fb00-4c1a-b0ba-cd1be152fb0c\") " pod="openshift-network-operator/iptables-alerter-llqp4"
Jul 07 06:58:42.992106 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.990348    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/338d4939-ce17-426b-a06b-bb8f40ebe2a1-agent-certs\") pod \"konnectivity-agent-z246s\" (UID: \"338d4939-ce17-426b-a06b-bb8f40ebe2a1\") " pod="kube-system/konnectivity-agent-z246s"
Jul 07 06:58:42.992106 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.990378    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86018993-6534-4639-9254-a898f62948fe-tmp-dir\") pod \"node-resolver-r77h8\" (UID: \"86018993-6534-4639-9254-a898f62948fe\") " pod="openshift-dns/node-resolver-r77h8"
Jul 07 06:58:42.992592 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.990414    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5e1fa63c-3fd7-4883-9aad-5770f7c3b08f-system-cni-dir\") pod \"multus-rw27d\" (UID: \"5e1fa63c-3fd7-4883-9aad-5770f7c3b08f\") " pod="openshift-multus/multus-rw27d"
Jul 07 06:58:42.992592 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.990438    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5e1fa63c-3fd7-4883-9aad-5770f7c3b08f-os-release\") pod \"multus-rw27d\" (UID: \"5e1fa63c-3fd7-4883-9aad-5770f7c3b08f\") " pod="openshift-multus/multus-rw27d"
Jul 07 06:58:42.992592 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.990483    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5e1fa63c-3fd7-4883-9aad-5770f7c3b08f-system-cni-dir\") pod \"multus-rw27d\" (UID: \"5e1fa63c-3fd7-4883-9aad-5770f7c3b08f\") " pod="openshift-multus/multus-rw27d"
Jul 07 06:58:42.992592 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.990500    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5e1fa63c-3fd7-4883-9aad-5770f7c3b08f-os-release\") pod \"multus-rw27d\" (UID: \"5e1fa63c-3fd7-4883-9aad-5770f7c3b08f\") " pod="openshift-multus/multus-rw27d"
Jul 07 06:58:42.992592 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.990521    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnwmv\" (UniqueName: \"kubernetes.io/projected/7c1bef18-765e-4290-8c9c-80c985fcb304-kube-api-access-vnwmv\") pod \"node-ca-fzzb4\" (UID: \"7c1bef18-765e-4290-8c9c-80c985fcb304\") " pod="openshift-image-registry/node-ca-fzzb4"
Jul 07 06:58:42.992592 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.990549    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pb9zq\" (UniqueName: \"kubernetes.io/projected/86018993-6534-4639-9254-a898f62948fe-kube-api-access-pb9zq\") pod \"node-resolver-r77h8\" (UID: \"86018993-6534-4639-9254-a898f62948fe\") " pod="openshift-dns/node-resolver-r77h8"
Jul 07 06:58:42.992592 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.990575    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/85470f35-7e80-42cf-8e5d-d83e5276ca5e-systemd-units\") pod \"ovnkube-node-wvm5w\" (UID: \"85470f35-7e80-42cf-8e5d-d83e5276ca5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvm5w"
Jul 07 06:58:42.992592 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.990589    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/76cc2585-6fae-464f-afcf-f629ed055e11-cni-binary-copy\") pod \"multus-additional-cni-plugins-j5jdf\" (UID: \"76cc2585-6fae-464f-afcf-f629ed055e11\") " pod="openshift-multus/multus-additional-cni-plugins-j5jdf"
Jul 07 06:58:42.992592 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.990601    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/85470f35-7e80-42cf-8e5d-d83e5276ca5e-host-run-netns\") pod \"ovnkube-node-wvm5w\" (UID: \"85470f35-7e80-42cf-8e5d-d83e5276ca5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvm5w"
Jul 07 06:58:42.992592 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.990658    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/85470f35-7e80-42cf-8e5d-d83e5276ca5e-var-lib-openvswitch\") pod \"ovnkube-node-wvm5w\" (UID: \"85470f35-7e80-42cf-8e5d-d83e5276ca5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvm5w"
Jul 07 06:58:42.992592 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.990697    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/85470f35-7e80-42cf-8e5d-d83e5276ca5e-run-ovn\") pod \"ovnkube-node-wvm5w\" (UID: \"85470f35-7e80-42cf-8e5d-d83e5276ca5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvm5w"
Jul 07 06:58:42.992592 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.990717    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/85470f35-7e80-42cf-8e5d-d83e5276ca5e-node-log\") pod \"ovnkube-node-wvm5w\" (UID: \"85470f35-7e80-42cf-8e5d-d83e5276ca5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvm5w"
Jul 07 06:58:42.993000 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.992974    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/858752de-a754-4997-9bda-cd13f51366d4-tmp\") pod \"tuned-2tp8w\" (UID: \"858752de-a754-4997-9bda-cd13f51366d4\") " pod="openshift-cluster-node-tuning-operator/tuned-2tp8w"
Jul 07 06:58:42.993000 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.992990    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/858752de-a754-4997-9bda-cd13f51366d4-etc-tuned\") pod \"tuned-2tp8w\" (UID: \"858752de-a754-4997-9bda-cd13f51366d4\") " pod="openshift-cluster-node-tuning-operator/tuned-2tp8w"
Jul 07 06:58:42.993178 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:42.993160    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/338d4939-ce17-426b-a06b-bb8f40ebe2a1-agent-certs\") pod \"konnectivity-agent-z246s\" (UID: \"338d4939-ce17-426b-a06b-bb8f40ebe2a1\") " pod="kube-system/konnectivity-agent-z246s"
Jul 07 06:58:43.002613 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.002583    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5bw9\" (UniqueName: \"kubernetes.io/projected/5e1fa63c-3fd7-4883-9aad-5770f7c3b08f-kube-api-access-r5bw9\") pod \"multus-rw27d\" (UID: \"5e1fa63c-3fd7-4883-9aad-5770f7c3b08f\") " pod="openshift-multus/multus-rw27d"
Jul 07 06:58:43.002738 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.002624    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lwwf\" (UniqueName: \"kubernetes.io/projected/858752de-a754-4997-9bda-cd13f51366d4-kube-api-access-5lwwf\") pod \"tuned-2tp8w\" (UID: \"858752de-a754-4997-9bda-cd13f51366d4\") " pod="openshift-cluster-node-tuning-operator/tuned-2tp8w"
Jul 07 06:58:43.003177 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.003154    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5tlq\" (UniqueName: \"kubernetes.io/projected/323baf0a-36cc-4f8c-b00b-ccc911176299-kube-api-access-q5tlq\") pod \"aws-ebs-csi-driver-node-7r4r7\" (UID: \"323baf0a-36cc-4f8c-b00b-ccc911176299\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7r4r7"
Jul 07 06:58:43.003267 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.003234    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhn8b\" (UniqueName: \"kubernetes.io/projected/76cc2585-6fae-464f-afcf-f629ed055e11-kube-api-access-zhn8b\") pod \"multus-additional-cni-plugins-j5jdf\" (UID: \"76cc2585-6fae-464f-afcf-f629ed055e11\") " pod="openshift-multus/multus-additional-cni-plugins-j5jdf"
Jul 07 06:58:43.003446 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.003429    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s28r2\" (UniqueName: \"kubernetes.io/projected/99da3e49-fb00-4c1a-b0ba-cd1be152fb0c-kube-api-access-s28r2\") pod \"iptables-alerter-llqp4\" (UID: \"99da3e49-fb00-4c1a-b0ba-cd1be152fb0c\") " pod="openshift-network-operator/iptables-alerter-llqp4"
Jul 07 06:58:43.003736 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.003720    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2bss\" (UniqueName: \"kubernetes.io/projected/0dba90e8-7ad4-46cb-bc27-718b6f644f5d-kube-api-access-b2bss\") pod \"network-metrics-daemon-s9stz\" (UID: \"0dba90e8-7ad4-46cb-bc27-718b6f644f5d\") " pod="openshift-multus/network-metrics-daemon-s9stz"
Jul 07 06:58:43.091094 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.091062    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/85470f35-7e80-42cf-8e5d-d83e5276ca5e-etc-openvswitch\") pod \"ovnkube-node-wvm5w\" (UID: \"85470f35-7e80-42cf-8e5d-d83e5276ca5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvm5w"
Jul 07 06:58:43.091094 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.091094    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/85470f35-7e80-42cf-8e5d-d83e5276ca5e-host-cni-netd\") pod \"ovnkube-node-wvm5w\" (UID: \"85470f35-7e80-42cf-8e5d-d83e5276ca5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvm5w"
Jul 07 06:58:43.091311 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.091125    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/85470f35-7e80-42cf-8e5d-d83e5276ca5e-log-socket\") pod \"ovnkube-node-wvm5w\" (UID: \"85470f35-7e80-42cf-8e5d-d83e5276ca5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvm5w"
Jul 07 06:58:43.091311 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.091150    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/85470f35-7e80-42cf-8e5d-d83e5276ca5e-host-cni-bin\") pod \"ovnkube-node-wvm5w\" (UID: \"85470f35-7e80-42cf-8e5d-d83e5276ca5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvm5w"
Jul 07 06:58:43.091311 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.091185    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/85470f35-7e80-42cf-8e5d-d83e5276ca5e-host-cni-netd\") pod \"ovnkube-node-wvm5w\" (UID: \"85470f35-7e80-42cf-8e5d-d83e5276ca5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvm5w"
Jul 07 06:58:43.091311 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.091189    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/85470f35-7e80-42cf-8e5d-d83e5276ca5e-etc-openvswitch\") pod \"ovnkube-node-wvm5w\" (UID: \"85470f35-7e80-42cf-8e5d-d83e5276ca5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvm5w"
Jul 07 06:58:43.091311 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.091204    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/85470f35-7e80-42cf-8e5d-d83e5276ca5e-log-socket\") pod \"ovnkube-node-wvm5w\" (UID: \"85470f35-7e80-42cf-8e5d-d83e5276ca5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvm5w"
Jul 07 06:58:43.091311 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.091242    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/85470f35-7e80-42cf-8e5d-d83e5276ca5e-host-cni-bin\") pod \"ovnkube-node-wvm5w\" (UID: \"85470f35-7e80-42cf-8e5d-d83e5276ca5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvm5w"
Jul 07 06:58:43.091311 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.091281    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/85470f35-7e80-42cf-8e5d-d83e5276ca5e-ovnkube-script-lib\") pod \"ovnkube-node-wvm5w\" (UID: \"85470f35-7e80-42cf-8e5d-d83e5276ca5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvm5w"
Jul 07 06:58:43.091311 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.091312    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/575f1248-fecc-48d9-9c6a-db334596c8b6-kubelet-config\") pod \"global-pull-secret-syncer-gnlks\" (UID: \"575f1248-fecc-48d9-9c6a-db334596c8b6\") " pod="kube-system/global-pull-secret-syncer-gnlks"
Jul 07 06:58:43.091653 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.091335    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86018993-6534-4639-9254-a898f62948fe-tmp-dir\") pod \"node-resolver-r77h8\" (UID: \"86018993-6534-4639-9254-a898f62948fe\") " pod="openshift-dns/node-resolver-r77h8"
Jul 07 06:58:43.091653 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.091354    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnwmv\" (UniqueName: \"kubernetes.io/projected/7c1bef18-765e-4290-8c9c-80c985fcb304-kube-api-access-vnwmv\") pod \"node-ca-fzzb4\" (UID: \"7c1bef18-765e-4290-8c9c-80c985fcb304\") " pod="openshift-image-registry/node-ca-fzzb4"
Jul 07 06:58:43.091653 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.091373    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb9zq\" (UniqueName: \"kubernetes.io/projected/86018993-6534-4639-9254-a898f62948fe-kube-api-access-pb9zq\") pod \"node-resolver-r77h8\" (UID: \"86018993-6534-4639-9254-a898f62948fe\") " pod="openshift-dns/node-resolver-r77h8"
Jul 07 06:58:43.091653 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.091397    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/85470f35-7e80-42cf-8e5d-d83e5276ca5e-systemd-units\") pod \"ovnkube-node-wvm5w\" (UID: \"85470f35-7e80-42cf-8e5d-d83e5276ca5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvm5w"
Jul 07 06:58:43.091653 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.091406    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/575f1248-fecc-48d9-9c6a-db334596c8b6-kubelet-config\") pod \"global-pull-secret-syncer-gnlks\" (UID: \"575f1248-fecc-48d9-9c6a-db334596c8b6\") " pod="kube-system/global-pull-secret-syncer-gnlks"
Jul 07 06:58:43.091653 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.091424    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/85470f35-7e80-42cf-8e5d-d83e5276ca5e-host-run-netns\") pod \"ovnkube-node-wvm5w\" (UID: \"85470f35-7e80-42cf-8e5d-d83e5276ca5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvm5w"
Jul 07 06:58:43.091653 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.091449    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/85470f35-7e80-42cf-8e5d-d83e5276ca5e-var-lib-openvswitch\") pod \"ovnkube-node-wvm5w\" (UID: \"85470f35-7e80-42cf-8e5d-d83e5276ca5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvm5w"
Jul 07 06:58:43.091653 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.091475    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/85470f35-7e80-42cf-8e5d-d83e5276ca5e-run-ovn\") pod \"ovnkube-node-wvm5w\" (UID: \"85470f35-7e80-42cf-8e5d-d83e5276ca5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvm5w"
Jul 07 06:58:43.091653 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.091484    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/85470f35-7e80-42cf-8e5d-d83e5276ca5e-systemd-units\") pod \"ovnkube-node-wvm5w\" (UID: \"85470f35-7e80-42cf-8e5d-d83e5276ca5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvm5w"
Jul 07 06:58:43.091653 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.091497    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/85470f35-7e80-42cf-8e5d-d83e5276ca5e-node-log\") pod \"ovnkube-node-wvm5w\" (UID: \"85470f35-7e80-42cf-8e5d-d83e5276ca5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvm5w"
Jul 07 06:58:43.091653 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.091535    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/85470f35-7e80-42cf-8e5d-d83e5276ca5e-var-lib-openvswitch\") pod \"ovnkube-node-wvm5w\" (UID: \"85470f35-7e80-42cf-8e5d-d83e5276ca5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvm5w"
Jul 07 06:58:43.091653 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.091578    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/575f1248-fecc-48d9-9c6a-db334596c8b6-original-pull-secret\") pod \"global-pull-secret-syncer-gnlks\" (UID: \"575f1248-fecc-48d9-9c6a-db334596c8b6\") " pod="kube-system/global-pull-secret-syncer-gnlks"
Jul 07 06:58:43.091653 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.091625    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7c1bef18-765e-4290-8c9c-80c985fcb304-host\") pod \"node-ca-fzzb4\" (UID: \"7c1bef18-765e-4290-8c9c-80c985fcb304\") " pod="openshift-image-registry/node-ca-fzzb4"
Jul 07 06:58:43.091653 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.091632    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/85470f35-7e80-42cf-8e5d-d83e5276ca5e-node-log\") pod \"ovnkube-node-wvm5w\" (UID: \"85470f35-7e80-42cf-8e5d-d83e5276ca5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvm5w"
Jul 07 06:58:43.092343 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.091673    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/85470f35-7e80-42cf-8e5d-d83e5276ca5e-run-ovn\") pod \"ovnkube-node-wvm5w\" (UID: \"85470f35-7e80-42cf-8e5d-d83e5276ca5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvm5w"
Jul 07 06:58:43.092343 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.091700    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86018993-6534-4639-9254-a898f62948fe-tmp-dir\") pod \"node-resolver-r77h8\" (UID: \"86018993-6534-4639-9254-a898f62948fe\") " pod="openshift-dns/node-resolver-r77h8"
Jul 07 06:58:43.092343 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:43.091736    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 06:58:43.092343 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.091736    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85470f35-7e80-42cf-8e5d-d83e5276ca5e-host-run-ovn-kubernetes\") pod \"ovnkube-node-wvm5w\" (UID: \"85470f35-7e80-42cf-8e5d-d83e5276ca5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvm5w"
Jul 07 06:58:43.092343 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.091754    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/85470f35-7e80-42cf-8e5d-d83e5276ca5e-host-run-netns\") pod \"ovnkube-node-wvm5w\" (UID: \"85470f35-7e80-42cf-8e5d-d83e5276ca5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvm5w"
Jul 07 06:58:43.092343 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.091782    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7c1bef18-765e-4290-8c9c-80c985fcb304-host\") pod \"node-ca-fzzb4\" (UID: \"7c1bef18-765e-4290-8c9c-80c985fcb304\") " pod="openshift-image-registry/node-ca-fzzb4"
Jul 07 06:58:43.092343 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.091816    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85470f35-7e80-42cf-8e5d-d83e5276ca5e-host-run-ovn-kubernetes\") pod \"ovnkube-node-wvm5w\" (UID: \"85470f35-7e80-42cf-8e5d-d83e5276ca5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvm5w"
Jul 07 06:58:43.092343 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.091848    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85470f35-7e80-42cf-8e5d-d83e5276ca5e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wvm5w\" (UID: \"85470f35-7e80-42cf-8e5d-d83e5276ca5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvm5w"
Jul 07 06:58:43.092343 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.091880    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85470f35-7e80-42cf-8e5d-d83e5276ca5e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wvm5w\" (UID: \"85470f35-7e80-42cf-8e5d-d83e5276ca5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvm5w"
Jul 07 06:58:43.092343 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:43.091882    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/575f1248-fecc-48d9-9c6a-db334596c8b6-original-pull-secret podName:575f1248-fecc-48d9-9c6a-db334596c8b6 nodeName:}" failed. No retries permitted until 2026-07-07 06:58:43.591859948 +0000 UTC m=+3.135621412 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/575f1248-fecc-48d9-9c6a-db334596c8b6-original-pull-secret") pod "global-pull-secret-syncer-gnlks" (UID: "575f1248-fecc-48d9-9c6a-db334596c8b6") : object "kube-system"/"original-pull-secret" not registered
Jul 07 06:58:43.092343 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.091941    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7c1bef18-765e-4290-8c9c-80c985fcb304-serviceca\") pod \"node-ca-fzzb4\" (UID: \"7c1bef18-765e-4290-8c9c-80c985fcb304\") " pod="openshift-image-registry/node-ca-fzzb4"
Jul 07 06:58:43.092343 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.091967    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/85470f35-7e80-42cf-8e5d-d83e5276ca5e-ovnkube-script-lib\") pod \"ovnkube-node-wvm5w\" (UID: \"85470f35-7e80-42cf-8e5d-d83e5276ca5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvm5w"
Jul 07 06:58:43.092343 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.091992    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv9bp\" (UniqueName: \"kubernetes.io/projected/4e917e03-b190-40ac-8c93-cd4d794fa127-kube-api-access-nv9bp\") pod \"network-check-target-nfjzx\" (UID: \"4e917e03-b190-40ac-8c93-cd4d794fa127\") " pod="openshift-network-diagnostics/network-check-target-nfjzx"
Jul 07 06:58:43.092343 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.092019    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/85470f35-7e80-42cf-8e5d-d83e5276ca5e-env-overrides\") pod \"ovnkube-node-wvm5w\" (UID: \"85470f35-7e80-42cf-8e5d-d83e5276ca5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvm5w"
Jul 07 06:58:43.092343 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.092047    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/85470f35-7e80-42cf-8e5d-d83e5276ca5e-ovn-node-metrics-cert\") pod \"ovnkube-node-wvm5w\" (UID: \"85470f35-7e80-42cf-8e5d-d83e5276ca5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvm5w"
Jul 07 06:58:43.092343 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.092089    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/85470f35-7e80-42cf-8e5d-d83e5276ca5e-run-systemd\") pod \"ovnkube-node-wvm5w\" (UID: \"85470f35-7e80-42cf-8e5d-d83e5276ca5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvm5w"
Jul 07 06:58:43.092343 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.092122    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/85470f35-7e80-42cf-8e5d-d83e5276ca5e-run-openvswitch\") pod \"ovnkube-node-wvm5w\" (UID: \"85470f35-7e80-42cf-8e5d-d83e5276ca5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvm5w"
Jul 07 06:58:43.092944 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.092150    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/85470f35-7e80-42cf-8e5d-d83e5276ca5e-run-systemd\") pod \"ovnkube-node-wvm5w\" (UID: \"85470f35-7e80-42cf-8e5d-d83e5276ca5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvm5w"
Jul 07 06:58:43.092944 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.092161    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/85470f35-7e80-42cf-8e5d-d83e5276ca5e-ovnkube-config\") pod \"ovnkube-node-wvm5w\" (UID: \"85470f35-7e80-42cf-8e5d-d83e5276ca5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvm5w"
Jul 07 06:58:43.092944 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.092178    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/85470f35-7e80-42cf-8e5d-d83e5276ca5e-run-openvswitch\") pod \"ovnkube-node-wvm5w\" (UID: \"85470f35-7e80-42cf-8e5d-d83e5276ca5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvm5w"
Jul 07 06:58:43.092944 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.092194    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/85470f35-7e80-42cf-8e5d-d83e5276ca5e-host-slash\") pod \"ovnkube-node-wvm5w\" (UID: \"85470f35-7e80-42cf-8e5d-d83e5276ca5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvm5w"
Jul 07 06:58:43.092944 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.092239    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/86018993-6534-4639-9254-a898f62948fe-hosts-file\") pod \"node-resolver-r77h8\" (UID: \"86018993-6534-4639-9254-a898f62948fe\") " pod="openshift-dns/node-resolver-r77h8"
Jul 07 06:58:43.092944 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.092247    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/85470f35-7e80-42cf-8e5d-d83e5276ca5e-host-slash\") pod \"ovnkube-node-wvm5w\" (UID: \"85470f35-7e80-42cf-8e5d-d83e5276ca5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvm5w"
Jul 07 06:58:43.092944 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.092267    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/85470f35-7e80-42cf-8e5d-d83e5276ca5e-host-kubelet\") pod \"ovnkube-node-wvm5w\" (UID: \"85470f35-7e80-42cf-8e5d-d83e5276ca5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvm5w"
Jul 07 06:58:43.092944 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.092293    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fgh7\" (UniqueName: \"kubernetes.io/projected/85470f35-7e80-42cf-8e5d-d83e5276ca5e-kube-api-access-5fgh7\") pod \"ovnkube-node-wvm5w\" (UID: \"85470f35-7e80-42cf-8e5d-d83e5276ca5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvm5w"
Jul 07 06:58:43.092944 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.092307    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/86018993-6534-4639-9254-a898f62948fe-hosts-file\") pod \"node-resolver-r77h8\" (UID: \"86018993-6534-4639-9254-a898f62948fe\") " pod="openshift-dns/node-resolver-r77h8"
Jul 07 06:58:43.092944 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.092320    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/575f1248-fecc-48d9-9c6a-db334596c8b6-dbus\") pod \"global-pull-secret-syncer-gnlks\" (UID: \"575f1248-fecc-48d9-9c6a-db334596c8b6\") " pod="kube-system/global-pull-secret-syncer-gnlks"
Jul 07 06:58:43.092944 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.092343    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/85470f35-7e80-42cf-8e5d-d83e5276ca5e-host-kubelet\") pod \"ovnkube-node-wvm5w\" (UID: \"85470f35-7e80-42cf-8e5d-d83e5276ca5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvm5w"
Jul 07 06:58:43.092944 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.092420    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/575f1248-fecc-48d9-9c6a-db334596c8b6-dbus\") pod \"global-pull-secret-syncer-gnlks\" (UID: \"575f1248-fecc-48d9-9c6a-db334596c8b6\") " pod="kube-system/global-pull-secret-syncer-gnlks"
Jul 07 06:58:43.092944 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.092436    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/85470f35-7e80-42cf-8e5d-d83e5276ca5e-env-overrides\") pod \"ovnkube-node-wvm5w\" (UID: \"85470f35-7e80-42cf-8e5d-d83e5276ca5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvm5w"
Jul 07 06:58:43.092944 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.092421    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7c1bef18-765e-4290-8c9c-80c985fcb304-serviceca\") pod \"node-ca-fzzb4\" (UID: \"7c1bef18-765e-4290-8c9c-80c985fcb304\") " pod="openshift-image-registry/node-ca-fzzb4"
Jul 07 06:58:43.092944 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.092606    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/85470f35-7e80-42cf-8e5d-d83e5276ca5e-ovnkube-config\") pod \"ovnkube-node-wvm5w\" (UID: \"85470f35-7e80-42cf-8e5d-d83e5276ca5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvm5w"
Jul 07 06:58:43.094468 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.094448    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/85470f35-7e80-42cf-8e5d-d83e5276ca5e-ovn-node-metrics-cert\") pod \"ovnkube-node-wvm5w\" (UID: \"85470f35-7e80-42cf-8e5d-d83e5276ca5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvm5w"
Jul 07 06:58:43.104886 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.104867    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb9zq\" (UniqueName: \"kubernetes.io/projected/86018993-6534-4639-9254-a898f62948fe-kube-api-access-pb9zq\") pod \"node-resolver-r77h8\" (UID: \"86018993-6534-4639-9254-a898f62948fe\") " pod="openshift-dns/node-resolver-r77h8"
Jul 07 06:58:43.105979 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:43.105949    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 06:58:43.106079 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:43.105985    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 06:58:43.106079 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:43.106000    2579 projected.go:196] Error preparing data for projected volume kube-api-access-nv9bp for pod openshift-network-diagnostics/network-check-target-nfjzx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 06:58:43.106079 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:43.106073    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e917e03-b190-40ac-8c93-cd4d794fa127-kube-api-access-nv9bp podName:4e917e03-b190-40ac-8c93-cd4d794fa127 nodeName:}" failed. No retries permitted until 2026-07-07 06:58:43.606057873 +0000 UTC m=+3.149819360 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-nv9bp" (UniqueName: "kubernetes.io/projected/4e917e03-b190-40ac-8c93-cd4d794fa127-kube-api-access-nv9bp") pod "network-check-target-nfjzx" (UID: "4e917e03-b190-40ac-8c93-cd4d794fa127") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 06:58:43.108178 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.108157    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fgh7\" (UniqueName: \"kubernetes.io/projected/85470f35-7e80-42cf-8e5d-d83e5276ca5e-kube-api-access-5fgh7\") pod \"ovnkube-node-wvm5w\" (UID: \"85470f35-7e80-42cf-8e5d-d83e5276ca5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wvm5w"
Jul 07 06:58:43.108289 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.108274    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnwmv\" (UniqueName: \"kubernetes.io/projected/7c1bef18-765e-4290-8c9c-80c985fcb304-kube-api-access-vnwmv\") pod \"node-ca-fzzb4\" (UID: \"7c1bef18-765e-4290-8c9c-80c985fcb304\") " pod="openshift-image-registry/node-ca-fzzb4"
Jul 07 06:58:43.114780 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.114760    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 06:58:43.175115 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.175012    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2tp8w"
Jul 07 06:58:43.182043 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.182026    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rw27d"
Jul 07 06:58:43.185403 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:43.185381    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod858752de_a754_4997_9bda_cd13f51366d4.slice/crio-2ef2a81546a1473e10e601d4fdccc4b1bb2339909314e0262c15c4caeea9eaa4 WatchSource:0}: Error finding container 2ef2a81546a1473e10e601d4fdccc4b1bb2339909314e0262c15c4caeea9eaa4: Status 404 returned error can't find the container with id 2ef2a81546a1473e10e601d4fdccc4b1bb2339909314e0262c15c4caeea9eaa4
Jul 07 06:58:43.190845 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.190828    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-j5jdf"
Jul 07 06:58:43.191230 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:43.191202    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e1fa63c_3fd7_4883_9aad_5770f7c3b08f.slice/crio-ac5c3a9bb7f35e997c486c1385d9ef63796373f178da3b968047316c7087338c WatchSource:0}: Error finding container ac5c3a9bb7f35e997c486c1385d9ef63796373f178da3b968047316c7087338c: Status 404 returned error can't find the container with id ac5c3a9bb7f35e997c486c1385d9ef63796373f178da3b968047316c7087338c
Jul 07 06:58:43.196450 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.196427    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-llqp4"
Jul 07 06:58:43.202471 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.202450    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-z246s"
Jul 07 06:58:43.204413 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:43.204372    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76cc2585_6fae_464f_afcf_f629ed055e11.slice/crio-727c046b59457c666812b4e2c541253885402661427c7364438d882d649c53f3 WatchSource:0}: Error finding container 727c046b59457c666812b4e2c541253885402661427c7364438d882d649c53f3: Status 404 returned error can't find the container with id 727c046b59457c666812b4e2c541253885402661427c7364438d882d649c53f3
Jul 07 06:58:43.207221 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:43.207199    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99da3e49_fb00_4c1a_b0ba_cd1be152fb0c.slice/crio-276c6a743a6d2209f87594f597f6933c3d3e473b99a1a284f6c1fd4e8e2bff47 WatchSource:0}: Error finding container 276c6a743a6d2209f87594f597f6933c3d3e473b99a1a284f6c1fd4e8e2bff47: Status 404 returned error can't find the container with id 276c6a743a6d2209f87594f597f6933c3d3e473b99a1a284f6c1fd4e8e2bff47
Jul 07 06:58:43.208181 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.208161    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7r4r7"
Jul 07 06:58:43.212363 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:43.212344    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod338d4939_ce17_426b_a06b_bb8f40ebe2a1.slice/crio-73cd30fa08c303bd8056f41daadcde3476c73ea33340d9b01d4b77ba72ce340f WatchSource:0}: Error finding container 73cd30fa08c303bd8056f41daadcde3476c73ea33340d9b01d4b77ba72ce340f: Status 404 returned error can't find the container with id 73cd30fa08c303bd8056f41daadcde3476c73ea33340d9b01d4b77ba72ce340f
Jul 07 06:58:43.214387 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.214343    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-r77h8"
Jul 07 06:58:43.217840 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:43.217818    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod323baf0a_36cc_4f8c_b00b_ccc911176299.slice/crio-210e4fd53daafcf8a614d67d38410bd80878c9a5f7d5ad03035b9d5da60d740a WatchSource:0}: Error finding container 210e4fd53daafcf8a614d67d38410bd80878c9a5f7d5ad03035b9d5da60d740a: Status 404 returned error can't find the container with id 210e4fd53daafcf8a614d67d38410bd80878c9a5f7d5ad03035b9d5da60d740a
Jul 07 06:58:43.219637 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.219618    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-fzzb4"
Jul 07 06:58:43.224666 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.224645    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wvm5w"
Jul 07 06:58:43.224830 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:43.224807    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86018993_6534_4639_9254_a898f62948fe.slice/crio-a9b682d208481e181801c74019d1a7ee7e9728a801d20e8883c0a86cec20ab78 WatchSource:0}: Error finding container a9b682d208481e181801c74019d1a7ee7e9728a801d20e8883c0a86cec20ab78: Status 404 returned error can't find the container with id a9b682d208481e181801c74019d1a7ee7e9728a801d20e8883c0a86cec20ab78
Jul 07 06:58:43.230596 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:43.230573    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c1bef18_765e_4290_8c9c_80c985fcb304.slice/crio-1847db6b717688845385ad37b5244532cef8dfd8aa0d68b49e62ebb0db734799 WatchSource:0}: Error finding container 1847db6b717688845385ad37b5244532cef8dfd8aa0d68b49e62ebb0db734799: Status 404 returned error can't find the container with id 1847db6b717688845385ad37b5244532cef8dfd8aa0d68b49e62ebb0db734799
Jul 07 06:58:43.235979 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:58:43.235952    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85470f35_7e80_42cf_8e5d_d83e5276ca5e.slice/crio-17190df8fd2f2a2bf3dd9f4acfe09b1bf58de83a2bb0bb44966a695614f95a12 WatchSource:0}: Error finding container 17190df8fd2f2a2bf3dd9f4acfe09b1bf58de83a2bb0bb44966a695614f95a12: Status 404 returned error can't find the container with id 17190df8fd2f2a2bf3dd9f4acfe09b1bf58de83a2bb0bb44966a695614f95a12
Jul 07 06:58:43.495522 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.495409    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0dba90e8-7ad4-46cb-bc27-718b6f644f5d-metrics-certs\") pod \"network-metrics-daemon-s9stz\" (UID: \"0dba90e8-7ad4-46cb-bc27-718b6f644f5d\") " pod="openshift-multus/network-metrics-daemon-s9stz"
Jul 07 06:58:43.495695 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:43.495620    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 06:58:43.495695 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:43.495686    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0dba90e8-7ad4-46cb-bc27-718b6f644f5d-metrics-certs podName:0dba90e8-7ad4-46cb-bc27-718b6f644f5d nodeName:}" failed. No retries permitted until 2026-07-07 06:58:44.495668332 +0000 UTC m=+4.039429814 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0dba90e8-7ad4-46cb-bc27-718b6f644f5d-metrics-certs") pod "network-metrics-daemon-s9stz" (UID: "0dba90e8-7ad4-46cb-bc27-718b6f644f5d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 06:58:43.596888 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.596848    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/575f1248-fecc-48d9-9c6a-db334596c8b6-original-pull-secret\") pod \"global-pull-secret-syncer-gnlks\" (UID: \"575f1248-fecc-48d9-9c6a-db334596c8b6\") " pod="kube-system/global-pull-secret-syncer-gnlks"
Jul 07 06:58:43.597088 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:43.597056    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 06:58:43.597172 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:43.597129    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/575f1248-fecc-48d9-9c6a-db334596c8b6-original-pull-secret podName:575f1248-fecc-48d9-9c6a-db334596c8b6 nodeName:}" failed. No retries permitted until 2026-07-07 06:58:44.597109162 +0000 UTC m=+4.140870627 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/575f1248-fecc-48d9-9c6a-db334596c8b6-original-pull-secret") pod "global-pull-secret-syncer-gnlks" (UID: "575f1248-fecc-48d9-9c6a-db334596c8b6") : object "kube-system"/"original-pull-secret" not registered
Jul 07 06:58:43.697544 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.697489    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv9bp\" (UniqueName: \"kubernetes.io/projected/4e917e03-b190-40ac-8c93-cd4d794fa127-kube-api-access-nv9bp\") pod \"network-check-target-nfjzx\" (UID: \"4e917e03-b190-40ac-8c93-cd4d794fa127\") " pod="openshift-network-diagnostics/network-check-target-nfjzx"
Jul 07 06:58:43.697803 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:43.697748    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 06:58:43.697803 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:43.697772    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 06:58:43.697803 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:43.697788    2579 projected.go:196] Error preparing data for projected volume kube-api-access-nv9bp for pod openshift-network-diagnostics/network-check-target-nfjzx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 06:58:43.698061 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:43.697857    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e917e03-b190-40ac-8c93-cd4d794fa127-kube-api-access-nv9bp podName:4e917e03-b190-40ac-8c93-cd4d794fa127 nodeName:}" failed. No retries permitted until 2026-07-07 06:58:44.697837877 +0000 UTC m=+4.241599359 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-nv9bp" (UniqueName: "kubernetes.io/projected/4e917e03-b190-40ac-8c93-cd4d794fa127-kube-api-access-nv9bp") pod "network-check-target-nfjzx" (UID: "4e917e03-b190-40ac-8c93-cd4d794fa127") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 06:58:43.943276 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.943096    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 06:53:43 +0000 UTC" deadline="2027-12-12 12:59:35.254686329 +0000 UTC"
Jul 07 06:58:43.943276 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.943151    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12558h0m51.31153892s"
Jul 07 06:58:43.980594 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.980420    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wvm5w" event={"ID":"85470f35-7e80-42cf-8e5d-d83e5276ca5e","Type":"ContainerStarted","Data":"17190df8fd2f2a2bf3dd9f4acfe09b1bf58de83a2bb0bb44966a695614f95a12"}
Jul 07 06:58:43.994849 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:43.994811    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-r77h8" event={"ID":"86018993-6534-4639-9254-a898f62948fe","Type":"ContainerStarted","Data":"a9b682d208481e181801c74019d1a7ee7e9728a801d20e8883c0a86cec20ab78"}
Jul 07 06:58:44.000621 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:44.000584    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-llqp4" event={"ID":"99da3e49-fb00-4c1a-b0ba-cd1be152fb0c","Type":"ContainerStarted","Data":"276c6a743a6d2209f87594f597f6933c3d3e473b99a1a284f6c1fd4e8e2bff47"}
Jul 07 06:58:44.023861 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:44.023822    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j5jdf" event={"ID":"76cc2585-6fae-464f-afcf-f629ed055e11","Type":"ContainerStarted","Data":"727c046b59457c666812b4e2c541253885402661427c7364438d882d649c53f3"}
Jul 07 06:58:44.040630 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:44.040579    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rw27d" event={"ID":"5e1fa63c-3fd7-4883-9aad-5770f7c3b08f","Type":"ContainerStarted","Data":"ac5c3a9bb7f35e997c486c1385d9ef63796373f178da3b968047316c7087338c"}
Jul 07 06:58:44.050323 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:44.050290    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2tp8w" event={"ID":"858752de-a754-4997-9bda-cd13f51366d4","Type":"ContainerStarted","Data":"2ef2a81546a1473e10e601d4fdccc4b1bb2339909314e0262c15c4caeea9eaa4"}
Jul 07 06:58:44.069375 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:44.069142    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-fzzb4" event={"ID":"7c1bef18-765e-4290-8c9c-80c985fcb304","Type":"ContainerStarted","Data":"1847db6b717688845385ad37b5244532cef8dfd8aa0d68b49e62ebb0db734799"}
Jul 07 06:58:44.084822 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:44.084771    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7r4r7" event={"ID":"323baf0a-36cc-4f8c-b00b-ccc911176299","Type":"ContainerStarted","Data":"210e4fd53daafcf8a614d67d38410bd80878c9a5f7d5ad03035b9d5da60d740a"}
Jul 07 06:58:44.088853 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:44.088770    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-z246s" event={"ID":"338d4939-ce17-426b-a06b-bb8f40ebe2a1","Type":"ContainerStarted","Data":"73cd30fa08c303bd8056f41daadcde3476c73ea33340d9b01d4b77ba72ce340f"}
Jul 07 06:58:44.506429 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:44.506385    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0dba90e8-7ad4-46cb-bc27-718b6f644f5d-metrics-certs\") pod \"network-metrics-daemon-s9stz\" (UID: \"0dba90e8-7ad4-46cb-bc27-718b6f644f5d\") " pod="openshift-multus/network-metrics-daemon-s9stz"
Jul 07 06:58:44.506689 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:44.506668    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 06:58:44.506767 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:44.506750    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0dba90e8-7ad4-46cb-bc27-718b6f644f5d-metrics-certs podName:0dba90e8-7ad4-46cb-bc27-718b6f644f5d nodeName:}" failed. No retries permitted until 2026-07-07 06:58:46.506727452 +0000 UTC m=+6.050488932 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0dba90e8-7ad4-46cb-bc27-718b6f644f5d-metrics-certs") pod "network-metrics-daemon-s9stz" (UID: "0dba90e8-7ad4-46cb-bc27-718b6f644f5d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 06:58:44.607544 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:44.606862    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/575f1248-fecc-48d9-9c6a-db334596c8b6-original-pull-secret\") pod \"global-pull-secret-syncer-gnlks\" (UID: \"575f1248-fecc-48d9-9c6a-db334596c8b6\") " pod="kube-system/global-pull-secret-syncer-gnlks"
Jul 07 06:58:44.607544 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:44.607070    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 06:58:44.607544 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:44.607138    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/575f1248-fecc-48d9-9c6a-db334596c8b6-original-pull-secret podName:575f1248-fecc-48d9-9c6a-db334596c8b6 nodeName:}" failed. No retries permitted until 2026-07-07 06:58:46.607118627 +0000 UTC m=+6.150880109 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/575f1248-fecc-48d9-9c6a-db334596c8b6-original-pull-secret") pod "global-pull-secret-syncer-gnlks" (UID: "575f1248-fecc-48d9-9c6a-db334596c8b6") : object "kube-system"/"original-pull-secret" not registered
Jul 07 06:58:44.707343 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:44.707302    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv9bp\" (UniqueName: \"kubernetes.io/projected/4e917e03-b190-40ac-8c93-cd4d794fa127-kube-api-access-nv9bp\") pod \"network-check-target-nfjzx\" (UID: \"4e917e03-b190-40ac-8c93-cd4d794fa127\") " pod="openshift-network-diagnostics/network-check-target-nfjzx"
Jul 07 06:58:44.707549 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:44.707512    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 06:58:44.707549 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:44.707530    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 06:58:44.707549 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:44.707543    2579 projected.go:196] Error preparing data for projected volume kube-api-access-nv9bp for pod openshift-network-diagnostics/network-check-target-nfjzx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 06:58:44.707716 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:44.707605    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e917e03-b190-40ac-8c93-cd4d794fa127-kube-api-access-nv9bp podName:4e917e03-b190-40ac-8c93-cd4d794fa127 nodeName:}" failed. No retries permitted until 2026-07-07 06:58:46.707585526 +0000 UTC m=+6.251347002 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-nv9bp" (UniqueName: "kubernetes.io/projected/4e917e03-b190-40ac-8c93-cd4d794fa127-kube-api-access-nv9bp") pod "network-check-target-nfjzx" (UID: "4e917e03-b190-40ac-8c93-cd4d794fa127") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 06:58:44.944241 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:44.944141    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 06:53:43 +0000 UTC" deadline="2028-03-26 17:19:41.67275751 +0000 UTC"
Jul 07 06:58:44.944241 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:44.944190    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15082h20m56.728573803s"
Jul 07 06:58:44.961756 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:44.961723    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nfjzx"
Jul 07 06:58:44.961950 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:44.961866    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nfjzx" podUID="4e917e03-b190-40ac-8c93-cd4d794fa127"
Jul 07 06:58:44.962657 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:44.962508    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gnlks"
Jul 07 06:58:44.962657 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:44.962647    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gnlks" podUID="575f1248-fecc-48d9-9c6a-db334596c8b6"
Jul 07 06:58:44.962844 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:44.962828    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s9stz"
Jul 07 06:58:44.962972 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:44.962936    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s9stz" podUID="0dba90e8-7ad4-46cb-bc27-718b6f644f5d"
Jul 07 06:58:46.521869 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:46.521834    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0dba90e8-7ad4-46cb-bc27-718b6f644f5d-metrics-certs\") pod \"network-metrics-daemon-s9stz\" (UID: \"0dba90e8-7ad4-46cb-bc27-718b6f644f5d\") " pod="openshift-multus/network-metrics-daemon-s9stz"
Jul 07 06:58:46.522552 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:46.522073    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 06:58:46.522552 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:46.522136    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0dba90e8-7ad4-46cb-bc27-718b6f644f5d-metrics-certs podName:0dba90e8-7ad4-46cb-bc27-718b6f644f5d nodeName:}" failed. No retries permitted until 2026-07-07 06:58:50.522116764 +0000 UTC m=+10.065878241 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0dba90e8-7ad4-46cb-bc27-718b6f644f5d-metrics-certs") pod "network-metrics-daemon-s9stz" (UID: "0dba90e8-7ad4-46cb-bc27-718b6f644f5d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 06:58:46.622797 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:46.622756    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/575f1248-fecc-48d9-9c6a-db334596c8b6-original-pull-secret\") pod \"global-pull-secret-syncer-gnlks\" (UID: \"575f1248-fecc-48d9-9c6a-db334596c8b6\") " pod="kube-system/global-pull-secret-syncer-gnlks"
Jul 07 06:58:46.623001 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:46.622969    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 06:58:46.623094 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:46.623060    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/575f1248-fecc-48d9-9c6a-db334596c8b6-original-pull-secret podName:575f1248-fecc-48d9-9c6a-db334596c8b6 nodeName:}" failed. No retries permitted until 2026-07-07 06:58:50.623038478 +0000 UTC m=+10.166799952 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/575f1248-fecc-48d9-9c6a-db334596c8b6-original-pull-secret") pod "global-pull-secret-syncer-gnlks" (UID: "575f1248-fecc-48d9-9c6a-db334596c8b6") : object "kube-system"/"original-pull-secret" not registered
Jul 07 06:58:46.724285 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:46.724244    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv9bp\" (UniqueName: \"kubernetes.io/projected/4e917e03-b190-40ac-8c93-cd4d794fa127-kube-api-access-nv9bp\") pod \"network-check-target-nfjzx\" (UID: \"4e917e03-b190-40ac-8c93-cd4d794fa127\") " pod="openshift-network-diagnostics/network-check-target-nfjzx"
Jul 07 06:58:46.724488 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:46.724468    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 06:58:46.724569 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:46.724497    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 06:58:46.724569 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:46.724511    2579 projected.go:196] Error preparing data for projected volume kube-api-access-nv9bp for pod openshift-network-diagnostics/network-check-target-nfjzx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 06:58:46.724692 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:46.724576    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e917e03-b190-40ac-8c93-cd4d794fa127-kube-api-access-nv9bp podName:4e917e03-b190-40ac-8c93-cd4d794fa127 nodeName:}" failed. No retries permitted until 2026-07-07 06:58:50.724554785 +0000 UTC m=+10.268316270 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-nv9bp" (UniqueName: "kubernetes.io/projected/4e917e03-b190-40ac-8c93-cd4d794fa127-kube-api-access-nv9bp") pod "network-check-target-nfjzx" (UID: "4e917e03-b190-40ac-8c93-cd4d794fa127") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 06:58:46.963178 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:46.963031    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gnlks"
Jul 07 06:58:46.963178 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:46.963085    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nfjzx"
Jul 07 06:58:46.963405 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:46.963048    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s9stz"
Jul 07 06:58:46.963405 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:46.963266    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gnlks" podUID="575f1248-fecc-48d9-9c6a-db334596c8b6"
Jul 07 06:58:46.963516 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:46.963482    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s9stz" podUID="0dba90e8-7ad4-46cb-bc27-718b6f644f5d"
Jul 07 06:58:46.965947 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:46.963625    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nfjzx" podUID="4e917e03-b190-40ac-8c93-cd4d794fa127"
Jul 07 06:58:48.958260 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:48.958225    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s9stz"
Jul 07 06:58:48.958704 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:48.958267    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gnlks"
Jul 07 06:58:48.958704 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:48.958232    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nfjzx"
Jul 07 06:58:48.958704 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:48.958382    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s9stz" podUID="0dba90e8-7ad4-46cb-bc27-718b6f644f5d"
Jul 07 06:58:48.958704 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:48.958495    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gnlks" podUID="575f1248-fecc-48d9-9c6a-db334596c8b6"
Jul 07 06:58:48.958704 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:48.958612    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nfjzx" podUID="4e917e03-b190-40ac-8c93-cd4d794fa127"
Jul 07 06:58:50.559695 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:50.559585    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0dba90e8-7ad4-46cb-bc27-718b6f644f5d-metrics-certs\") pod \"network-metrics-daemon-s9stz\" (UID: \"0dba90e8-7ad4-46cb-bc27-718b6f644f5d\") " pod="openshift-multus/network-metrics-daemon-s9stz"
Jul 07 06:58:50.560251 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:50.559770    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 06:58:50.560251 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:50.559857    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0dba90e8-7ad4-46cb-bc27-718b6f644f5d-metrics-certs podName:0dba90e8-7ad4-46cb-bc27-718b6f644f5d nodeName:}" failed. No retries permitted until 2026-07-07 06:58:58.55983406 +0000 UTC m=+18.103595535 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0dba90e8-7ad4-46cb-bc27-718b6f644f5d-metrics-certs") pod "network-metrics-daemon-s9stz" (UID: "0dba90e8-7ad4-46cb-bc27-718b6f644f5d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 06:58:50.659973 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:50.659922    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/575f1248-fecc-48d9-9c6a-db334596c8b6-original-pull-secret\") pod \"global-pull-secret-syncer-gnlks\" (UID: \"575f1248-fecc-48d9-9c6a-db334596c8b6\") " pod="kube-system/global-pull-secret-syncer-gnlks"
Jul 07 06:58:50.660162 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:50.660108    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 06:58:50.660222 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:50.660177    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/575f1248-fecc-48d9-9c6a-db334596c8b6-original-pull-secret podName:575f1248-fecc-48d9-9c6a-db334596c8b6 nodeName:}" failed. No retries permitted until 2026-07-07 06:58:58.660162449 +0000 UTC m=+18.203923909 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/575f1248-fecc-48d9-9c6a-db334596c8b6-original-pull-secret") pod "global-pull-secret-syncer-gnlks" (UID: "575f1248-fecc-48d9-9c6a-db334596c8b6") : object "kube-system"/"original-pull-secret" not registered
Jul 07 06:58:50.761142 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:50.760791    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv9bp\" (UniqueName: \"kubernetes.io/projected/4e917e03-b190-40ac-8c93-cd4d794fa127-kube-api-access-nv9bp\") pod \"network-check-target-nfjzx\" (UID: \"4e917e03-b190-40ac-8c93-cd4d794fa127\") " pod="openshift-network-diagnostics/network-check-target-nfjzx"
Jul 07 06:58:50.761142 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:50.760992    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 06:58:50.761142 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:50.761015    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 06:58:50.761142 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:50.761052    2579 projected.go:196] Error preparing data for projected volume kube-api-access-nv9bp for pod openshift-network-diagnostics/network-check-target-nfjzx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 06:58:50.761142 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:50.761125    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e917e03-b190-40ac-8c93-cd4d794fa127-kube-api-access-nv9bp podName:4e917e03-b190-40ac-8c93-cd4d794fa127 nodeName:}" failed. No retries permitted until 2026-07-07 06:58:58.761102921 +0000 UTC m=+18.304864406 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-nv9bp" (UniqueName: "kubernetes.io/projected/4e917e03-b190-40ac-8c93-cd4d794fa127-kube-api-access-nv9bp") pod "network-check-target-nfjzx" (UID: "4e917e03-b190-40ac-8c93-cd4d794fa127") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 06:58:50.959715 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:50.959377    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s9stz"
Jul 07 06:58:50.959715 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:50.959411    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nfjzx"
Jul 07 06:58:50.959715 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:50.959518    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s9stz" podUID="0dba90e8-7ad4-46cb-bc27-718b6f644f5d"
Jul 07 06:58:50.959715 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:50.959540    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nfjzx" podUID="4e917e03-b190-40ac-8c93-cd4d794fa127"
Jul 07 06:58:50.959715 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:50.959582    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gnlks"
Jul 07 06:58:50.959715 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:50.959653    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gnlks" podUID="575f1248-fecc-48d9-9c6a-db334596c8b6"
Jul 07 06:58:52.958281 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:52.958237    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s9stz"
Jul 07 06:58:52.958743 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:52.958376    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s9stz" podUID="0dba90e8-7ad4-46cb-bc27-718b6f644f5d"
Jul 07 06:58:52.958743 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:52.958238    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nfjzx"
Jul 07 06:58:52.958743 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:52.958436    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gnlks"
Jul 07 06:58:52.958743 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:52.958541    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nfjzx" podUID="4e917e03-b190-40ac-8c93-cd4d794fa127"
Jul 07 06:58:52.958743 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:52.958632    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gnlks" podUID="575f1248-fecc-48d9-9c6a-db334596c8b6"
Jul 07 06:58:54.958916 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:54.958448    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s9stz"
Jul 07 06:58:54.958916 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:54.958593    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s9stz" podUID="0dba90e8-7ad4-46cb-bc27-718b6f644f5d"
Jul 07 06:58:54.958916 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:54.958659    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nfjzx"
Jul 07 06:58:54.958916 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:54.958757    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nfjzx" podUID="4e917e03-b190-40ac-8c93-cd4d794fa127"
Jul 07 06:58:54.958916 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:54.958793    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gnlks"
Jul 07 06:58:54.958916 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:54.958889    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gnlks" podUID="575f1248-fecc-48d9-9c6a-db334596c8b6"
Jul 07 06:58:56.137915 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:56.137681    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-97.ec2.internal" event={"ID":"a566f4272b430543cc74511348672fa6","Type":"ContainerStarted","Data":"6914c874075321506474184011da54e0654f1d7d66821a9cd94f6f48c4e8e3a0"}
Jul 07 06:58:56.155557 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:56.155493    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-97.ec2.internal" podStartSLOduration=14.155473677 podStartE2EDuration="14.155473677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 06:58:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 06:58:56.155134022 +0000 UTC m=+15.698895505" watchObservedRunningTime="2026-07-07 06:58:56.155473677 +0000 UTC m=+15.699235162"
Jul 07 06:58:56.958339 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:56.958226    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nfjzx"
Jul 07 06:58:56.958339 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:56.958278    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gnlks"
Jul 07 06:58:56.958339 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:56.958236    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s9stz"
Jul 07 06:58:56.958623 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:56.958363    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nfjzx" podUID="4e917e03-b190-40ac-8c93-cd4d794fa127"
Jul 07 06:58:56.958623 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:56.958566    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s9stz" podUID="0dba90e8-7ad4-46cb-bc27-718b6f644f5d"
Jul 07 06:58:56.958623 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:56.958607    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gnlks" podUID="575f1248-fecc-48d9-9c6a-db334596c8b6"
Jul 07 06:58:58.632334 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:58.632287    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0dba90e8-7ad4-46cb-bc27-718b6f644f5d-metrics-certs\") pod \"network-metrics-daemon-s9stz\" (UID: \"0dba90e8-7ad4-46cb-bc27-718b6f644f5d\") " pod="openshift-multus/network-metrics-daemon-s9stz"
Jul 07 06:58:58.632829 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:58.632496    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 06:58:58.632829 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:58.632578    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0dba90e8-7ad4-46cb-bc27-718b6f644f5d-metrics-certs podName:0dba90e8-7ad4-46cb-bc27-718b6f644f5d nodeName:}" failed. No retries permitted until 2026-07-07 06:59:14.632557285 +0000 UTC m=+34.176318750 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0dba90e8-7ad4-46cb-bc27-718b6f644f5d-metrics-certs") pod "network-metrics-daemon-s9stz" (UID: "0dba90e8-7ad4-46cb-bc27-718b6f644f5d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 06:58:58.733290 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:58.733250    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/575f1248-fecc-48d9-9c6a-db334596c8b6-original-pull-secret\") pod \"global-pull-secret-syncer-gnlks\" (UID: \"575f1248-fecc-48d9-9c6a-db334596c8b6\") " pod="kube-system/global-pull-secret-syncer-gnlks"
Jul 07 06:58:58.733447 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:58.733384    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 06:58:58.733447 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:58.733441    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/575f1248-fecc-48d9-9c6a-db334596c8b6-original-pull-secret podName:575f1248-fecc-48d9-9c6a-db334596c8b6 nodeName:}" failed. No retries permitted until 2026-07-07 06:59:14.73342855 +0000 UTC m=+34.277190009 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/575f1248-fecc-48d9-9c6a-db334596c8b6-original-pull-secret") pod "global-pull-secret-syncer-gnlks" (UID: "575f1248-fecc-48d9-9c6a-db334596c8b6") : object "kube-system"/"original-pull-secret" not registered
Jul 07 06:58:58.834073 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:58.834034    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv9bp\" (UniqueName: \"kubernetes.io/projected/4e917e03-b190-40ac-8c93-cd4d794fa127-kube-api-access-nv9bp\") pod \"network-check-target-nfjzx\" (UID: \"4e917e03-b190-40ac-8c93-cd4d794fa127\") " pod="openshift-network-diagnostics/network-check-target-nfjzx"
Jul 07 06:58:58.834225 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:58.834171    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 06:58:58.834225 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:58.834189    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 06:58:58.834225 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:58.834206    2579 projected.go:196] Error preparing data for projected volume kube-api-access-nv9bp for pod openshift-network-diagnostics/network-check-target-nfjzx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 06:58:58.834347 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:58.834256    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e917e03-b190-40ac-8c93-cd4d794fa127-kube-api-access-nv9bp podName:4e917e03-b190-40ac-8c93-cd4d794fa127 nodeName:}" failed. No retries permitted until 2026-07-07 06:59:14.83424277 +0000 UTC m=+34.378004229 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-nv9bp" (UniqueName: "kubernetes.io/projected/4e917e03-b190-40ac-8c93-cd4d794fa127-kube-api-access-nv9bp") pod "network-check-target-nfjzx" (UID: "4e917e03-b190-40ac-8c93-cd4d794fa127") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 06:58:58.958359 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:58.958279    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nfjzx"
Jul 07 06:58:58.958532 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:58.958279    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gnlks"
Jul 07 06:58:58.958532 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:58.958406    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nfjzx" podUID="4e917e03-b190-40ac-8c93-cd4d794fa127"
Jul 07 06:58:58.958633 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:58.958524    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gnlks" podUID="575f1248-fecc-48d9-9c6a-db334596c8b6"
Jul 07 06:58:58.958633 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:58:58.958536    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s9stz"
Jul 07 06:58:58.958759 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:58:58.958663    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s9stz" podUID="0dba90e8-7ad4-46cb-bc27-718b6f644f5d"
Jul 07 06:59:00.959763 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:00.959735    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nfjzx"
Jul 07 06:59:00.960242 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:00.959838    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nfjzx" podUID="4e917e03-b190-40ac-8c93-cd4d794fa127"
Jul 07 06:59:00.960242 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:00.959934    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gnlks"
Jul 07 06:59:00.960242 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:00.960063    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gnlks" podUID="575f1248-fecc-48d9-9c6a-db334596c8b6"
Jul 07 06:59:00.960242 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:00.960096    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s9stz"
Jul 07 06:59:00.960242 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:00.960235    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s9stz" podUID="0dba90e8-7ad4-46cb-bc27-718b6f644f5d"
Jul 07 06:59:02.958350 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:02.958309    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s9stz"
Jul 07 06:59:02.958834 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:02.958316    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nfjzx"
Jul 07 06:59:02.958834 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:02.958471    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s9stz" podUID="0dba90e8-7ad4-46cb-bc27-718b6f644f5d"
Jul 07 06:59:02.958834 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:02.958329    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gnlks"
Jul 07 06:59:02.958834 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:02.958518    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nfjzx" podUID="4e917e03-b190-40ac-8c93-cd4d794fa127"
Jul 07 06:59:02.958834 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:02.958603    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gnlks" podUID="575f1248-fecc-48d9-9c6a-db334596c8b6"
Jul 07 06:59:04.153657 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:04.153241    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-fzzb4" event={"ID":"7c1bef18-765e-4290-8c9c-80c985fcb304","Type":"ContainerStarted","Data":"5326d03f5702e8b5070e5d588d67537fb87ef58062279bac21793ab17d0a78c4"}
Jul 07 06:59:04.154574 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:04.154554    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7r4r7" event={"ID":"323baf0a-36cc-4f8c-b00b-ccc911176299","Type":"ContainerStarted","Data":"31add1a86fa3bc689f4dc9c74a030472782a8069863b6010a3ff8f49dbd7c802"}
Jul 07 06:59:04.155741 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:04.155711    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-z246s" event={"ID":"338d4939-ce17-426b-a06b-bb8f40ebe2a1","Type":"ContainerStarted","Data":"08741c0c761833caa514819397a31a96dcfcbb8baccce37a1159ba12136f5f80"}
Jul 07 06:59:04.156970 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:04.156934    2579 generic.go:356] "Generic (PLEG): container finished" podID="407910b7b4a6fcb93fc8f7af29beb88b" containerID="7cad2ccb00222d296c36ab69b04acf68a097d11ab1652fddf218307c543a2bb8" exitCode=0
Jul 07 06:59:04.157088 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:04.156982    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-97.ec2.internal" event={"ID":"407910b7b4a6fcb93fc8f7af29beb88b","Type":"ContainerDied","Data":"7cad2ccb00222d296c36ab69b04acf68a097d11ab1652fddf218307c543a2bb8"}
Jul 07 06:59:04.159879 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:04.159860    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wvm5w" event={"ID":"85470f35-7e80-42cf-8e5d-d83e5276ca5e","Type":"ContainerStarted","Data":"2e370faf5617eb200863b3238f6f14ae0e827aadf82237e0bb76d35939959f10"}
Jul 07 06:59:04.159993 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:04.159920    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wvm5w" event={"ID":"85470f35-7e80-42cf-8e5d-d83e5276ca5e","Type":"ContainerStarted","Data":"9c2b4d17c6798e988f2c08d4ce38466092c0059af792767abf233a5749ce4b38"}
Jul 07 06:59:04.159993 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:04.159950    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wvm5w" event={"ID":"85470f35-7e80-42cf-8e5d-d83e5276ca5e","Type":"ContainerStarted","Data":"17357e9400fa145f94d5a8ab30b9d27c8c32b60d73cf1b6ae140add3e7e3d7fa"}
Jul 07 06:59:04.160075 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:04.160001    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wvm5w" event={"ID":"85470f35-7e80-42cf-8e5d-d83e5276ca5e","Type":"ContainerStarted","Data":"8a6b7ce7a8ccdb1d77385022cb1123d928fb73ed520493f1bd6963c78f0eed82"}
Jul 07 06:59:04.160075 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:04.160028    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wvm5w" event={"ID":"85470f35-7e80-42cf-8e5d-d83e5276ca5e","Type":"ContainerStarted","Data":"b6884cfab02e3d6a19e87cabe328673c0f8a80579234e61b9bf182a7ddddedb6"}
Jul 07 06:59:04.160075 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:04.160062    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wvm5w" event={"ID":"85470f35-7e80-42cf-8e5d-d83e5276ca5e","Type":"ContainerStarted","Data":"e8de193310342eef768ba19e4c7fec9175fac848c89ecbf45bd63e73b511ad1f"}
Jul 07 06:59:04.161153 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:04.161136    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-r77h8" event={"ID":"86018993-6534-4639-9254-a898f62948fe","Type":"ContainerStarted","Data":"22f32094269cb81700bd45413ffeb4b850b2d4e3e5ed13d908491fdf457a46ec"}
Jul 07 06:59:04.162450 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:04.162428    2579 generic.go:356] "Generic (PLEG): container finished" podID="76cc2585-6fae-464f-afcf-f629ed055e11" containerID="b69f4a337ad842d876c929f9e4a9c53cc45810e2ca66bc5455fff1abcaa4bcdd" exitCode=0
Jul 07 06:59:04.162547 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:04.162501    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j5jdf" event={"ID":"76cc2585-6fae-464f-afcf-f629ed055e11","Type":"ContainerDied","Data":"b69f4a337ad842d876c929f9e4a9c53cc45810e2ca66bc5455fff1abcaa4bcdd"}
Jul 07 06:59:04.163976 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:04.163937    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rw27d" event={"ID":"5e1fa63c-3fd7-4883-9aad-5770f7c3b08f","Type":"ContainerStarted","Data":"a0837b7d33651cf398f28fa2aa2d8fee4fd2fa85448dc61a314b5415a4f93bda"}
Jul 07 06:59:04.165307 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:04.165274    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2tp8w" event={"ID":"858752de-a754-4997-9bda-cd13f51366d4","Type":"ContainerStarted","Data":"26081577c752273d71b674c0f99de1afca5aa5ea79906951ef7818e88f744757"}
Jul 07 06:59:04.171752 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:04.171707    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-fzzb4" podStartSLOduration=10.021090852 podStartE2EDuration="22.171690228s" totalImagesPullingTime="12.150599376s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 06:58:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 06:59:04.171045721 +0000 UTC m=+23.714807204" watchObservedRunningTime="2026-07-07 06:59:04.171690228 +0000 UTC m=+23.715451708"
Jul 07 06:59:04.237355 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:04.237303    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-2tp8w" podStartSLOduration=11.040984267 podStartE2EDuration="23.237287656s" totalImagesPullingTime="12.196303389s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 06:58:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 06:59:04.236818104 +0000 UTC m=+23.780579586" watchObservedRunningTime="2026-07-07 06:59:04.237287656 +0000 UTC m=+23.781049136"
Jul 07 06:59:04.258523 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:04.258480    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-rw27d" podStartSLOduration=3.090283746 podStartE2EDuration="23.25846601s" totalImagesPullingTime="20.168182264s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 06:58:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 06:59:04.257773474 +0000 UTC m=+23.801534952" watchObservedRunningTime="2026-07-07 06:59:04.25846601 +0000 UTC m=+23.802227490"
Jul 07 06:59:04.280351 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:04.280304    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-z246s" podStartSLOduration=10.111579127 podStartE2EDuration="22.28028969s" totalImagesPullingTime="12.168710563s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 06:58:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 06:59:04.280142172 +0000 UTC m=+23.823903655" watchObservedRunningTime="2026-07-07 06:59:04.28028969 +0000 UTC m=+23.824051161"
Jul 07 06:59:04.789697 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:04.789651    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-z246s"
Jul 07 06:59:04.790584 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:04.790555    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-z246s"
Jul 07 06:59:04.809257 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:04.809207    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-r77h8" podStartSLOduration=10.651042546 podStartE2EDuration="22.809189092s" totalImagesPullingTime="12.158146546s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 06:58:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 06:59:04.308487446 +0000 UTC m=+23.852248928" watchObservedRunningTime="2026-07-07 06:59:04.809189092 +0000 UTC m=+24.352950572"
Jul 07 06:59:04.958397 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:04.958310    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gnlks"
Jul 07 06:59:04.958397 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:04.958356    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nfjzx"
Jul 07 06:59:04.958618 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:04.958438    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gnlks" podUID="575f1248-fecc-48d9-9c6a-db334596c8b6"
Jul 07 06:59:04.958618 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:04.958497    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s9stz"
Jul 07 06:59:04.958618 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:04.958604    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nfjzx" podUID="4e917e03-b190-40ac-8c93-cd4d794fa127"
Jul 07 06:59:04.958780 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:04.958700    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s9stz" podUID="0dba90e8-7ad4-46cb-bc27-718b6f644f5d"
Jul 07 06:59:05.169151 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:05.169117    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-llqp4" event={"ID":"99da3e49-fb00-4c1a-b0ba-cd1be152fb0c","Type":"ContainerStarted","Data":"f55f90057b7295a7604bf8990a59be5c1bbdf17de8937989c5d2857f747e764e"}
Jul 07 06:59:05.170902 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:05.170877    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-97.ec2.internal" event={"ID":"407910b7b4a6fcb93fc8f7af29beb88b","Type":"ContainerStarted","Data":"6e4e1de7f334dda91a366d7302ee92d7ed09858eba8969f9e6efcd5e7695465a"}
Jul 07 06:59:05.178561 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:05.178455    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-z246s"
Jul 07 06:59:05.179764 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:05.179199    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-z246s"
Jul 07 06:59:05.198039 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:05.197940    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-llqp4" podStartSLOduration=3.122263463 podStartE2EDuration="23.197920382s" totalImagesPullingTime="20.075656919s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 06:58:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 06:59:05.197585965 +0000 UTC m=+24.741347458" watchObservedRunningTime="2026-07-07 06:59:05.197920382 +0000 UTC m=+24.741681866"
Jul 07 06:59:05.216246 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:05.216132    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-97.ec2.internal" podStartSLOduration=23.216112365 podStartE2EDuration="23.216112365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 06:58:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 06:59:05.215256643 +0000 UTC m=+24.759018125" watchObservedRunningTime="2026-07-07 06:59:05.216112365 +0000 UTC m=+24.759873848"
Jul 07 06:59:05.307018 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:05.306993    2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 07 06:59:05.959303 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:05.959147    2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-07T06:59:05.307013161Z","UUID":"d0e84d50-b717-426e-abfe-05cbba97e12f","Handler":null,"Name":"","Endpoint":""}
Jul 07 06:59:05.961702 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:05.961679    2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 07 06:59:05.961844 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:05.961724    2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 07 06:59:06.176244 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:06.176197    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7r4r7" event={"ID":"323baf0a-36cc-4f8c-b00b-ccc911176299","Type":"ContainerStarted","Data":"4c88bfe5884af5b110ed3051997cc71f1aa2369c3bd014931f576cede88b2332"}
Jul 07 06:59:06.180880 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:06.180849    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wvm5w" event={"ID":"85470f35-7e80-42cf-8e5d-d83e5276ca5e","Type":"ContainerStarted","Data":"3347ec21517df5faf98818121b2b2a128227673058e7352a834eedaab37e257c"}
Jul 07 06:59:06.958611 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:06.958517    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nfjzx"
Jul 07 06:59:06.958611 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:06.958562    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s9stz"
Jul 07 06:59:06.958825 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:06.958517    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gnlks"
Jul 07 06:59:06.958825 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:06.958636    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nfjzx" podUID="4e917e03-b190-40ac-8c93-cd4d794fa127"
Jul 07 06:59:06.958825 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:06.958701    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gnlks" podUID="575f1248-fecc-48d9-9c6a-db334596c8b6"
Jul 07 06:59:06.958825 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:06.958779    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s9stz" podUID="0dba90e8-7ad4-46cb-bc27-718b6f644f5d"
Jul 07 06:59:07.185218 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:07.185186    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7r4r7" event={"ID":"323baf0a-36cc-4f8c-b00b-ccc911176299","Type":"ContainerStarted","Data":"2a77f9ff20ac377fb1e9d812e434281e342419659f8f6dcae0ce17edd8ab158e"}
Jul 07 06:59:07.211699 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:07.211595    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7r4r7" podStartSLOduration=10.312153682 podStartE2EDuration="25.21157927s" totalImagesPullingTime="14.899425588s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 06:58:42 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 06:59:07.211462199 +0000 UTC m=+26.755223680" watchObservedRunningTime="2026-07-07 06:59:07.21157927 +0000 UTC m=+26.755340751"
Jul 07 06:59:08.959071 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:08.959031    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nfjzx"
Jul 07 06:59:08.959532 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:08.959031    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s9stz"
Jul 07 06:59:08.959532 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:08.959169    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gnlks"
Jul 07 06:59:08.959532 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:08.959166    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nfjzx" podUID="4e917e03-b190-40ac-8c93-cd4d794fa127"
Jul 07 06:59:08.959532 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:08.959322    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s9stz" podUID="0dba90e8-7ad4-46cb-bc27-718b6f644f5d"
Jul 07 06:59:08.959532 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:08.959406    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gnlks" podUID="575f1248-fecc-48d9-9c6a-db334596c8b6"
Jul 07 06:59:10.195314 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:10.194948    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wvm5w" event={"ID":"85470f35-7e80-42cf-8e5d-d83e5276ca5e","Type":"ContainerStarted","Data":"6655234d39fd8ce1e5d9b3deae3bcec2ee86cb854366790f3756970cbec18d05"}
Jul 07 06:59:10.196175 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:10.195323    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wvm5w"
Jul 07 06:59:10.196175 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:10.195375    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wvm5w"
Jul 07 06:59:10.196873 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:10.196844    2579 generic.go:356] "Generic (PLEG): container finished" podID="76cc2585-6fae-464f-afcf-f629ed055e11" containerID="7c18e8eb32a43a4fa02dd0a3ec859d731d2bc654232563ea8f71352180d9c66e" exitCode=0
Jul 07 06:59:10.197012 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:10.196890    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j5jdf" event={"ID":"76cc2585-6fae-464f-afcf-f629ed055e11","Type":"ContainerDied","Data":"7c18e8eb32a43a4fa02dd0a3ec859d731d2bc654232563ea8f71352180d9c66e"}
Jul 07 06:59:10.211073 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:10.211052    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wvm5w"
Jul 07 06:59:10.226968 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:10.226913    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-wvm5w" podStartSLOduration=8.088045246 podStartE2EDuration="28.226899217s" totalImagesPullingTime="20.138853971s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 06:58:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 06:59:10.22653136 +0000 UTC m=+29.770292842" watchObservedRunningTime="2026-07-07 06:59:10.226899217 +0000 UTC m=+29.770660698"
Jul 07 06:59:10.959282 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:10.959054    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s9stz"
Jul 07 06:59:10.959532 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:10.959084    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nfjzx"
Jul 07 06:59:10.959532 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:10.959125    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gnlks"
Jul 07 06:59:10.959532 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:10.959375    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s9stz" podUID="0dba90e8-7ad4-46cb-bc27-718b6f644f5d"
Jul 07 06:59:10.959532 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:10.959452    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gnlks" podUID="575f1248-fecc-48d9-9c6a-db334596c8b6"
Jul 07 06:59:10.959743 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:10.959536    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nfjzx" podUID="4e917e03-b190-40ac-8c93-cd4d794fa127"
Jul 07 06:59:11.199996 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:11.199968    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wvm5w"
Jul 07 06:59:11.214669 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:11.214598    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wvm5w"
Jul 07 06:59:11.988678 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:11.988654    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-nfjzx"]
Jul 07 06:59:11.988826 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:11.988769    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nfjzx"
Jul 07 06:59:11.988868 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:11.988845    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nfjzx" podUID="4e917e03-b190-40ac-8c93-cd4d794fa127"
Jul 07 06:59:11.995250 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:11.995223    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-s9stz"]
Jul 07 06:59:11.995381 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:11.995334    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s9stz"
Jul 07 06:59:11.995461 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:11.995435    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s9stz" podUID="0dba90e8-7ad4-46cb-bc27-718b6f644f5d"
Jul 07 06:59:11.995827 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:11.995806    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gnlks"]
Jul 07 06:59:11.995908 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:11.995897    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gnlks"
Jul 07 06:59:11.996004 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:11.995986    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gnlks" podUID="575f1248-fecc-48d9-9c6a-db334596c8b6"
Jul 07 06:59:12.204009 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:12.203977    2579 generic.go:356] "Generic (PLEG): container finished" podID="76cc2585-6fae-464f-afcf-f629ed055e11" containerID="f8ec8ec1a86d51410e3522b52a2f4a59c339cff155518dcbb487cbaf32caaa5a" exitCode=0
Jul 07 06:59:12.204463 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:12.204053    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j5jdf" event={"ID":"76cc2585-6fae-464f-afcf-f629ed055e11","Type":"ContainerDied","Data":"f8ec8ec1a86d51410e3522b52a2f4a59c339cff155518dcbb487cbaf32caaa5a"}
Jul 07 06:59:13.959150 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:13.959060    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gnlks"
Jul 07 06:59:13.959150 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:13.959085    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nfjzx"
Jul 07 06:59:13.959543 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:13.959070    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s9stz"
Jul 07 06:59:13.959543 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:13.959164    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gnlks" podUID="575f1248-fecc-48d9-9c6a-db334596c8b6"
Jul 07 06:59:13.959543 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:13.959248    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s9stz" podUID="0dba90e8-7ad4-46cb-bc27-718b6f644f5d"
Jul 07 06:59:13.959543 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:13.959299    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nfjzx" podUID="4e917e03-b190-40ac-8c93-cd4d794fa127"
Jul 07 06:59:14.210505 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:14.210427    2579 generic.go:356] "Generic (PLEG): container finished" podID="76cc2585-6fae-464f-afcf-f629ed055e11" containerID="6b6166fff036a0b18dcf52d5e3c70426ee3685ad9fe7b5b9633f1f058fd7e69a" exitCode=0
Jul 07 06:59:14.210505 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:14.210482    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j5jdf" event={"ID":"76cc2585-6fae-464f-afcf-f629ed055e11","Type":"ContainerDied","Data":"6b6166fff036a0b18dcf52d5e3c70426ee3685ad9fe7b5b9633f1f058fd7e69a"}
Jul 07 06:59:14.652296 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:14.652064    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0dba90e8-7ad4-46cb-bc27-718b6f644f5d-metrics-certs\") pod \"network-metrics-daemon-s9stz\" (UID: \"0dba90e8-7ad4-46cb-bc27-718b6f644f5d\") " pod="openshift-multus/network-metrics-daemon-s9stz"
Jul 07 06:59:14.652296 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:14.652221    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 06:59:14.652461 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:14.652304    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0dba90e8-7ad4-46cb-bc27-718b6f644f5d-metrics-certs podName:0dba90e8-7ad4-46cb-bc27-718b6f644f5d nodeName:}" failed. No retries permitted until 2026-07-07 06:59:46.652288557 +0000 UTC m=+66.196050016 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0dba90e8-7ad4-46cb-bc27-718b6f644f5d-metrics-certs") pod "network-metrics-daemon-s9stz" (UID: "0dba90e8-7ad4-46cb-bc27-718b6f644f5d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 06:59:14.752840 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:14.752807    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/575f1248-fecc-48d9-9c6a-db334596c8b6-original-pull-secret\") pod \"global-pull-secret-syncer-gnlks\" (UID: \"575f1248-fecc-48d9-9c6a-db334596c8b6\") " pod="kube-system/global-pull-secret-syncer-gnlks"
Jul 07 06:59:14.753013 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:14.752922    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 06:59:14.753013 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:14.752991    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/575f1248-fecc-48d9-9c6a-db334596c8b6-original-pull-secret podName:575f1248-fecc-48d9-9c6a-db334596c8b6 nodeName:}" failed. No retries permitted until 2026-07-07 06:59:46.752977284 +0000 UTC m=+66.296738742 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/575f1248-fecc-48d9-9c6a-db334596c8b6-original-pull-secret") pod "global-pull-secret-syncer-gnlks" (UID: "575f1248-fecc-48d9-9c6a-db334596c8b6") : object "kube-system"/"original-pull-secret" not registered
Jul 07 06:59:14.853462 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:14.853423    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv9bp\" (UniqueName: \"kubernetes.io/projected/4e917e03-b190-40ac-8c93-cd4d794fa127-kube-api-access-nv9bp\") pod \"network-check-target-nfjzx\" (UID: \"4e917e03-b190-40ac-8c93-cd4d794fa127\") " pod="openshift-network-diagnostics/network-check-target-nfjzx"
Jul 07 06:59:14.853619 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:14.853570    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 06:59:14.853619 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:14.853584    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 06:59:14.853619 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:14.853595    2579 projected.go:196] Error preparing data for projected volume kube-api-access-nv9bp for pod openshift-network-diagnostics/network-check-target-nfjzx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 06:59:14.853768 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:14.853644    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e917e03-b190-40ac-8c93-cd4d794fa127-kube-api-access-nv9bp podName:4e917e03-b190-40ac-8c93-cd4d794fa127 nodeName:}" failed. No retries permitted until 2026-07-07 06:59:46.853632328 +0000 UTC m=+66.397393791 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-nv9bp" (UniqueName: "kubernetes.io/projected/4e917e03-b190-40ac-8c93-cd4d794fa127-kube-api-access-nv9bp") pod "network-check-target-nfjzx" (UID: "4e917e03-b190-40ac-8c93-cd4d794fa127") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 06:59:15.958627 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:15.958596    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gnlks"
Jul 07 06:59:15.959058 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:15.958712    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nfjzx"
Jul 07 06:59:15.959058 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:15.958725    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gnlks" podUID="575f1248-fecc-48d9-9c6a-db334596c8b6"
Jul 07 06:59:15.959058 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:15.958738    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s9stz"
Jul 07 06:59:15.959058 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:15.958815    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nfjzx" podUID="4e917e03-b190-40ac-8c93-cd4d794fa127"
Jul 07 06:59:15.959058 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:15.958915    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s9stz" podUID="0dba90e8-7ad4-46cb-bc27-718b6f644f5d"
Jul 07 06:59:16.254137 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:16.254114    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-97.ec2.internal" event="NodeReady"
Jul 07 06:59:16.254305 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:16.254257    2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 07 06:59:16.291135 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:16.291104    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-65b55697fd-4z2qz"]
Jul 07 06:59:16.294622 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:16.294599    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65b55697fd-4z2qz"
Jul 07 06:59:16.297388 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:16.297362    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 07 06:59:16.297506 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:16.297395    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 07 06:59:16.297696 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:16.297678    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 07 06:59:16.297788 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:16.297680    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-7j9bg\""
Jul 07 06:59:16.304337 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:16.304316    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 07 06:59:16.307088 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:16.307025    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-hpwxr"]
Jul 07 06:59:16.311229 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:16.311209    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-65b55697fd-4z2qz"]
Jul 07 06:59:16.311381 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:16.311331    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hpwxr"
Jul 07 06:59:16.311697 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:16.311673    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-jhkfq"]
Jul 07 06:59:16.313915 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:16.313895    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vfjqn\""
Jul 07 06:59:16.314031 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:16.313932    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 07 06:59:16.314265 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:16.314205    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 07 06:59:16.315828 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:16.315810    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jhkfq"
Jul 07 06:59:16.322832 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:16.318867    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 07 06:59:16.322832 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:16.318891    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 07 06:59:16.322832 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:16.319145    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-7mlnr\""
Jul 07 06:59:16.322832 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:16.319342    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 07 06:59:16.325980 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:16.325545    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hpwxr"]
Jul 07 06:59:16.326746 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:16.326725    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jhkfq"]
Jul 07 06:59:16.466120 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:16.466090    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpvws\" (UniqueName: \"kubernetes.io/projected/2915b9bd-a9ae-4d0f-a362-43b2ce659350-kube-api-access-bpvws\") pod \"ingress-canary-jhkfq\" (UID: \"2915b9bd-a9ae-4d0f-a362-43b2ce659350\") " pod="openshift-ingress-canary/ingress-canary-jhkfq"
Jul 07 06:59:16.466323 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:16.466124    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4fb6f6e3-b570-408c-aaaa-a77d5bea0b24-metrics-tls\") pod \"dns-default-hpwxr\" (UID: \"4fb6f6e3-b570-408c-aaaa-a77d5bea0b24\") " pod="openshift-dns/dns-default-hpwxr"
Jul 07 06:59:16.466323 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:16.466153    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4fb6f6e3-b570-408c-aaaa-a77d5bea0b24-config-volume\") pod \"dns-default-hpwxr\" (UID: \"4fb6f6e3-b570-408c-aaaa-a77d5bea0b24\") " pod="openshift-dns/dns-default-hpwxr"
Jul 07 06:59:16.466323 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:16.466222    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fb6f6e3-b570-408c-aaaa-a77d5bea0b24-tmp-dir\") pod \"dns-default-hpwxr\" (UID: \"4fb6f6e3-b570-408c-aaaa-a77d5bea0b24\") " pod="openshift-dns/dns-default-hpwxr"
Jul 07 06:59:16.466323 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:16.466289    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2915b9bd-a9ae-4d0f-a362-43b2ce659350-cert\") pod \"ingress-canary-jhkfq\" (UID: \"2915b9bd-a9ae-4d0f-a362-43b2ce659350\") " pod="openshift-ingress-canary/ingress-canary-jhkfq"
Jul 07 06:59:16.466489 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:16.466328    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8b86dea1-145b-4f19-b0f8-ae53d29815bb-registry-certificates\") pod \"image-registry-65b55697fd-4z2qz\" (UID: \"8b86dea1-145b-4f19-b0f8-ae53d29815bb\") " pod="openshift-image-registry/image-registry-65b55697fd-4z2qz"
Jul 07 06:59:16.466489 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:16.466361    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8b86dea1-145b-4f19-b0f8-ae53d29815bb-installation-pull-secrets\") pod \"image-registry-65b55697fd-4z2qz\" (UID: \"8b86dea1-145b-4f19-b0f8-ae53d29815bb\") " pod="openshift-image-registry/image-registry-65b55697fd-4z2qz"
Jul 07 06:59:16.466489 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:16.466409    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8b86dea1-145b-4f19-b0f8-ae53d29815bb-trusted-ca\") pod \"image-registry-65b55697fd-4z2qz\" (UID: \"8b86dea1-145b-4f19-b0f8-ae53d29815bb\") " pod="openshift-image-registry/image-registry-65b55697fd-4z2qz"
Jul 07 06:59:16.466489 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:16.466467    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8b86dea1-145b-4f19-b0f8-ae53d29815bb-registry-tls\") pod \"image-registry-65b55697fd-4z2qz\" (UID: \"8b86dea1-145b-4f19-b0f8-ae53d29815bb\") " pod="openshift-image-registry/image-registry-65b55697fd-4z2qz"
Jul 07 06:59:16.466636 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:16.466539    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qnbb\" (UniqueName: \"kubernetes.io/projected/4fb6f6e3-b570-408c-aaaa-a77d5bea0b24-kube-api-access-9qnbb\") pod \"dns-default-hpwxr\" (UID: \"4fb6f6e3-b570-408c-aaaa-a77d5bea0b24\") " pod="openshift-dns/dns-default-hpwxr"
Jul 07 06:59:16.466636 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:16.466565    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mlhx\" (UniqueName: \"kubernetes.io/projected/8b86dea1-145b-4f19-b0f8-ae53d29815bb-kube-api-access-8mlhx\") pod \"image-registry-65b55697fd-4z2qz\" (UID: \"8b86dea1-145b-4f19-b0f8-ae53d29815bb\") " pod="openshift-image-registry/image-registry-65b55697fd-4z2qz"
Jul 07 06:59:16.466636 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:16.466593    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8b86dea1-145b-4f19-b0f8-ae53d29815bb-ca-trust-extracted\") pod \"image-registry-65b55697fd-4z2qz\" (UID: \"8b86dea1-145b-4f19-b0f8-ae53d29815bb\") " pod="openshift-image-registry/image-registry-65b55697fd-4z2qz"
Jul 07 06:59:16.466636 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:16.466614    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8b86dea1-145b-4f19-b0f8-ae53d29815bb-bound-sa-token\") pod \"image-registry-65b55697fd-4z2qz\" (UID: \"8b86dea1-145b-4f19-b0f8-ae53d29815bb\") " pod="openshift-image-registry/image-registry-65b55697fd-4z2qz"
Jul 07 06:59:16.466790 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:16.466687    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8b86dea1-145b-4f19-b0f8-ae53d29815bb-image-registry-private-configuration\") pod \"image-registry-65b55697fd-4z2qz\" (UID: \"8b86dea1-145b-4f19-b0f8-ae53d29815bb\") " pod="openshift-image-registry/image-registry-65b55697fd-4z2qz"
Jul 07 06:59:16.567971 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:16.567861    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qnbb\" (UniqueName: \"kubernetes.io/projected/4fb6f6e3-b570-408c-aaaa-a77d5bea0b24-kube-api-access-9qnbb\") pod \"dns-default-hpwxr\" (UID: \"4fb6f6e3-b570-408c-aaaa-a77d5bea0b24\") " pod="openshift-dns/dns-default-hpwxr"
Jul 07 06:59:16.567971 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:16.567910    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mlhx\" (UniqueName: \"kubernetes.io/projected/8b86dea1-145b-4f19-b0f8-ae53d29815bb-kube-api-access-8mlhx\") pod \"image-registry-65b55697fd-4z2qz\" (UID: \"8b86dea1-145b-4f19-b0f8-ae53d29815bb\") " pod="openshift-image-registry/image-registry-65b55697fd-4z2qz"
Jul 07 06:59:16.568205 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:16.568045    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8b86dea1-145b-4f19-b0f8-ae53d29815bb-ca-trust-extracted\") pod \"image-registry-65b55697fd-4z2qz\" (UID: \"8b86dea1-145b-4f19-b0f8-ae53d29815bb\") " pod="openshift-image-registry/image-registry-65b55697fd-4z2qz"
Jul 07 06:59:16.568205 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:16.568093    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8b86dea1-145b-4f19-b0f8-ae53d29815bb-bound-sa-token\") pod \"image-registry-65b55697fd-4z2qz\" (UID: \"8b86dea1-145b-4f19-b0f8-ae53d29815bb\") " pod="openshift-image-registry/image-registry-65b55697fd-4z2qz"
Jul 07 06:59:16.568205 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:16.568140    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8b86dea1-145b-4f19-b0f8-ae53d29815bb-image-registry-private-configuration\") pod \"image-registry-65b55697fd-4z2qz\" (UID: \"8b86dea1-145b-4f19-b0f8-ae53d29815bb\") " pod="openshift-image-registry/image-registry-65b55697fd-4z2qz"
Jul 07 06:59:16.568205 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:16.568179    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpvws\" (UniqueName: \"kubernetes.io/projected/2915b9bd-a9ae-4d0f-a362-43b2ce659350-kube-api-access-bpvws\") pod \"ingress-canary-jhkfq\" (UID: \"2915b9bd-a9ae-4d0f-a362-43b2ce659350\") " pod="openshift-ingress-canary/ingress-canary-jhkfq"
Jul 07 06:59:16.568410 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:16.568221    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4fb6f6e3-b570-408c-aaaa-a77d5bea0b24-metrics-tls\") pod \"dns-default-hpwxr\" (UID: \"4fb6f6e3-b570-408c-aaaa-a77d5bea0b24\") " pod="openshift-dns/dns-default-hpwxr"
Jul 07 06:59:16.568410 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:16.568252    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4fb6f6e3-b570-408c-aaaa-a77d5bea0b24-config-volume\") pod \"dns-default-hpwxr\" (UID: \"4fb6f6e3-b570-408c-aaaa-a77d5bea0b24\") " pod="openshift-dns/dns-default-hpwxr"
Jul 07 06:59:16.568410 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:16.568274    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fb6f6e3-b570-408c-aaaa-a77d5bea0b24-tmp-dir\") pod \"dns-default-hpwxr\" (UID: \"4fb6f6e3-b570-408c-aaaa-a77d5bea0b24\") " pod="openshift-dns/dns-default-hpwxr"
Jul 07 06:59:16.568410 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:16.568311    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2915b9bd-a9ae-4d0f-a362-43b2ce659350-cert\") pod \"ingress-canary-jhkfq\" (UID: \"2915b9bd-a9ae-4d0f-a362-43b2ce659350\") " pod="openshift-ingress-canary/ingress-canary-jhkfq"
Jul 07 06:59:16.568410 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:16.568336    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8b86dea1-145b-4f19-b0f8-ae53d29815bb-registry-certificates\") pod \"image-registry-65b55697fd-4z2qz\" (UID: \"8b86dea1-145b-4f19-b0f8-ae53d29815bb\") " pod="openshift-image-registry/image-registry-65b55697fd-4z2qz"
Jul 07 06:59:16.568410 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:16.568372    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8b86dea1-145b-4f19-b0f8-ae53d29815bb-installation-pull-secrets\") pod \"image-registry-65b55697fd-4z2qz\" (UID: \"8b86dea1-145b-4f19-b0f8-ae53d29815bb\") " pod="openshift-image-registry/image-registry-65b55697fd-4z2qz"
Jul 07 06:59:16.568410 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:16.568404    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8b86dea1-145b-4f19-b0f8-ae53d29815bb-trusted-ca\") pod \"image-registry-65b55697fd-4z2qz\" (UID: \"8b86dea1-145b-4f19-b0f8-ae53d29815bb\") " pod="openshift-image-registry/image-registry-65b55697fd-4z2qz"
Jul 07 06:59:16.568720 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:16.568418    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8b86dea1-145b-4f19-b0f8-ae53d29815bb-ca-trust-extracted\") pod \"image-registry-65b55697fd-4z2qz\" (UID: \"8b86dea1-145b-4f19-b0f8-ae53d29815bb\") " pod="openshift-image-registry/image-registry-65b55697fd-4z2qz"
Jul 07 06:59:16.568720 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:16.568506    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8b86dea1-145b-4f19-b0f8-ae53d29815bb-registry-tls\") pod \"image-registry-65b55697fd-4z2qz\" (UID: \"8b86dea1-145b-4f19-b0f8-ae53d29815bb\") " pod="openshift-image-registry/image-registry-65b55697fd-4z2qz"
Jul 07 06:59:16.568720 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:16.568646    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 07 06:59:16.568720 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:16.568661    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65b55697fd-4z2qz: secret "image-registry-tls" not found
Jul 07 06:59:16.568720 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:16.568716    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b86dea1-145b-4f19-b0f8-ae53d29815bb-registry-tls podName:8b86dea1-145b-4f19-b0f8-ae53d29815bb nodeName:}" failed. No retries permitted until 2026-07-07 06:59:17.068697636 +0000 UTC m=+36.612459101 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8b86dea1-145b-4f19-b0f8-ae53d29815bb-registry-tls") pod "image-registry-65b55697fd-4z2qz" (UID: "8b86dea1-145b-4f19-b0f8-ae53d29815bb") : secret "image-registry-tls" not found
Jul 07 06:59:16.569007 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:16.568946    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8b86dea1-145b-4f19-b0f8-ae53d29815bb-registry-certificates\") pod \"image-registry-65b55697fd-4z2qz\" (UID: \"8b86dea1-145b-4f19-b0f8-ae53d29815bb\") " pod="openshift-image-registry/image-registry-65b55697fd-4z2qz"
Jul 07 06:59:16.569068 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:16.569025    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4fb6f6e3-b570-408c-aaaa-a77d5bea0b24-config-volume\") pod \"dns-default-hpwxr\" (UID: \"4fb6f6e3-b570-408c-aaaa-a77d5bea0b24\") " pod="openshift-dns/dns-default-hpwxr"
Jul 07 06:59:16.569135 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:16.569112    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fb6f6e3-b570-408c-aaaa-a77d5bea0b24-tmp-dir\") pod \"dns-default-hpwxr\" (UID: \"4fb6f6e3-b570-408c-aaaa-a77d5bea0b24\") " pod="openshift-dns/dns-default-hpwxr"
Jul 07 06:59:16.569182 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:16.569123    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 06:59:16.569238 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:16.569196    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fb6f6e3-b570-408c-aaaa-a77d5bea0b24-metrics-tls podName:4fb6f6e3-b570-408c-aaaa-a77d5bea0b24 nodeName:}" failed. No retries permitted until 2026-07-07 06:59:17.069177267 +0000 UTC m=+36.612938742 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4fb6f6e3-b570-408c-aaaa-a77d5bea0b24-metrics-tls") pod "dns-default-hpwxr" (UID: "4fb6f6e3-b570-408c-aaaa-a77d5bea0b24") : secret "dns-default-metrics-tls" not found
Jul 07 06:59:16.569648 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:16.569625    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8b86dea1-145b-4f19-b0f8-ae53d29815bb-trusted-ca\") pod \"image-registry-65b55697fd-4z2qz\" (UID: \"8b86dea1-145b-4f19-b0f8-ae53d29815bb\") " pod="openshift-image-registry/image-registry-65b55697fd-4z2qz"
Jul 07 06:59:16.573264 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:16.573241    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8b86dea1-145b-4f19-b0f8-ae53d29815bb-image-registry-private-configuration\") pod \"image-registry-65b55697fd-4z2qz\" (UID: \"8b86dea1-145b-4f19-b0f8-ae53d29815bb\") " pod="openshift-image-registry/image-registry-65b55697fd-4z2qz"
Jul 07 06:59:16.573264 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:16.573249    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2915b9bd-a9ae-4d0f-a362-43b2ce659350-cert\") pod \"ingress-canary-jhkfq\" (UID: \"2915b9bd-a9ae-4d0f-a362-43b2ce659350\") " pod="openshift-ingress-canary/ingress-canary-jhkfq"
Jul 07 06:59:16.573434 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:16.573415    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8b86dea1-145b-4f19-b0f8-ae53d29815bb-installation-pull-secrets\") pod \"image-registry-65b55697fd-4z2qz\" (UID: \"8b86dea1-145b-4f19-b0f8-ae53d29815bb\") " pod="openshift-image-registry/image-registry-65b55697fd-4z2qz"
Jul 07 06:59:16.579509 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:16.579481    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpvws\" (UniqueName: \"kubernetes.io/projected/2915b9bd-a9ae-4d0f-a362-43b2ce659350-kube-api-access-bpvws\") pod \"ingress-canary-jhkfq\" (UID: \"2915b9bd-a9ae-4d0f-a362-43b2ce659350\") " pod="openshift-ingress-canary/ingress-canary-jhkfq"
Jul 07 06:59:16.579642 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:16.579518    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8b86dea1-145b-4f19-b0f8-ae53d29815bb-bound-sa-token\") pod \"image-registry-65b55697fd-4z2qz\" (UID: \"8b86dea1-145b-4f19-b0f8-ae53d29815bb\") " pod="openshift-image-registry/image-registry-65b55697fd-4z2qz"
Jul 07 06:59:16.579729 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:16.579707    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mlhx\" (UniqueName: \"kubernetes.io/projected/8b86dea1-145b-4f19-b0f8-ae53d29815bb-kube-api-access-8mlhx\") pod \"image-registry-65b55697fd-4z2qz\" (UID: \"8b86dea1-145b-4f19-b0f8-ae53d29815bb\") " pod="openshift-image-registry/image-registry-65b55697fd-4z2qz"
Jul 07 06:59:16.580141 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:16.580104    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qnbb\" (UniqueName: \"kubernetes.io/projected/4fb6f6e3-b570-408c-aaaa-a77d5bea0b24-kube-api-access-9qnbb\") pod \"dns-default-hpwxr\" (UID: \"4fb6f6e3-b570-408c-aaaa-a77d5bea0b24\") " pod="openshift-dns/dns-default-hpwxr"
Jul 07 06:59:16.630769 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:16.630717    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jhkfq"
Jul 07 06:59:16.788852 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:16.788819    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jhkfq"]
Jul 07 06:59:16.795309 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:59:16.795264    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2915b9bd_a9ae_4d0f_a362_43b2ce659350.slice/crio-2d848d1a776a9964df4077299ce8eeed66c2381168614298423221602b5efceb WatchSource:0}: Error finding container 2d848d1a776a9964df4077299ce8eeed66c2381168614298423221602b5efceb: Status 404 returned error can't find the container with id 2d848d1a776a9964df4077299ce8eeed66c2381168614298423221602b5efceb
Jul 07 06:59:17.073656 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:17.073620    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8b86dea1-145b-4f19-b0f8-ae53d29815bb-registry-tls\") pod \"image-registry-65b55697fd-4z2qz\" (UID: \"8b86dea1-145b-4f19-b0f8-ae53d29815bb\") " pod="openshift-image-registry/image-registry-65b55697fd-4z2qz"
Jul 07 06:59:17.074394 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:17.073685    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4fb6f6e3-b570-408c-aaaa-a77d5bea0b24-metrics-tls\") pod \"dns-default-hpwxr\" (UID: \"4fb6f6e3-b570-408c-aaaa-a77d5bea0b24\") " pod="openshift-dns/dns-default-hpwxr"
Jul 07 06:59:17.074394 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:17.073776    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 06:59:17.074394 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:17.073781    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 07 06:59:17.074394 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:17.073798    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65b55697fd-4z2qz: secret "image-registry-tls" not found
Jul 07 06:59:17.074394 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:17.073827    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fb6f6e3-b570-408c-aaaa-a77d5bea0b24-metrics-tls podName:4fb6f6e3-b570-408c-aaaa-a77d5bea0b24 nodeName:}" failed. No retries permitted until 2026-07-07 06:59:18.073814078 +0000 UTC m=+37.617575536 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4fb6f6e3-b570-408c-aaaa-a77d5bea0b24-metrics-tls") pod "dns-default-hpwxr" (UID: "4fb6f6e3-b570-408c-aaaa-a77d5bea0b24") : secret "dns-default-metrics-tls" not found
Jul 07 06:59:17.074394 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:17.073843    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b86dea1-145b-4f19-b0f8-ae53d29815bb-registry-tls podName:8b86dea1-145b-4f19-b0f8-ae53d29815bb nodeName:}" failed. No retries permitted until 2026-07-07 06:59:18.073835176 +0000 UTC m=+37.617596635 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8b86dea1-145b-4f19-b0f8-ae53d29815bb-registry-tls") pod "image-registry-65b55697fd-4z2qz" (UID: "8b86dea1-145b-4f19-b0f8-ae53d29815bb") : secret "image-registry-tls" not found
Jul 07 06:59:17.219538 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:17.219277    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jhkfq" event={"ID":"2915b9bd-a9ae-4d0f-a362-43b2ce659350","Type":"ContainerStarted","Data":"2d848d1a776a9964df4077299ce8eeed66c2381168614298423221602b5efceb"}
Jul 07 06:59:17.958231 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:17.958177    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nfjzx"
Jul 07 06:59:17.958231 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:17.958236    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gnlks"
Jul 07 06:59:17.958479 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:17.958308    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s9stz"
Jul 07 06:59:17.962652 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:17.962590    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 07 06:59:17.962652 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:17.962606    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 07 06:59:17.962652 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:17.962641    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 07 06:59:17.962923 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:17.962656    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2lkzt\""
Jul 07 06:59:17.962923 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:17.962700    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-59nzl\""
Jul 07 06:59:17.962923 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:17.962589    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 07 06:59:18.082081 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:18.082041    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8b86dea1-145b-4f19-b0f8-ae53d29815bb-registry-tls\") pod \"image-registry-65b55697fd-4z2qz\" (UID: \"8b86dea1-145b-4f19-b0f8-ae53d29815bb\") " pod="openshift-image-registry/image-registry-65b55697fd-4z2qz"
Jul 07 06:59:18.082523 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:18.082134    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4fb6f6e3-b570-408c-aaaa-a77d5bea0b24-metrics-tls\") pod \"dns-default-hpwxr\" (UID: \"4fb6f6e3-b570-408c-aaaa-a77d5bea0b24\") " pod="openshift-dns/dns-default-hpwxr"
Jul 07 06:59:18.082523 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:18.082212    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 07 06:59:18.082523 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:18.082230    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65b55697fd-4z2qz: secret "image-registry-tls" not found
Jul 07 06:59:18.082523 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:18.082248    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 06:59:18.082523 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:18.082291    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b86dea1-145b-4f19-b0f8-ae53d29815bb-registry-tls podName:8b86dea1-145b-4f19-b0f8-ae53d29815bb nodeName:}" failed. No retries permitted until 2026-07-07 06:59:20.082274188 +0000 UTC m=+39.626035665 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8b86dea1-145b-4f19-b0f8-ae53d29815bb-registry-tls") pod "image-registry-65b55697fd-4z2qz" (UID: "8b86dea1-145b-4f19-b0f8-ae53d29815bb") : secret "image-registry-tls" not found
Jul 07 06:59:18.082523 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:18.082305    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fb6f6e3-b570-408c-aaaa-a77d5bea0b24-metrics-tls podName:4fb6f6e3-b570-408c-aaaa-a77d5bea0b24 nodeName:}" failed. No retries permitted until 2026-07-07 06:59:20.082299263 +0000 UTC m=+39.626060723 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4fb6f6e3-b570-408c-aaaa-a77d5bea0b24-metrics-tls") pod "dns-default-hpwxr" (UID: "4fb6f6e3-b570-408c-aaaa-a77d5bea0b24") : secret "dns-default-metrics-tls" not found
Jul 07 06:59:19.227267 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:19.227236    2579 generic.go:356] "Generic (PLEG): container finished" podID="76cc2585-6fae-464f-afcf-f629ed055e11" containerID="cebb1181a8f2f560d6942a99ea068138592fe5aaad41eea6a9ca05e111796da4" exitCode=0
Jul 07 06:59:19.227736 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:19.227287    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j5jdf" event={"ID":"76cc2585-6fae-464f-afcf-f629ed055e11","Type":"ContainerDied","Data":"cebb1181a8f2f560d6942a99ea068138592fe5aaad41eea6a9ca05e111796da4"}
Jul 07 06:59:20.098326 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:20.098280    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8b86dea1-145b-4f19-b0f8-ae53d29815bb-registry-tls\") pod \"image-registry-65b55697fd-4z2qz\" (UID: \"8b86dea1-145b-4f19-b0f8-ae53d29815bb\") " pod="openshift-image-registry/image-registry-65b55697fd-4z2qz"
Jul 07 06:59:20.098495 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:20.098403    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4fb6f6e3-b570-408c-aaaa-a77d5bea0b24-metrics-tls\") pod \"dns-default-hpwxr\" (UID: \"4fb6f6e3-b570-408c-aaaa-a77d5bea0b24\") " pod="openshift-dns/dns-default-hpwxr"
Jul 07 06:59:20.098495 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:20.098443    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 07 06:59:20.098495 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:20.098466    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65b55697fd-4z2qz: secret "image-registry-tls" not found
Jul 07 06:59:20.098656 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:20.098519    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 06:59:20.098656 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:20.098537    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b86dea1-145b-4f19-b0f8-ae53d29815bb-registry-tls podName:8b86dea1-145b-4f19-b0f8-ae53d29815bb nodeName:}" failed. No retries permitted until 2026-07-07 06:59:24.098515279 +0000 UTC m=+43.642276770 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8b86dea1-145b-4f19-b0f8-ae53d29815bb-registry-tls") pod "image-registry-65b55697fd-4z2qz" (UID: "8b86dea1-145b-4f19-b0f8-ae53d29815bb") : secret "image-registry-tls" not found
Jul 07 06:59:20.098656 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:20.098577    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fb6f6e3-b570-408c-aaaa-a77d5bea0b24-metrics-tls podName:4fb6f6e3-b570-408c-aaaa-a77d5bea0b24 nodeName:}" failed. No retries permitted until 2026-07-07 06:59:24.098561317 +0000 UTC m=+43.642322797 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4fb6f6e3-b570-408c-aaaa-a77d5bea0b24-metrics-tls") pod "dns-default-hpwxr" (UID: "4fb6f6e3-b570-408c-aaaa-a77d5bea0b24") : secret "dns-default-metrics-tls" not found
Jul 07 06:59:20.233092 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:20.233056    2579 generic.go:356] "Generic (PLEG): container finished" podID="76cc2585-6fae-464f-afcf-f629ed055e11" containerID="7435667faf4c447b07139a8057de039910a89f935f9c570ec49be9f88f51e526" exitCode=0
Jul 07 06:59:20.233526 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:20.233120    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j5jdf" event={"ID":"76cc2585-6fae-464f-afcf-f629ed055e11","Type":"ContainerDied","Data":"7435667faf4c447b07139a8057de039910a89f935f9c570ec49be9f88f51e526"}
Jul 07 06:59:21.240050 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:21.239824    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j5jdf" event={"ID":"76cc2585-6fae-464f-afcf-f629ed055e11","Type":"ContainerStarted","Data":"eb28175db3c2feff64ee451408c9c69e37722edf2f8208501fca46b0d6cd1739"}
Jul 07 06:59:21.241428 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:21.241403    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jhkfq" event={"ID":"2915b9bd-a9ae-4d0f-a362-43b2ce659350","Type":"ContainerStarted","Data":"53f0019fe372b61e1afbdc825709a5fd512cbca320c5e1e5f20ab91118dbc57e"}
Jul 07 06:59:21.273255 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:21.273137    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-j5jdf" podStartSLOduration=14.842834648 podStartE2EDuration="39.273116543s" totalImagesPullingTime="24.430281895s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 06:58:42 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 06:59:21.270623665 +0000 UTC m=+40.814385172" watchObservedRunningTime="2026-07-07 06:59:21.273116543 +0000 UTC m=+40.816878025"
Jul 07 06:59:21.288087 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:21.288035    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-jhkfq" podStartSLOduration=1.080708192 podStartE2EDuration="5.288019159s" totalImagesPullingTime="4.207310967s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 06:59:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 06:59:21.287613667 +0000 UTC m=+40.831375129" watchObservedRunningTime="2026-07-07 06:59:21.288019159 +0000 UTC m=+40.831780639"
Jul 07 06:59:24.130333 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:24.130293    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4fb6f6e3-b570-408c-aaaa-a77d5bea0b24-metrics-tls\") pod \"dns-default-hpwxr\" (UID: \"4fb6f6e3-b570-408c-aaaa-a77d5bea0b24\") " pod="openshift-dns/dns-default-hpwxr"
Jul 07 06:59:24.130725 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:24.130359    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8b86dea1-145b-4f19-b0f8-ae53d29815bb-registry-tls\") pod \"image-registry-65b55697fd-4z2qz\" (UID: \"8b86dea1-145b-4f19-b0f8-ae53d29815bb\") " pod="openshift-image-registry/image-registry-65b55697fd-4z2qz"
Jul 07 06:59:24.130725 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:24.130459    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 07 06:59:24.130725 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:24.130460    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 06:59:24.130725 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:24.130531    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fb6f6e3-b570-408c-aaaa-a77d5bea0b24-metrics-tls podName:4fb6f6e3-b570-408c-aaaa-a77d5bea0b24 nodeName:}" failed. No retries permitted until 2026-07-07 06:59:32.130515806 +0000 UTC m=+51.674277270 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4fb6f6e3-b570-408c-aaaa-a77d5bea0b24-metrics-tls") pod "dns-default-hpwxr" (UID: "4fb6f6e3-b570-408c-aaaa-a77d5bea0b24") : secret "dns-default-metrics-tls" not found
Jul 07 06:59:24.130725 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:24.130470    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65b55697fd-4z2qz: secret "image-registry-tls" not found
Jul 07 06:59:24.130725 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:24.130598    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b86dea1-145b-4f19-b0f8-ae53d29815bb-registry-tls podName:8b86dea1-145b-4f19-b0f8-ae53d29815bb nodeName:}" failed. No retries permitted until 2026-07-07 06:59:32.130587116 +0000 UTC m=+51.674348575 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8b86dea1-145b-4f19-b0f8-ae53d29815bb-registry-tls") pod "image-registry-65b55697fd-4z2qz" (UID: "8b86dea1-145b-4f19-b0f8-ae53d29815bb") : secret "image-registry-tls" not found
Jul 07 06:59:32.190073 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:32.190028    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8b86dea1-145b-4f19-b0f8-ae53d29815bb-registry-tls\") pod \"image-registry-65b55697fd-4z2qz\" (UID: \"8b86dea1-145b-4f19-b0f8-ae53d29815bb\") " pod="openshift-image-registry/image-registry-65b55697fd-4z2qz"
Jul 07 06:59:32.190557 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:32.190099    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4fb6f6e3-b570-408c-aaaa-a77d5bea0b24-metrics-tls\") pod \"dns-default-hpwxr\" (UID: \"4fb6f6e3-b570-408c-aaaa-a77d5bea0b24\") " pod="openshift-dns/dns-default-hpwxr"
Jul 07 06:59:32.190557 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:32.190185    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 06:59:32.190557 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:32.190254    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fb6f6e3-b570-408c-aaaa-a77d5bea0b24-metrics-tls podName:4fb6f6e3-b570-408c-aaaa-a77d5bea0b24 nodeName:}" failed. No retries permitted until 2026-07-07 06:59:48.190238661 +0000 UTC m=+67.734000124 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4fb6f6e3-b570-408c-aaaa-a77d5bea0b24-metrics-tls") pod "dns-default-hpwxr" (UID: "4fb6f6e3-b570-408c-aaaa-a77d5bea0b24") : secret "dns-default-metrics-tls" not found
Jul 07 06:59:32.190557 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:32.190182    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 07 06:59:32.190557 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:32.190296    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65b55697fd-4z2qz: secret "image-registry-tls" not found
Jul 07 06:59:32.190557 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:32.190351    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b86dea1-145b-4f19-b0f8-ae53d29815bb-registry-tls podName:8b86dea1-145b-4f19-b0f8-ae53d29815bb nodeName:}" failed. No retries permitted until 2026-07-07 06:59:48.190338687 +0000 UTC m=+67.734100145 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8b86dea1-145b-4f19-b0f8-ae53d29815bb-registry-tls") pod "image-registry-65b55697fd-4z2qz" (UID: "8b86dea1-145b-4f19-b0f8-ae53d29815bb") : secret "image-registry-tls" not found
Jul 07 06:59:40.395348 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:40.395311    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-5d9gv"]
Jul 07 06:59:40.397212 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:40.397197    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-5d9gv"
Jul 07 06:59:40.400046 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:40.400025    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 07 06:59:40.401208 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:40.401190    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 07 06:59:40.401311 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:40.401217    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 07 06:59:40.401311 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:40.401239    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 07 06:59:40.401311 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:40.401217    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-7n4dm\""
Jul 07 06:59:40.416713 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:40.416688    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-5d9gv"]
Jul 07 06:59:40.449373 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:40.449349    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsrkh\" (UniqueName: \"kubernetes.io/projected/deb1b79a-dcc1-49e3-aaaf-6fd19b663de1-kube-api-access-zsrkh\") pod \"kube-storage-version-migrator-operator-846bf668f7-5d9gv\" (UID: \"deb1b79a-dcc1-49e3-aaaf-6fd19b663de1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-5d9gv"
Jul 07 06:59:40.449518 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:40.449380    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/deb1b79a-dcc1-49e3-aaaf-6fd19b663de1-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-5d9gv\" (UID: \"deb1b79a-dcc1-49e3-aaaf-6fd19b663de1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-5d9gv"
Jul 07 06:59:40.449518 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:40.449436    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/deb1b79a-dcc1-49e3-aaaf-6fd19b663de1-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-5d9gv\" (UID: \"deb1b79a-dcc1-49e3-aaaf-6fd19b663de1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-5d9gv"
Jul 07 06:59:40.489406 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:40.489377    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-b8lvb"]
Jul 07 06:59:40.491162 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:40.491143    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-b8lvb"
Jul 07 06:59:40.494298 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:40.494281    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-dzxct\""
Jul 07 06:59:40.501268 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:40.501248    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-b8lvb"]
Jul 07 06:59:40.550654 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:40.550623    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhx7q\" (UniqueName: \"kubernetes.io/projected/7bc6d126-dea3-4c67-947b-110fd52614f0-kube-api-access-zhx7q\") pod \"network-check-source-989f8b8dc-b8lvb\" (UID: \"7bc6d126-dea3-4c67-947b-110fd52614f0\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-b8lvb"
Jul 07 06:59:40.550654 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:40.550660    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/deb1b79a-dcc1-49e3-aaaf-6fd19b663de1-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-5d9gv\" (UID: \"deb1b79a-dcc1-49e3-aaaf-6fd19b663de1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-5d9gv"
Jul 07 06:59:40.550868 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:40.550751    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsrkh\" (UniqueName: \"kubernetes.io/projected/deb1b79a-dcc1-49e3-aaaf-6fd19b663de1-kube-api-access-zsrkh\") pod \"kube-storage-version-migrator-operator-846bf668f7-5d9gv\" (UID: \"deb1b79a-dcc1-49e3-aaaf-6fd19b663de1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-5d9gv"
Jul 07 06:59:40.550868 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:40.550783    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/deb1b79a-dcc1-49e3-aaaf-6fd19b663de1-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-5d9gv\" (UID: \"deb1b79a-dcc1-49e3-aaaf-6fd19b663de1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-5d9gv"
Jul 07 06:59:40.551312 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:40.551293    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/deb1b79a-dcc1-49e3-aaaf-6fd19b663de1-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-5d9gv\" (UID: \"deb1b79a-dcc1-49e3-aaaf-6fd19b663de1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-5d9gv"
Jul 07 06:59:40.554627 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:40.554602    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/deb1b79a-dcc1-49e3-aaaf-6fd19b663de1-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-5d9gv\" (UID: \"deb1b79a-dcc1-49e3-aaaf-6fd19b663de1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-5d9gv"
Jul 07 06:59:40.570854 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:40.570829    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsrkh\" (UniqueName: \"kubernetes.io/projected/deb1b79a-dcc1-49e3-aaaf-6fd19b663de1-kube-api-access-zsrkh\") pod \"kube-storage-version-migrator-operator-846bf668f7-5d9gv\" (UID: \"deb1b79a-dcc1-49e3-aaaf-6fd19b663de1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-5d9gv"
Jul 07 06:59:40.596511 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:40.596484    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-8h4xr"]
Jul 07 06:59:40.598367 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:40.598350    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-8h4xr"
Jul 07 06:59:40.600891 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:40.600874    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 07 06:59:40.601128 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:40.601104    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-566wk\""
Jul 07 06:59:40.601288 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:40.601272    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 07 06:59:40.601625 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:40.601610    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 07 06:59:40.604076 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:40.604061    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 07 06:59:40.611798 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:40.611778    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-8h4xr"]
Jul 07 06:59:40.651995 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:40.651892    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/565f2b59-eb10-4b8c-a103-8e2e52dcb823-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-8h4xr\" (UID: \"565f2b59-eb10-4b8c-a103-8e2e52dcb823\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-8h4xr"
Jul 07 06:59:40.652147 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:40.652004    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhx7q\" (UniqueName: \"kubernetes.io/projected/7bc6d126-dea3-4c67-947b-110fd52614f0-kube-api-access-zhx7q\") pod \"network-check-source-989f8b8dc-b8lvb\" (UID: \"7bc6d126-dea3-4c67-947b-110fd52614f0\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-b8lvb"
Jul 07 06:59:40.652147 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:40.652030    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/565f2b59-eb10-4b8c-a103-8e2e52dcb823-config\") pod \"service-ca-operator-5d46d9ffd4-8h4xr\" (UID: \"565f2b59-eb10-4b8c-a103-8e2e52dcb823\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-8h4xr"
Jul 07 06:59:40.652147 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:40.652055    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkttw\" (UniqueName: \"kubernetes.io/projected/565f2b59-eb10-4b8c-a103-8e2e52dcb823-kube-api-access-mkttw\") pod \"service-ca-operator-5d46d9ffd4-8h4xr\" (UID: \"565f2b59-eb10-4b8c-a103-8e2e52dcb823\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-8h4xr"
Jul 07 06:59:40.664108 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:40.664079    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhx7q\" (UniqueName: \"kubernetes.io/projected/7bc6d126-dea3-4c67-947b-110fd52614f0-kube-api-access-zhx7q\") pod \"network-check-source-989f8b8dc-b8lvb\" (UID: \"7bc6d126-dea3-4c67-947b-110fd52614f0\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-b8lvb"
Jul 07 06:59:40.706485 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:40.706457    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-5d9gv"
Jul 07 06:59:40.752765 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:40.752731    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/565f2b59-eb10-4b8c-a103-8e2e52dcb823-config\") pod \"service-ca-operator-5d46d9ffd4-8h4xr\" (UID: \"565f2b59-eb10-4b8c-a103-8e2e52dcb823\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-8h4xr"
Jul 07 06:59:40.752895 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:40.752785    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkttw\" (UniqueName: \"kubernetes.io/projected/565f2b59-eb10-4b8c-a103-8e2e52dcb823-kube-api-access-mkttw\") pod \"service-ca-operator-5d46d9ffd4-8h4xr\" (UID: \"565f2b59-eb10-4b8c-a103-8e2e52dcb823\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-8h4xr"
Jul 07 06:59:40.752895 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:40.752871    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/565f2b59-eb10-4b8c-a103-8e2e52dcb823-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-8h4xr\" (UID: \"565f2b59-eb10-4b8c-a103-8e2e52dcb823\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-8h4xr"
Jul 07 06:59:40.753867 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:40.753837    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/565f2b59-eb10-4b8c-a103-8e2e52dcb823-config\") pod \"service-ca-operator-5d46d9ffd4-8h4xr\" (UID: \"565f2b59-eb10-4b8c-a103-8e2e52dcb823\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-8h4xr"
Jul 07 06:59:40.755207 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:40.755183    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/565f2b59-eb10-4b8c-a103-8e2e52dcb823-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-8h4xr\" (UID: \"565f2b59-eb10-4b8c-a103-8e2e52dcb823\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-8h4xr"
Jul 07 06:59:40.764759 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:40.764728    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkttw\" (UniqueName: \"kubernetes.io/projected/565f2b59-eb10-4b8c-a103-8e2e52dcb823-kube-api-access-mkttw\") pod \"service-ca-operator-5d46d9ffd4-8h4xr\" (UID: \"565f2b59-eb10-4b8c-a103-8e2e52dcb823\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-8h4xr"
Jul 07 06:59:40.799640 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:40.799615    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-b8lvb"
Jul 07 06:59:40.829803 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:40.829774    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-5d9gv"]
Jul 07 06:59:40.832722 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:59:40.832694    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeb1b79a_dcc1_49e3_aaaf_6fd19b663de1.slice/crio-a0e6e9e778ec2b47e9d480890325ce99b6a93c17aa999ff48fea17ca35428ddd WatchSource:0}: Error finding container a0e6e9e778ec2b47e9d480890325ce99b6a93c17aa999ff48fea17ca35428ddd: Status 404 returned error can't find the container with id a0e6e9e778ec2b47e9d480890325ce99b6a93c17aa999ff48fea17ca35428ddd
Jul 07 06:59:40.909379 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:40.909295    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-566wk\""
Jul 07 06:59:40.917030 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:40.917005    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-8h4xr"
Jul 07 06:59:40.949365 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:40.949333    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-b8lvb"]
Jul 07 06:59:40.952731 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:59:40.952706    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bc6d126_dea3_4c67_947b_110fd52614f0.slice/crio-4bdaf0fce4b0e7141532e544874e65f5a1cb19c07697715c2d375ae007947bda WatchSource:0}: Error finding container 4bdaf0fce4b0e7141532e544874e65f5a1cb19c07697715c2d375ae007947bda: Status 404 returned error can't find the container with id 4bdaf0fce4b0e7141532e544874e65f5a1cb19c07697715c2d375ae007947bda
Jul 07 06:59:41.040512 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:41.040481    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-8h4xr"]
Jul 07 06:59:41.043668 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:59:41.043630    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod565f2b59_eb10_4b8c_a103_8e2e52dcb823.slice/crio-e384eea0d9a1b98ba64a0671dd5f4e69e2c2ecbf67d5fe686cc0fe02014e44ed WatchSource:0}: Error finding container e384eea0d9a1b98ba64a0671dd5f4e69e2c2ecbf67d5fe686cc0fe02014e44ed: Status 404 returned error can't find the container with id e384eea0d9a1b98ba64a0671dd5f4e69e2c2ecbf67d5fe686cc0fe02014e44ed
Jul 07 06:59:41.289122 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:41.289086    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-8h4xr" event={"ID":"565f2b59-eb10-4b8c-a103-8e2e52dcb823","Type":"ContainerStarted","Data":"e384eea0d9a1b98ba64a0671dd5f4e69e2c2ecbf67d5fe686cc0fe02014e44ed"}
Jul 07 06:59:41.290054 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:41.290034    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-b8lvb" event={"ID":"7bc6d126-dea3-4c67-947b-110fd52614f0","Type":"ContainerStarted","Data":"4bdaf0fce4b0e7141532e544874e65f5a1cb19c07697715c2d375ae007947bda"}
Jul 07 06:59:41.291068 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:41.291047    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-5d9gv" event={"ID":"deb1b79a-dcc1-49e3-aaaf-6fd19b663de1","Type":"ContainerStarted","Data":"a0e6e9e778ec2b47e9d480890325ce99b6a93c17aa999ff48fea17ca35428ddd"}
Jul 07 06:59:43.219696 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:43.219662    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wvm5w"
Jul 07 06:59:45.304409 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:45.304370    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-8h4xr" event={"ID":"565f2b59-eb10-4b8c-a103-8e2e52dcb823","Type":"ContainerStarted","Data":"40440345f29150189f2acb35def60eeb4e560e4bc733bf30f3a9d4269f1a67f4"}
Jul 07 06:59:45.306138 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:45.306116    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-b8lvb" event={"ID":"7bc6d126-dea3-4c67-947b-110fd52614f0","Type":"ContainerStarted","Data":"85016fd5be687392eaced1be51a225f9c171f5b3ac0014a2f9e3f33043fff83f"}
Jul 07 06:59:45.307758 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:45.307736    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-5d9gv" event={"ID":"deb1b79a-dcc1-49e3-aaaf-6fd19b663de1","Type":"ContainerStarted","Data":"25e06c3a05158a5a1929cbf640e4f05ed7561af1bf9a2e3a9d0fe329e375f670"}
Jul 07 06:59:45.324112 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:45.324039    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-8h4xr" podStartSLOduration=1.525987602 podStartE2EDuration="5.324019525s" totalImagesPullingTime="3.798031923s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 06:59:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 06:59:45.323183063 +0000 UTC m=+64.866944545" watchObservedRunningTime="2026-07-07 06:59:45.324019525 +0000 UTC m=+64.867781009"
Jul 07 06:59:45.346970 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:45.346894    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-b8lvb" podStartSLOduration=1.447222006 podStartE2EDuration="5.346873234s" totalImagesPullingTime="3.899651228s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 06:59:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 06:59:45.346311586 +0000 UTC m=+64.890073068" watchObservedRunningTime="2026-07-07 06:59:45.346873234 +0000 UTC m=+64.890634715"
Jul 07 06:59:45.364529 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:45.364478    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-5d9gv" podStartSLOduration=1.351250321 podStartE2EDuration="5.364459027s" totalImagesPullingTime="4.013208706s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 06:59:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 06:59:45.363629939 +0000 UTC m=+64.907391421" watchObservedRunningTime="2026-07-07 06:59:45.364459027 +0000 UTC m=+64.908220510"
Jul 07 06:59:46.707888 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:46.707852    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0dba90e8-7ad4-46cb-bc27-718b6f644f5d-metrics-certs\") pod \"network-metrics-daemon-s9stz\" (UID: \"0dba90e8-7ad4-46cb-bc27-718b6f644f5d\") " pod="openshift-multus/network-metrics-daemon-s9stz"
Jul 07 06:59:46.710925 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:46.710902    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 07 06:59:46.718345 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:46.718309    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 07 06:59:46.718483 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:46.718380    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0dba90e8-7ad4-46cb-bc27-718b6f644f5d-metrics-certs podName:0dba90e8-7ad4-46cb-bc27-718b6f644f5d nodeName:}" failed. No retries permitted until 2026-07-07 07:00:50.718362892 +0000 UTC m=+130.262124351 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0dba90e8-7ad4-46cb-bc27-718b6f644f5d-metrics-certs") pod "network-metrics-daemon-s9stz" (UID: "0dba90e8-7ad4-46cb-bc27-718b6f644f5d") : secret "metrics-daemon-secret" not found
Jul 07 06:59:46.808672 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:46.808631    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/575f1248-fecc-48d9-9c6a-db334596c8b6-original-pull-secret\") pod \"global-pull-secret-syncer-gnlks\" (UID: \"575f1248-fecc-48d9-9c6a-db334596c8b6\") " pod="kube-system/global-pull-secret-syncer-gnlks"
Jul 07 06:59:46.811764 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:46.811732    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 07 06:59:46.821454 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:46.821430    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/575f1248-fecc-48d9-9c6a-db334596c8b6-original-pull-secret\") pod \"global-pull-secret-syncer-gnlks\" (UID: \"575f1248-fecc-48d9-9c6a-db334596c8b6\") " pod="kube-system/global-pull-secret-syncer-gnlks"
Jul 07 06:59:46.909827 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:46.909789    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv9bp\" (UniqueName: \"kubernetes.io/projected/4e917e03-b190-40ac-8c93-cd4d794fa127-kube-api-access-nv9bp\") pod \"network-check-target-nfjzx\" (UID: \"4e917e03-b190-40ac-8c93-cd4d794fa127\") " pod="openshift-network-diagnostics/network-check-target-nfjzx"
Jul 07 06:59:46.912345 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:46.912316    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv9bp\" (UniqueName: \"kubernetes.io/projected/4e917e03-b190-40ac-8c93-cd4d794fa127-kube-api-access-nv9bp\") pod \"network-check-target-nfjzx\" (UID: \"4e917e03-b190-40ac-8c93-cd4d794fa127\") " pod="openshift-network-diagnostics/network-check-target-nfjzx"
Jul 07 06:59:47.072885 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:47.072856    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2lkzt\""
Jul 07 06:59:47.080222 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:47.080197    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nfjzx"
Jul 07 06:59:47.082305 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:47.082288    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gnlks"
Jul 07 06:59:47.210660 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:47.210629    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-nfjzx"]
Jul 07 06:59:47.214022 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:59:47.213995    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e917e03_b190_40ac_8c93_cd4d794fa127.slice/crio-7e44f7f12aca03c4bb049fa3b52181a657a1593c56bdc43e82547cae135e49f9 WatchSource:0}: Error finding container 7e44f7f12aca03c4bb049fa3b52181a657a1593c56bdc43e82547cae135e49f9: Status 404 returned error can't find the container with id 7e44f7f12aca03c4bb049fa3b52181a657a1593c56bdc43e82547cae135e49f9
Jul 07 06:59:47.227755 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:47.227733    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gnlks"]
Jul 07 06:59:47.231788 ip-10-0-138-97 kubenswrapper[2579]: W0707 06:59:47.231763    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod575f1248_fecc_48d9_9c6a_db334596c8b6.slice/crio-7db339f40e79f4ba5ffaf74dcf4af8b96919b20d09b1fd9aa510b1c893793b05 WatchSource:0}: Error finding container 7db339f40e79f4ba5ffaf74dcf4af8b96919b20d09b1fd9aa510b1c893793b05: Status 404 returned error can't find the container with id 7db339f40e79f4ba5ffaf74dcf4af8b96919b20d09b1fd9aa510b1c893793b05
Jul 07 06:59:47.314100 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:47.314066    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gnlks" event={"ID":"575f1248-fecc-48d9-9c6a-db334596c8b6","Type":"ContainerStarted","Data":"7db339f40e79f4ba5ffaf74dcf4af8b96919b20d09b1fd9aa510b1c893793b05"}
Jul 07 06:59:47.315545 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:47.315523    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-nfjzx" event={"ID":"4e917e03-b190-40ac-8c93-cd4d794fa127","Type":"ContainerStarted","Data":"134fbf65f7ad0a4f6f726903ccb14f6c71716e12e30e41ac95e9d9aaa3e79ad5"}
Jul 07 06:59:47.315674 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:47.315556    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-nfjzx" event={"ID":"4e917e03-b190-40ac-8c93-cd4d794fa127","Type":"ContainerStarted","Data":"7e44f7f12aca03c4bb049fa3b52181a657a1593c56bdc43e82547cae135e49f9"}
Jul 07 06:59:47.315674 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:47.315649    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-nfjzx"
Jul 07 06:59:47.341904 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:47.341799    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-nfjzx" podStartSLOduration=65.341782241 podStartE2EDuration="1m5.341782241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 06:58:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 06:59:47.341674579 +0000 UTC m=+66.885436062" watchObservedRunningTime="2026-07-07 06:59:47.341782241 +0000 UTC m=+66.885543722"
Jul 07 06:59:47.426531 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:47.426498    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-brh4b"]
Jul 07 06:59:47.428549 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:47.428529    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-brh4b"
Jul 07 06:59:47.431808 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:47.431768    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 07 06:59:47.431923 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:47.431856    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-s62sg\""
Jul 07 06:59:47.431923 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:47.431913    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 07 06:59:47.432487 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:47.432289    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 07 06:59:47.439165 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:47.439147    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 07 06:59:47.452578 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:47.452550    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-brh4b"]
Jul 07 06:59:47.514808 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:47.514773    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a852d8bb-79ad-4ead-8002-8e51919be749-data-volume\") pod \"insights-runtime-extractor-brh4b\" (UID: \"a852d8bb-79ad-4ead-8002-8e51919be749\") " pod="openshift-insights/insights-runtime-extractor-brh4b"
Jul 07 06:59:47.515008 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:47.514836    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmm4w\" (UniqueName: \"kubernetes.io/projected/a852d8bb-79ad-4ead-8002-8e51919be749-kube-api-access-hmm4w\") pod \"insights-runtime-extractor-brh4b\" (UID: \"a852d8bb-79ad-4ead-8002-8e51919be749\") " pod="openshift-insights/insights-runtime-extractor-brh4b"
Jul 07 06:59:47.515008 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:47.514912    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a852d8bb-79ad-4ead-8002-8e51919be749-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-brh4b\" (UID: \"a852d8bb-79ad-4ead-8002-8e51919be749\") " pod="openshift-insights/insights-runtime-extractor-brh4b"
Jul 07 06:59:47.515008 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:47.514942    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a852d8bb-79ad-4ead-8002-8e51919be749-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-brh4b\" (UID: \"a852d8bb-79ad-4ead-8002-8e51919be749\") " pod="openshift-insights/insights-runtime-extractor-brh4b"
Jul 07 06:59:47.515008 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:47.514983    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a852d8bb-79ad-4ead-8002-8e51919be749-crio-socket\") pod \"insights-runtime-extractor-brh4b\" (UID: \"a852d8bb-79ad-4ead-8002-8e51919be749\") " pod="openshift-insights/insights-runtime-extractor-brh4b"
Jul 07 06:59:47.616344 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:47.616257    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a852d8bb-79ad-4ead-8002-8e51919be749-crio-socket\") pod \"insights-runtime-extractor-brh4b\" (UID: \"a852d8bb-79ad-4ead-8002-8e51919be749\") " pod="openshift-insights/insights-runtime-extractor-brh4b"
Jul 07 06:59:47.616344 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:47.616317    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a852d8bb-79ad-4ead-8002-8e51919be749-data-volume\") pod \"insights-runtime-extractor-brh4b\" (UID: \"a852d8bb-79ad-4ead-8002-8e51919be749\") " pod="openshift-insights/insights-runtime-extractor-brh4b"
Jul 07 06:59:47.616543 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:47.616366    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmm4w\" (UniqueName: \"kubernetes.io/projected/a852d8bb-79ad-4ead-8002-8e51919be749-kube-api-access-hmm4w\") pod \"insights-runtime-extractor-brh4b\" (UID: \"a852d8bb-79ad-4ead-8002-8e51919be749\") " pod="openshift-insights/insights-runtime-extractor-brh4b"
Jul 07 06:59:47.616543 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:47.616396    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a852d8bb-79ad-4ead-8002-8e51919be749-crio-socket\") pod \"insights-runtime-extractor-brh4b\" (UID: \"a852d8bb-79ad-4ead-8002-8e51919be749\") " pod="openshift-insights/insights-runtime-extractor-brh4b"
Jul 07 06:59:47.616543 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:47.616403    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a852d8bb-79ad-4ead-8002-8e51919be749-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-brh4b\" (UID: \"a852d8bb-79ad-4ead-8002-8e51919be749\") " pod="openshift-insights/insights-runtime-extractor-brh4b"
Jul 07 06:59:47.616543 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:47.616467    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a852d8bb-79ad-4ead-8002-8e51919be749-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-brh4b\" (UID: \"a852d8bb-79ad-4ead-8002-8e51919be749\") " pod="openshift-insights/insights-runtime-extractor-brh4b"
Jul 07 06:59:47.616726 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:47.616607    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 06:59:47.616726 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:47.616635    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 07 06:59:47.616798 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:47.616730    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a852d8bb-79ad-4ead-8002-8e51919be749-kube-rbac-proxy-cm podName:a852d8bb-79ad-4ead-8002-8e51919be749 nodeName:}" failed. No retries permitted until 2026-07-07 06:59:48.116701125 +0000 UTC m=+67.660462599 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a852d8bb-79ad-4ead-8002-8e51919be749-kube-rbac-proxy-cm") pod "insights-runtime-extractor-brh4b" (UID: "a852d8bb-79ad-4ead-8002-8e51919be749") : configmap "kube-rbac-proxy" not found
Jul 07 06:59:47.616798 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:47.616757    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a852d8bb-79ad-4ead-8002-8e51919be749-data-volume\") pod \"insights-runtime-extractor-brh4b\" (UID: \"a852d8bb-79ad-4ead-8002-8e51919be749\") " pod="openshift-insights/insights-runtime-extractor-brh4b"
Jul 07 06:59:47.616798 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:47.616775    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a852d8bb-79ad-4ead-8002-8e51919be749-insights-runtime-extractor-tls podName:a852d8bb-79ad-4ead-8002-8e51919be749 nodeName:}" failed. No retries permitted until 2026-07-07 06:59:48.116756715 +0000 UTC m=+67.660518188 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a852d8bb-79ad-4ead-8002-8e51919be749-insights-runtime-extractor-tls") pod "insights-runtime-extractor-brh4b" (UID: "a852d8bb-79ad-4ead-8002-8e51919be749") : secret "insights-runtime-extractor-tls" not found
Jul 07 06:59:47.626458 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:47.626430    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmm4w\" (UniqueName: \"kubernetes.io/projected/a852d8bb-79ad-4ead-8002-8e51919be749-kube-api-access-hmm4w\") pod \"insights-runtime-extractor-brh4b\" (UID: \"a852d8bb-79ad-4ead-8002-8e51919be749\") " pod="openshift-insights/insights-runtime-extractor-brh4b"
Jul 07 06:59:48.119575 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:48.119538    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a852d8bb-79ad-4ead-8002-8e51919be749-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-brh4b\" (UID: \"a852d8bb-79ad-4ead-8002-8e51919be749\") " pod="openshift-insights/insights-runtime-extractor-brh4b"
Jul 07 06:59:48.119575 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:48.119577    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a852d8bb-79ad-4ead-8002-8e51919be749-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-brh4b\" (UID: \"a852d8bb-79ad-4ead-8002-8e51919be749\") " pod="openshift-insights/insights-runtime-extractor-brh4b"
Jul 07 06:59:48.120068 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:48.119697    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 06:59:48.120068 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:48.119757    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 07 06:59:48.120068 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:48.119765    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a852d8bb-79ad-4ead-8002-8e51919be749-insights-runtime-extractor-tls podName:a852d8bb-79ad-4ead-8002-8e51919be749 nodeName:}" failed. No retries permitted until 2026-07-07 06:59:49.119750963 +0000 UTC m=+68.663512425 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a852d8bb-79ad-4ead-8002-8e51919be749-insights-runtime-extractor-tls") pod "insights-runtime-extractor-brh4b" (UID: "a852d8bb-79ad-4ead-8002-8e51919be749") : secret "insights-runtime-extractor-tls" not found
Jul 07 06:59:48.120068 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:48.119836    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a852d8bb-79ad-4ead-8002-8e51919be749-kube-rbac-proxy-cm podName:a852d8bb-79ad-4ead-8002-8e51919be749 nodeName:}" failed. No retries permitted until 2026-07-07 06:59:49.119817216 +0000 UTC m=+68.663578687 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a852d8bb-79ad-4ead-8002-8e51919be749-kube-rbac-proxy-cm") pod "insights-runtime-extractor-brh4b" (UID: "a852d8bb-79ad-4ead-8002-8e51919be749") : configmap "kube-rbac-proxy" not found
Jul 07 06:59:48.220498 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:48.220463    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8b86dea1-145b-4f19-b0f8-ae53d29815bb-registry-tls\") pod \"image-registry-65b55697fd-4z2qz\" (UID: \"8b86dea1-145b-4f19-b0f8-ae53d29815bb\") " pod="openshift-image-registry/image-registry-65b55697fd-4z2qz"
Jul 07 06:59:48.220655 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:48.220533    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4fb6f6e3-b570-408c-aaaa-a77d5bea0b24-metrics-tls\") pod \"dns-default-hpwxr\" (UID: \"4fb6f6e3-b570-408c-aaaa-a77d5bea0b24\") " pod="openshift-dns/dns-default-hpwxr"
Jul 07 06:59:48.220720 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:48.220692    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 07 06:59:48.220720 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:48.220696    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 06:59:48.220798 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:48.220789    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fb6f6e3-b570-408c-aaaa-a77d5bea0b24-metrics-tls podName:4fb6f6e3-b570-408c-aaaa-a77d5bea0b24 nodeName:}" failed. No retries permitted until 2026-07-07 07:00:20.220768145 +0000 UTC m=+99.764529619 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4fb6f6e3-b570-408c-aaaa-a77d5bea0b24-metrics-tls") pod "dns-default-hpwxr" (UID: "4fb6f6e3-b570-408c-aaaa-a77d5bea0b24") : secret "dns-default-metrics-tls" not found
Jul 07 06:59:48.220846 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:48.220707    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65b55697fd-4z2qz: secret "image-registry-tls" not found
Jul 07 06:59:48.220895 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:48.220855    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b86dea1-145b-4f19-b0f8-ae53d29815bb-registry-tls podName:8b86dea1-145b-4f19-b0f8-ae53d29815bb nodeName:}" failed. No retries permitted until 2026-07-07 07:00:20.220841643 +0000 UTC m=+99.764603119 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8b86dea1-145b-4f19-b0f8-ae53d29815bb-registry-tls") pod "image-registry-65b55697fd-4z2qz" (UID: "8b86dea1-145b-4f19-b0f8-ae53d29815bb") : secret "image-registry-tls" not found
Jul 07 06:59:49.129103 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:49.129060    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a852d8bb-79ad-4ead-8002-8e51919be749-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-brh4b\" (UID: \"a852d8bb-79ad-4ead-8002-8e51919be749\") " pod="openshift-insights/insights-runtime-extractor-brh4b"
Jul 07 06:59:49.129103 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:49.129111    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a852d8bb-79ad-4ead-8002-8e51919be749-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-brh4b\" (UID: \"a852d8bb-79ad-4ead-8002-8e51919be749\") " pod="openshift-insights/insights-runtime-extractor-brh4b"
Jul 07 06:59:49.129624 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:49.129209    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 07 06:59:49.129624 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:49.129290    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a852d8bb-79ad-4ead-8002-8e51919be749-kube-rbac-proxy-cm podName:a852d8bb-79ad-4ead-8002-8e51919be749 nodeName:}" failed. No retries permitted until 2026-07-07 06:59:51.129271731 +0000 UTC m=+70.673033190 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a852d8bb-79ad-4ead-8002-8e51919be749-kube-rbac-proxy-cm") pod "insights-runtime-extractor-brh4b" (UID: "a852d8bb-79ad-4ead-8002-8e51919be749") : configmap "kube-rbac-proxy" not found
Jul 07 06:59:49.129624 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:49.129341    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 06:59:49.129624 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:49.129421    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a852d8bb-79ad-4ead-8002-8e51919be749-insights-runtime-extractor-tls podName:a852d8bb-79ad-4ead-8002-8e51919be749 nodeName:}" failed. No retries permitted until 2026-07-07 06:59:51.129403466 +0000 UTC m=+70.673164925 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a852d8bb-79ad-4ead-8002-8e51919be749-insights-runtime-extractor-tls") pod "insights-runtime-extractor-brh4b" (UID: "a852d8bb-79ad-4ead-8002-8e51919be749") : secret "insights-runtime-extractor-tls" not found
Jul 07 06:59:50.421491 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:50.421460    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-r77h8_86018993-6534-4639-9254-a898f62948fe/dns-node-resolver/0.log"
Jul 07 06:59:51.148691 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:51.148652    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a852d8bb-79ad-4ead-8002-8e51919be749-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-brh4b\" (UID: \"a852d8bb-79ad-4ead-8002-8e51919be749\") " pod="openshift-insights/insights-runtime-extractor-brh4b"
Jul 07 06:59:51.148886 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:51.148703    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a852d8bb-79ad-4ead-8002-8e51919be749-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-brh4b\" (UID: \"a852d8bb-79ad-4ead-8002-8e51919be749\") " pod="openshift-insights/insights-runtime-extractor-brh4b"
Jul 07 06:59:51.148886 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:51.148790    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 07 06:59:51.148886 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:51.148830    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 06:59:51.149054 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:51.148905    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a852d8bb-79ad-4ead-8002-8e51919be749-insights-runtime-extractor-tls podName:a852d8bb-79ad-4ead-8002-8e51919be749 nodeName:}" failed. No retries permitted until 2026-07-07 06:59:55.148886441 +0000 UTC m=+74.692647917 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a852d8bb-79ad-4ead-8002-8e51919be749-insights-runtime-extractor-tls") pod "insights-runtime-extractor-brh4b" (UID: "a852d8bb-79ad-4ead-8002-8e51919be749") : secret "insights-runtime-extractor-tls" not found
Jul 07 06:59:51.149054 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:51.148932    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a852d8bb-79ad-4ead-8002-8e51919be749-kube-rbac-proxy-cm podName:a852d8bb-79ad-4ead-8002-8e51919be749 nodeName:}" failed. No retries permitted until 2026-07-07 06:59:55.148925452 +0000 UTC m=+74.692686911 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a852d8bb-79ad-4ead-8002-8e51919be749-kube-rbac-proxy-cm") pod "insights-runtime-extractor-brh4b" (UID: "a852d8bb-79ad-4ead-8002-8e51919be749") : configmap "kube-rbac-proxy" not found
Jul 07 06:59:51.222143 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:51.222112    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-fzzb4_7c1bef18-765e-4290-8c9c-80c985fcb304/node-ca/0.log"
Jul 07 06:59:52.221675 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:52.221649    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jhkfq_2915b9bd-a9ae-4d0f-a362-43b2ce659350/serve-healthcheck-canary/0.log"
Jul 07 06:59:52.332308 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:52.332272    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gnlks" event={"ID":"575f1248-fecc-48d9-9c6a-db334596c8b6","Type":"ContainerStarted","Data":"717cca111cda4a88729907957800a1d7e5c0b490c705c17a978e9c251db110d6"}
Jul 07 06:59:52.352074 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:52.352022    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-gnlks" podStartSLOduration=65.952932501 podStartE2EDuration="1m10.352007138s" totalImagesPullingTime="4.399074637s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 06:58:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 06:59:52.351253248 +0000 UTC m=+71.895014730" watchObservedRunningTime="2026-07-07 06:59:52.352007138 +0000 UTC m=+71.895768616"
Jul 07 06:59:52.829024 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:52.828993    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-5d9gv_deb1b79a-dcc1-49e3-aaaf-6fd19b663de1/kube-storage-version-migrator-operator/0.log"
Jul 07 06:59:55.181179 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:55.181134    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a852d8bb-79ad-4ead-8002-8e51919be749-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-brh4b\" (UID: \"a852d8bb-79ad-4ead-8002-8e51919be749\") " pod="openshift-insights/insights-runtime-extractor-brh4b"
Jul 07 06:59:55.181664 ip-10-0-138-97 kubenswrapper[2579]: I0707 06:59:55.181259    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a852d8bb-79ad-4ead-8002-8e51919be749-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-brh4b\" (UID: \"a852d8bb-79ad-4ead-8002-8e51919be749\") " pod="openshift-insights/insights-runtime-extractor-brh4b"
Jul 07 06:59:55.181664 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:55.181261    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 07 06:59:55.181664 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:55.181345    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 06:59:55.181664 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:55.181356    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a852d8bb-79ad-4ead-8002-8e51919be749-kube-rbac-proxy-cm podName:a852d8bb-79ad-4ead-8002-8e51919be749 nodeName:}" failed. No retries permitted until 2026-07-07 07:00:03.181338808 +0000 UTC m=+82.725100285 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a852d8bb-79ad-4ead-8002-8e51919be749-kube-rbac-proxy-cm") pod "insights-runtime-extractor-brh4b" (UID: "a852d8bb-79ad-4ead-8002-8e51919be749") : configmap "kube-rbac-proxy" not found
Jul 07 06:59:55.181664 ip-10-0-138-97 kubenswrapper[2579]: E0707 06:59:55.181392    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a852d8bb-79ad-4ead-8002-8e51919be749-insights-runtime-extractor-tls podName:a852d8bb-79ad-4ead-8002-8e51919be749 nodeName:}" failed. No retries permitted until 2026-07-07 07:00:03.181381695 +0000 UTC m=+82.725143159 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a852d8bb-79ad-4ead-8002-8e51919be749-insights-runtime-extractor-tls") pod "insights-runtime-extractor-brh4b" (UID: "a852d8bb-79ad-4ead-8002-8e51919be749") : secret "insights-runtime-extractor-tls" not found
Jul 07 07:00:03.245649 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:03.245526    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a852d8bb-79ad-4ead-8002-8e51919be749-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-brh4b\" (UID: \"a852d8bb-79ad-4ead-8002-8e51919be749\") " pod="openshift-insights/insights-runtime-extractor-brh4b"
Jul 07 07:00:03.245649 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:03.245566    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a852d8bb-79ad-4ead-8002-8e51919be749-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-brh4b\" (UID: \"a852d8bb-79ad-4ead-8002-8e51919be749\") " pod="openshift-insights/insights-runtime-extractor-brh4b"
Jul 07 07:00:03.246138 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:03.246121    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a852d8bb-79ad-4ead-8002-8e51919be749-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-brh4b\" (UID: \"a852d8bb-79ad-4ead-8002-8e51919be749\") " pod="openshift-insights/insights-runtime-extractor-brh4b"
Jul 07 07:00:03.247928 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:03.247906    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a852d8bb-79ad-4ead-8002-8e51919be749-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-brh4b\" (UID: \"a852d8bb-79ad-4ead-8002-8e51919be749\") " pod="openshift-insights/insights-runtime-extractor-brh4b"
Jul 07 07:00:03.338839 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:03.338789    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-brh4b"
Jul 07 07:00:03.467423 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:03.467385    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-brh4b"]
Jul 07 07:00:03.470280 ip-10-0-138-97 kubenswrapper[2579]: W0707 07:00:03.470249    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda852d8bb_79ad_4ead_8002_8e51919be749.slice/crio-9bc6841fc9197db6ad5a2436858808a2068e704bd9785a2b3286d2fc1b0d8807 WatchSource:0}: Error finding container 9bc6841fc9197db6ad5a2436858808a2068e704bd9785a2b3286d2fc1b0d8807: Status 404 returned error can't find the container with id 9bc6841fc9197db6ad5a2436858808a2068e704bd9785a2b3286d2fc1b0d8807
Jul 07 07:00:04.374655 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:04.374616    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-brh4b" event={"ID":"a852d8bb-79ad-4ead-8002-8e51919be749","Type":"ContainerStarted","Data":"9bc6841fc9197db6ad5a2436858808a2068e704bd9785a2b3286d2fc1b0d8807"}
Jul 07 07:00:08.388301 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:08.388260    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-brh4b" event={"ID":"a852d8bb-79ad-4ead-8002-8e51919be749","Type":"ContainerStarted","Data":"1399a5b3fe80d8de6c732336e1b7b7ce708ef67495002e5cdb8dc9479f451a12"}
Jul 07 07:00:09.394201 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:09.394167    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-brh4b" event={"ID":"a852d8bb-79ad-4ead-8002-8e51919be749","Type":"ContainerStarted","Data":"aa26846f6ed69bdd888b26780fdf3cc3a3d4bbcc7b950e306a3362575658abfd"}
Jul 07 07:00:09.738006 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:09.737969    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84b8f8b644-lzp2z"]
Jul 07 07:00:09.741172 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:09.741152    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84b8f8b644-lzp2z"
Jul 07 07:00:09.749809 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:09.749786    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 07 07:00:09.751134 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:09.750862    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 07 07:00:09.751134 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:09.750935    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 07 07:00:09.751134 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:09.751027    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 07 07:00:09.751765 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:09.751747    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-nsbsg\""
Jul 07 07:00:09.765362 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:09.765340    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84b8f8b644-lzp2z"]
Jul 07 07:00:09.779199 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:09.779177    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-55f6d89cf-fkpgt"]
Jul 07 07:00:09.782226 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:09.782209    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-55f6d89cf-fkpgt"
Jul 07 07:00:09.796027 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:09.796002    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 07 07:00:09.804420 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:09.802539    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-55f6d89cf-fkpgt"]
Jul 07 07:00:09.896610 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:09.896567    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b71dd51e-ea47-45aa-a472-9dff7472e20c-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-84b8f8b644-lzp2z\" (UID: \"b71dd51e-ea47-45aa-a472-9dff7472e20c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84b8f8b644-lzp2z"
Jul 07 07:00:09.896783 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:09.896699    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2k8m\" (UniqueName: \"kubernetes.io/projected/b71dd51e-ea47-45aa-a472-9dff7472e20c-kube-api-access-r2k8m\") pod \"managed-serviceaccount-addon-agent-84b8f8b644-lzp2z\" (UID: \"b71dd51e-ea47-45aa-a472-9dff7472e20c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84b8f8b644-lzp2z"
Jul 07 07:00:09.896843 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:09.896792    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3434ff8c-97ac-4c97-adcc-66049d141514-tmp\") pod \"klusterlet-addon-workmgr-55f6d89cf-fkpgt\" (UID: \"3434ff8c-97ac-4c97-adcc-66049d141514\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-55f6d89cf-fkpgt"
Jul 07 07:00:09.896843 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:09.896822    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/3434ff8c-97ac-4c97-adcc-66049d141514-klusterlet-config\") pod \"klusterlet-addon-workmgr-55f6d89cf-fkpgt\" (UID: \"3434ff8c-97ac-4c97-adcc-66049d141514\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-55f6d89cf-fkpgt"
Jul 07 07:00:09.896933 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:09.896860    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffkwb\" (UniqueName: \"kubernetes.io/projected/3434ff8c-97ac-4c97-adcc-66049d141514-kube-api-access-ffkwb\") pod \"klusterlet-addon-workmgr-55f6d89cf-fkpgt\" (UID: \"3434ff8c-97ac-4c97-adcc-66049d141514\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-55f6d89cf-fkpgt"
Jul 07 07:00:09.997514 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:09.997416    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b71dd51e-ea47-45aa-a472-9dff7472e20c-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-84b8f8b644-lzp2z\" (UID: \"b71dd51e-ea47-45aa-a472-9dff7472e20c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84b8f8b644-lzp2z"
Jul 07 07:00:09.997514 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:09.997507    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2k8m\" (UniqueName: \"kubernetes.io/projected/b71dd51e-ea47-45aa-a472-9dff7472e20c-kube-api-access-r2k8m\") pod \"managed-serviceaccount-addon-agent-84b8f8b644-lzp2z\" (UID: \"b71dd51e-ea47-45aa-a472-9dff7472e20c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84b8f8b644-lzp2z"
Jul 07 07:00:09.997743 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:09.997542    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3434ff8c-97ac-4c97-adcc-66049d141514-tmp\") pod \"klusterlet-addon-workmgr-55f6d89cf-fkpgt\" (UID: \"3434ff8c-97ac-4c97-adcc-66049d141514\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-55f6d89cf-fkpgt"
Jul 07 07:00:09.997743 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:09.997563    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/3434ff8c-97ac-4c97-adcc-66049d141514-klusterlet-config\") pod \"klusterlet-addon-workmgr-55f6d89cf-fkpgt\" (UID: \"3434ff8c-97ac-4c97-adcc-66049d141514\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-55f6d89cf-fkpgt"
Jul 07 07:00:09.997743 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:09.997598    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffkwb\" (UniqueName: \"kubernetes.io/projected/3434ff8c-97ac-4c97-adcc-66049d141514-kube-api-access-ffkwb\") pod \"klusterlet-addon-workmgr-55f6d89cf-fkpgt\" (UID: \"3434ff8c-97ac-4c97-adcc-66049d141514\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-55f6d89cf-fkpgt"
Jul 07 07:00:09.998047 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:09.998022    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3434ff8c-97ac-4c97-adcc-66049d141514-tmp\") pod \"klusterlet-addon-workmgr-55f6d89cf-fkpgt\" (UID: \"3434ff8c-97ac-4c97-adcc-66049d141514\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-55f6d89cf-fkpgt"
Jul 07 07:00:10.000354 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:10.000333    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/3434ff8c-97ac-4c97-adcc-66049d141514-klusterlet-config\") pod \"klusterlet-addon-workmgr-55f6d89cf-fkpgt\" (UID: \"3434ff8c-97ac-4c97-adcc-66049d141514\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-55f6d89cf-fkpgt"
Jul 07 07:00:10.000354 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:10.000343    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b71dd51e-ea47-45aa-a472-9dff7472e20c-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-84b8f8b644-lzp2z\" (UID: \"b71dd51e-ea47-45aa-a472-9dff7472e20c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84b8f8b644-lzp2z"
Jul 07 07:00:10.015321 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:10.015295    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffkwb\" (UniqueName: \"kubernetes.io/projected/3434ff8c-97ac-4c97-adcc-66049d141514-kube-api-access-ffkwb\") pod \"klusterlet-addon-workmgr-55f6d89cf-fkpgt\" (UID: \"3434ff8c-97ac-4c97-adcc-66049d141514\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-55f6d89cf-fkpgt"
Jul 07 07:00:10.015478 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:10.015450    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2k8m\" (UniqueName: \"kubernetes.io/projected/b71dd51e-ea47-45aa-a472-9dff7472e20c-kube-api-access-r2k8m\") pod \"managed-serviceaccount-addon-agent-84b8f8b644-lzp2z\" (UID: \"b71dd51e-ea47-45aa-a472-9dff7472e20c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84b8f8b644-lzp2z"
Jul 07 07:00:10.064949 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:10.064916    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84b8f8b644-lzp2z"
Jul 07 07:00:10.093476 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:10.093442    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-55f6d89cf-fkpgt"
Jul 07 07:00:11.131576 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:11.131550    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-55f6d89cf-fkpgt"]
Jul 07 07:00:11.132877 ip-10-0-138-97 kubenswrapper[2579]: W0707 07:00:11.132852    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3434ff8c_97ac_4c97_adcc_66049d141514.slice/crio-811820274d2c39df8e44c368f979bf32591269b3b75ee267f910bc41a57235c1 WatchSource:0}: Error finding container 811820274d2c39df8e44c368f979bf32591269b3b75ee267f910bc41a57235c1: Status 404 returned error can't find the container with id 811820274d2c39df8e44c368f979bf32591269b3b75ee267f910bc41a57235c1
Jul 07 07:00:11.153359 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:11.153331    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84b8f8b644-lzp2z"]
Jul 07 07:00:11.155612 ip-10-0-138-97 kubenswrapper[2579]: W0707 07:00:11.155586    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb71dd51e_ea47_45aa_a472_9dff7472e20c.slice/crio-eb1641eb227063359edb2cb1a653beefa8f783b215cc1988bb34b8b74f5ce69e WatchSource:0}: Error finding container eb1641eb227063359edb2cb1a653beefa8f783b215cc1988bb34b8b74f5ce69e: Status 404 returned error can't find the container with id eb1641eb227063359edb2cb1a653beefa8f783b215cc1988bb34b8b74f5ce69e
Jul 07 07:00:11.401501 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:11.401408    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84b8f8b644-lzp2z" event={"ID":"b71dd51e-ea47-45aa-a472-9dff7472e20c","Type":"ContainerStarted","Data":"eb1641eb227063359edb2cb1a653beefa8f783b215cc1988bb34b8b74f5ce69e"}
Jul 07 07:00:11.403446 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:11.403409    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-brh4b" event={"ID":"a852d8bb-79ad-4ead-8002-8e51919be749","Type":"ContainerStarted","Data":"536d4671aaf23bfcf94fa8c3bfe16c0b58bddf3994f2792532bfc56d454b1ecd"}
Jul 07 07:00:11.404394 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:11.404373    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-55f6d89cf-fkpgt" event={"ID":"3434ff8c-97ac-4c97-adcc-66049d141514","Type":"ContainerStarted","Data":"811820274d2c39df8e44c368f979bf32591269b3b75ee267f910bc41a57235c1"}
Jul 07 07:00:11.429151 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:11.429102    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-brh4b" podStartSLOduration=16.954128492 podStartE2EDuration="24.42908788s" totalImagesPullingTime="7.474959388s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 06:59:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 07:00:11.428670581 +0000 UTC m=+90.972432062" watchObservedRunningTime="2026-07-07 07:00:11.42908788 +0000 UTC m=+90.972849361"
Jul 07 07:00:16.423165 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:16.423126    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84b8f8b644-lzp2z" event={"ID":"b71dd51e-ea47-45aa-a472-9dff7472e20c","Type":"ContainerStarted","Data":"2815397a9eb91b6f83188191314c98cdd9bc72a37b0afa481f2d192d7a8bd55d"}
Jul 07 07:00:16.424431 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:16.424406    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-55f6d89cf-fkpgt" event={"ID":"3434ff8c-97ac-4c97-adcc-66049d141514","Type":"ContainerStarted","Data":"45072a68b34c17e1a66f1d74f211250fe61bde616bfd6009d71e9d4c48311953"}
Jul 07 07:00:16.424540 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:16.424515    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-55f6d89cf-fkpgt"
Jul 07 07:00:16.426051 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:16.426031    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-55f6d89cf-fkpgt"
Jul 07 07:00:16.441991 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:16.441935    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84b8f8b644-lzp2z" podStartSLOduration=2.747717535 podStartE2EDuration="7.441922674s" totalImagesPullingTime="4.694205139s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 07:00:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 07:00:16.440396788 +0000 UTC m=+95.984158269" watchObservedRunningTime="2026-07-07 07:00:16.441922674 +0000 UTC m=+95.985684155"
Jul 07 07:00:16.462524 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:16.462475    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-55f6d89cf-fkpgt" podStartSLOduration=2.729385701 podStartE2EDuration="7.462461204s" totalImagesPullingTime="4.733075503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 07:00:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 07:00:16.461483444 +0000 UTC m=+96.005244926" watchObservedRunningTime="2026-07-07 07:00:16.462461204 +0000 UTC m=+96.006222686"
Jul 07 07:00:18.320821 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:18.320786    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-nfjzx"
Jul 07 07:00:20.278628 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:20.278583    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8b86dea1-145b-4f19-b0f8-ae53d29815bb-registry-tls\") pod \"image-registry-65b55697fd-4z2qz\" (UID: \"8b86dea1-145b-4f19-b0f8-ae53d29815bb\") " pod="openshift-image-registry/image-registry-65b55697fd-4z2qz"
Jul 07 07:00:20.279113 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:20.278642    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4fb6f6e3-b570-408c-aaaa-a77d5bea0b24-metrics-tls\") pod \"dns-default-hpwxr\" (UID: \"4fb6f6e3-b570-408c-aaaa-a77d5bea0b24\") " pod="openshift-dns/dns-default-hpwxr"
Jul 07 07:00:20.280971 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:20.280930    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4fb6f6e3-b570-408c-aaaa-a77d5bea0b24-metrics-tls\") pod \"dns-default-hpwxr\" (UID: \"4fb6f6e3-b570-408c-aaaa-a77d5bea0b24\") " pod="openshift-dns/dns-default-hpwxr"
Jul 07 07:00:20.281135 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:20.281112    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8b86dea1-145b-4f19-b0f8-ae53d29815bb-registry-tls\") pod \"image-registry-65b55697fd-4z2qz\" (UID: \"8b86dea1-145b-4f19-b0f8-ae53d29815bb\") " pod="openshift-image-registry/image-registry-65b55697fd-4z2qz"
Jul 07 07:00:20.508780 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:20.508750    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-7j9bg\""
Jul 07 07:00:20.516198 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:20.516183    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65b55697fd-4z2qz"
Jul 07 07:00:20.528053 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:20.528029    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vfjqn\""
Jul 07 07:00:20.536235 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:20.536191    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hpwxr"
Jul 07 07:00:20.705623 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:20.705600    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hpwxr"]
Jul 07 07:00:20.707503 ip-10-0-138-97 kubenswrapper[2579]: W0707 07:00:20.707474    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fb6f6e3_b570_408c_aaaa_a77d5bea0b24.slice/crio-c34c1fe24d8f047676ce77cbafbae37f6f665414671f781449901c49fb391569 WatchSource:0}: Error finding container c34c1fe24d8f047676ce77cbafbae37f6f665414671f781449901c49fb391569: Status 404 returned error can't find the container with id c34c1fe24d8f047676ce77cbafbae37f6f665414671f781449901c49fb391569
Jul 07 07:00:20.709603 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:20.709580    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-65b55697fd-4z2qz"]
Jul 07 07:00:20.712520 ip-10-0-138-97 kubenswrapper[2579]: W0707 07:00:20.712490    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b86dea1_145b_4f19_b0f8_ae53d29815bb.slice/crio-913d786f77ce72006ece3099fee753ad4fed09af5f439b225e17cd4df5ba5e28 WatchSource:0}: Error finding container 913d786f77ce72006ece3099fee753ad4fed09af5f439b225e17cd4df5ba5e28: Status 404 returned error can't find the container with id 913d786f77ce72006ece3099fee753ad4fed09af5f439b225e17cd4df5ba5e28
Jul 07 07:00:21.440630 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:21.440596    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65b55697fd-4z2qz" event={"ID":"8b86dea1-145b-4f19-b0f8-ae53d29815bb","Type":"ContainerStarted","Data":"37d2a8efbcc4e0d2a0fc735ab0bf8bdec7407a5bd26f33eed60dfffd113023f8"}
Jul 07 07:00:21.441081 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:21.440652    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-65b55697fd-4z2qz"
Jul 07 07:00:21.441081 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:21.440667    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65b55697fd-4z2qz" event={"ID":"8b86dea1-145b-4f19-b0f8-ae53d29815bb","Type":"ContainerStarted","Data":"913d786f77ce72006ece3099fee753ad4fed09af5f439b225e17cd4df5ba5e28"}
Jul 07 07:00:21.441688 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:21.441668    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hpwxr" event={"ID":"4fb6f6e3-b570-408c-aaaa-a77d5bea0b24","Type":"ContainerStarted","Data":"c34c1fe24d8f047676ce77cbafbae37f6f665414671f781449901c49fb391569"}
Jul 07 07:00:21.468431 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:21.468387    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-65b55697fd-4z2qz" podStartSLOduration=99.468373383 podStartE2EDuration="1m39.468373383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 06:58:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 07:00:21.467236799 +0000 UTC m=+101.010998281" watchObservedRunningTime="2026-07-07 07:00:21.468373383 +0000 UTC m=+101.012134846"
Jul 07 07:00:23.449756 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:23.449666    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hpwxr" event={"ID":"4fb6f6e3-b570-408c-aaaa-a77d5bea0b24","Type":"ContainerStarted","Data":"36d84629676b6bed5a6da683605f3247a024607bbf09a902cbff606a0c94bf60"}
Jul 07 07:00:23.449756 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:23.449731    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hpwxr" event={"ID":"4fb6f6e3-b570-408c-aaaa-a77d5bea0b24","Type":"ContainerStarted","Data":"ee4f4679e9b27119e3accf661808d7e896b8c4f5cd8ad6fb54eb898f02645489"}
Jul 07 07:00:23.450191 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:23.449823    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-hpwxr"
Jul 07 07:00:23.470794 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:23.470751    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-hpwxr" podStartSLOduration=65.074368277 podStartE2EDuration="1m7.470737726s" totalImagesPullingTime="2.396369449s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 06:59:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 07:00:23.470117233 +0000 UTC m=+103.013878714" watchObservedRunningTime="2026-07-07 07:00:23.470737726 +0000 UTC m=+103.014499207"
Jul 07 07:00:23.953096 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:23.953063    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6b76b64dbb-fg77q"]
Jul 07 07:00:23.955018 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:23.955000    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b76b64dbb-fg77q"
Jul 07 07:00:23.957786 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:23.957765    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 07 07:00:23.961638 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:23.961573    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 07 07:00:23.961638 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:23.961582    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 07 07:00:23.961780 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:23.961572    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 07 07:00:23.961910 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:23.961889    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 07 07:00:23.962045 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:23.961912    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-jh5lq\""
Jul 07 07:00:23.962045 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:23.961915    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 07 07:00:23.962045 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:23.961916    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 07 07:00:23.964270 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:23.964252    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 07 07:00:23.977904 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:23.977881    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b76b64dbb-fg77q"]
Jul 07 07:00:24.001522 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:24.001492    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f1df7ac9-1934-4203-a6a9-3d2db27fb21b-oauth-serving-cert\") pod \"console-6b76b64dbb-fg77q\" (UID: \"f1df7ac9-1934-4203-a6a9-3d2db27fb21b\") " pod="openshift-console/console-6b76b64dbb-fg77q"
Jul 07 07:00:24.001663 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:24.001528    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2zhq\" (UniqueName: \"kubernetes.io/projected/f1df7ac9-1934-4203-a6a9-3d2db27fb21b-kube-api-access-w2zhq\") pod \"console-6b76b64dbb-fg77q\" (UID: \"f1df7ac9-1934-4203-a6a9-3d2db27fb21b\") " pod="openshift-console/console-6b76b64dbb-fg77q"
Jul 07 07:00:24.001663 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:24.001548    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f1df7ac9-1934-4203-a6a9-3d2db27fb21b-console-serving-cert\") pod \"console-6b76b64dbb-fg77q\" (UID: \"f1df7ac9-1934-4203-a6a9-3d2db27fb21b\") " pod="openshift-console/console-6b76b64dbb-fg77q"
Jul 07 07:00:24.001663 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:24.001578    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1df7ac9-1934-4203-a6a9-3d2db27fb21b-trusted-ca-bundle\") pod \"console-6b76b64dbb-fg77q\" (UID: \"f1df7ac9-1934-4203-a6a9-3d2db27fb21b\") " pod="openshift-console/console-6b76b64dbb-fg77q"
Jul 07 07:00:24.001663 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:24.001649    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f1df7ac9-1934-4203-a6a9-3d2db27fb21b-console-config\") pod \"console-6b76b64dbb-fg77q\" (UID: \"f1df7ac9-1934-4203-a6a9-3d2db27fb21b\") " pod="openshift-console/console-6b76b64dbb-fg77q"
Jul 07 07:00:24.001794 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:24.001684    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f1df7ac9-1934-4203-a6a9-3d2db27fb21b-service-ca\") pod \"console-6b76b64dbb-fg77q\" (UID: \"f1df7ac9-1934-4203-a6a9-3d2db27fb21b\") " pod="openshift-console/console-6b76b64dbb-fg77q"
Jul 07 07:00:24.001794 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:24.001720    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f1df7ac9-1934-4203-a6a9-3d2db27fb21b-console-oauth-config\") pod \"console-6b76b64dbb-fg77q\" (UID: \"f1df7ac9-1934-4203-a6a9-3d2db27fb21b\") " pod="openshift-console/console-6b76b64dbb-fg77q"
Jul 07 07:00:24.102915 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:24.102876    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f1df7ac9-1934-4203-a6a9-3d2db27fb21b-console-oauth-config\") pod \"console-6b76b64dbb-fg77q\" (UID: \"f1df7ac9-1934-4203-a6a9-3d2db27fb21b\") " pod="openshift-console/console-6b76b64dbb-fg77q"
Jul 07 07:00:24.102915 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:24.102919    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f1df7ac9-1934-4203-a6a9-3d2db27fb21b-oauth-serving-cert\") pod \"console-6b76b64dbb-fg77q\" (UID: \"f1df7ac9-1934-4203-a6a9-3d2db27fb21b\") " pod="openshift-console/console-6b76b64dbb-fg77q"
Jul 07 07:00:24.103093 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:24.102943    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2zhq\" (UniqueName: \"kubernetes.io/projected/f1df7ac9-1934-4203-a6a9-3d2db27fb21b-kube-api-access-w2zhq\") pod \"console-6b76b64dbb-fg77q\" (UID: \"f1df7ac9-1934-4203-a6a9-3d2db27fb21b\") " pod="openshift-console/console-6b76b64dbb-fg77q"
Jul 07 07:00:24.103093 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:24.103020    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f1df7ac9-1934-4203-a6a9-3d2db27fb21b-console-serving-cert\") pod \"console-6b76b64dbb-fg77q\" (UID: \"f1df7ac9-1934-4203-a6a9-3d2db27fb21b\") " pod="openshift-console/console-6b76b64dbb-fg77q"
Jul 07 07:00:24.103093 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:24.103054    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1df7ac9-1934-4203-a6a9-3d2db27fb21b-trusted-ca-bundle\") pod \"console-6b76b64dbb-fg77q\" (UID: \"f1df7ac9-1934-4203-a6a9-3d2db27fb21b\") " pod="openshift-console/console-6b76b64dbb-fg77q"
Jul 07 07:00:24.103226 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:24.103103    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f1df7ac9-1934-4203-a6a9-3d2db27fb21b-console-config\") pod \"console-6b76b64dbb-fg77q\" (UID: \"f1df7ac9-1934-4203-a6a9-3d2db27fb21b\") " pod="openshift-console/console-6b76b64dbb-fg77q"
Jul 07 07:00:24.103226 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:24.103130    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f1df7ac9-1934-4203-a6a9-3d2db27fb21b-service-ca\") pod \"console-6b76b64dbb-fg77q\" (UID: \"f1df7ac9-1934-4203-a6a9-3d2db27fb21b\") " pod="openshift-console/console-6b76b64dbb-fg77q"
Jul 07 07:00:24.103826 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:24.103804    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f1df7ac9-1934-4203-a6a9-3d2db27fb21b-console-config\") pod \"console-6b76b64dbb-fg77q\" (UID: \"f1df7ac9-1934-4203-a6a9-3d2db27fb21b\") " pod="openshift-console/console-6b76b64dbb-fg77q"
Jul 07 07:00:24.103943 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:24.103846    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f1df7ac9-1934-4203-a6a9-3d2db27fb21b-service-ca\") pod \"console-6b76b64dbb-fg77q\" (UID: \"f1df7ac9-1934-4203-a6a9-3d2db27fb21b\") " pod="openshift-console/console-6b76b64dbb-fg77q"
Jul 07 07:00:24.103943 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:24.103860    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f1df7ac9-1934-4203-a6a9-3d2db27fb21b-oauth-serving-cert\") pod \"console-6b76b64dbb-fg77q\" (UID: \"f1df7ac9-1934-4203-a6a9-3d2db27fb21b\") " pod="openshift-console/console-6b76b64dbb-fg77q"
Jul 07 07:00:24.104070 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:24.103939    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1df7ac9-1934-4203-a6a9-3d2db27fb21b-trusted-ca-bundle\") pod \"console-6b76b64dbb-fg77q\" (UID: \"f1df7ac9-1934-4203-a6a9-3d2db27fb21b\") " pod="openshift-console/console-6b76b64dbb-fg77q"
Jul 07 07:00:24.105914 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:24.105892    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f1df7ac9-1934-4203-a6a9-3d2db27fb21b-console-serving-cert\") pod \"console-6b76b64dbb-fg77q\" (UID: \"f1df7ac9-1934-4203-a6a9-3d2db27fb21b\") " pod="openshift-console/console-6b76b64dbb-fg77q"
Jul 07 07:00:24.105914 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:24.105908    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f1df7ac9-1934-4203-a6a9-3d2db27fb21b-console-oauth-config\") pod \"console-6b76b64dbb-fg77q\" (UID: \"f1df7ac9-1934-4203-a6a9-3d2db27fb21b\") " pod="openshift-console/console-6b76b64dbb-fg77q"
Jul 07 07:00:24.113107 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:24.113086    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2zhq\" (UniqueName: \"kubernetes.io/projected/f1df7ac9-1934-4203-a6a9-3d2db27fb21b-kube-api-access-w2zhq\") pod \"console-6b76b64dbb-fg77q\" (UID: \"f1df7ac9-1934-4203-a6a9-3d2db27fb21b\") " pod="openshift-console/console-6b76b64dbb-fg77q"
Jul 07 07:00:24.263917 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:24.263880    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b76b64dbb-fg77q"
Jul 07 07:00:24.397500 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:24.395561    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b76b64dbb-fg77q"]
Jul 07 07:00:24.399461 ip-10-0-138-97 kubenswrapper[2579]: W0707 07:00:24.399416    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1df7ac9_1934_4203_a6a9_3d2db27fb21b.slice/crio-fc71b47dc885e8b3a89e415ad689557dc594f9bbf54e790bccfae1c4c1fd1fde WatchSource:0}: Error finding container fc71b47dc885e8b3a89e415ad689557dc594f9bbf54e790bccfae1c4c1fd1fde: Status 404 returned error can't find the container with id fc71b47dc885e8b3a89e415ad689557dc594f9bbf54e790bccfae1c4c1fd1fde
Jul 07 07:00:24.454512 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:24.454474    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b76b64dbb-fg77q" event={"ID":"f1df7ac9-1934-4203-a6a9-3d2db27fb21b","Type":"ContainerStarted","Data":"fc71b47dc885e8b3a89e415ad689557dc594f9bbf54e790bccfae1c4c1fd1fde"}
Jul 07 07:00:28.471273 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:28.471237    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b76b64dbb-fg77q" event={"ID":"f1df7ac9-1934-4203-a6a9-3d2db27fb21b","Type":"ContainerStarted","Data":"eb1ea754d98e7ec2522241e2adac3433217ab7b56de6b875da660455e27dc200"}
Jul 07 07:00:28.497761 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:28.497710    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6b76b64dbb-fg77q" podStartSLOduration=2.467456192 podStartE2EDuration="5.497694853s" totalImagesPullingTime="3.030238661s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 07:00:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 07:00:28.496745666 +0000 UTC m=+108.040507148" watchObservedRunningTime="2026-07-07 07:00:28.497694853 +0000 UTC m=+108.041456333"
Jul 07 07:00:30.946431 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:30.946399    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-szt4k"]
Jul 07 07:00:30.948468 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:30.948452    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-szt4k"
Jul 07 07:00:30.953360 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:30.953338    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 07 07:00:30.953473 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:30.953338    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 07 07:00:30.953840 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:30.953807    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 07 07:00:30.953840 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:30.953823    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 07 07:00:30.954007 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:30.953862    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-57xwt\""
Jul 07 07:00:30.954007 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:30.953868    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 07 07:00:30.962560 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:30.962540    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-szt4k"]
Jul 07 07:00:31.060375 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:31.060344    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk2f4\" (UniqueName: \"kubernetes.io/projected/e540ce11-a18b-4631-9d0f-0727a72c0a23-kube-api-access-zk2f4\") pod \"prometheus-operator-567f7b8b7c-szt4k\" (UID: \"e540ce11-a18b-4631-9d0f-0727a72c0a23\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-szt4k"
Jul 07 07:00:31.060533 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:31.060395    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e540ce11-a18b-4631-9d0f-0727a72c0a23-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-szt4k\" (UID: \"e540ce11-a18b-4631-9d0f-0727a72c0a23\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-szt4k"
Jul 07 07:00:31.060533 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:31.060429    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e540ce11-a18b-4631-9d0f-0727a72c0a23-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-szt4k\" (UID: \"e540ce11-a18b-4631-9d0f-0727a72c0a23\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-szt4k"
Jul 07 07:00:31.060533 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:31.060445    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e540ce11-a18b-4631-9d0f-0727a72c0a23-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-szt4k\" (UID: \"e540ce11-a18b-4631-9d0f-0727a72c0a23\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-szt4k"
Jul 07 07:00:31.160756 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:31.160720    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e540ce11-a18b-4631-9d0f-0727a72c0a23-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-szt4k\" (UID: \"e540ce11-a18b-4631-9d0f-0727a72c0a23\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-szt4k"
Jul 07 07:00:31.160756 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:31.160756    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e540ce11-a18b-4631-9d0f-0727a72c0a23-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-szt4k\" (UID: \"e540ce11-a18b-4631-9d0f-0727a72c0a23\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-szt4k"
Jul 07 07:00:31.161000 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:31.160794    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk2f4\" (UniqueName: \"kubernetes.io/projected/e540ce11-a18b-4631-9d0f-0727a72c0a23-kube-api-access-zk2f4\") pod \"prometheus-operator-567f7b8b7c-szt4k\" (UID: \"e540ce11-a18b-4631-9d0f-0727a72c0a23\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-szt4k"
Jul 07 07:00:31.161000 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:31.160824    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e540ce11-a18b-4631-9d0f-0727a72c0a23-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-szt4k\" (UID: \"e540ce11-a18b-4631-9d0f-0727a72c0a23\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-szt4k"
Jul 07 07:00:31.161525 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:31.161499    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e540ce11-a18b-4631-9d0f-0727a72c0a23-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-szt4k\" (UID: \"e540ce11-a18b-4631-9d0f-0727a72c0a23\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-szt4k"
Jul 07 07:00:31.163157 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:31.163128    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e540ce11-a18b-4631-9d0f-0727a72c0a23-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-szt4k\" (UID: \"e540ce11-a18b-4631-9d0f-0727a72c0a23\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-szt4k"
Jul 07 07:00:31.163249 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:31.163231    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e540ce11-a18b-4631-9d0f-0727a72c0a23-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-szt4k\" (UID: \"e540ce11-a18b-4631-9d0f-0727a72c0a23\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-szt4k"
Jul 07 07:00:31.173466 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:31.173444    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk2f4\" (UniqueName: \"kubernetes.io/projected/e540ce11-a18b-4631-9d0f-0727a72c0a23-kube-api-access-zk2f4\") pod \"prometheus-operator-567f7b8b7c-szt4k\" (UID: \"e540ce11-a18b-4631-9d0f-0727a72c0a23\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-szt4k"
Jul 07 07:00:31.256749 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:31.256720    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-szt4k"
Jul 07 07:00:31.394632 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:31.394602    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-szt4k"]
Jul 07 07:00:31.397745 ip-10-0-138-97 kubenswrapper[2579]: W0707 07:00:31.397713    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode540ce11_a18b_4631_9d0f_0727a72c0a23.slice/crio-e797c00fb8d31a123daefd9d117e7fe38d12eadff3b0fd00c2ddf4923c2e5952 WatchSource:0}: Error finding container e797c00fb8d31a123daefd9d117e7fe38d12eadff3b0fd00c2ddf4923c2e5952: Status 404 returned error can't find the container with id e797c00fb8d31a123daefd9d117e7fe38d12eadff3b0fd00c2ddf4923c2e5952
Jul 07 07:00:31.481366 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:31.481333    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-szt4k" event={"ID":"e540ce11-a18b-4631-9d0f-0727a72c0a23","Type":"ContainerStarted","Data":"e797c00fb8d31a123daefd9d117e7fe38d12eadff3b0fd00c2ddf4923c2e5952"}
Jul 07 07:00:32.614470 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:32.614431    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-65b55697fd-4z2qz"]
Jul 07 07:00:33.456571 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:33.456547    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-hpwxr"
Jul 07 07:00:33.491184 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:33.491149    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-szt4k" event={"ID":"e540ce11-a18b-4631-9d0f-0727a72c0a23","Type":"ContainerStarted","Data":"9bc7ed7f0a5150e619fad680efcf0c942fdc8b8686a817d5cbd07ad1e6493038"}
Jul 07 07:00:33.491340 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:33.491209    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-szt4k" event={"ID":"e540ce11-a18b-4631-9d0f-0727a72c0a23","Type":"ContainerStarted","Data":"b4afb76724b681b46375f3b9e315270c359b7dcc17c3d2b2bd54148943ebf319"}
Jul 07 07:00:33.512386 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:33.512342    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-szt4k" podStartSLOduration=1.9466554710000001 podStartE2EDuration="3.512327382s" totalImagesPullingTime="1.565671911s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 07:00:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 07:00:33.511194488 +0000 UTC m=+113.054955974" watchObservedRunningTime="2026-07-07 07:00:33.512327382 +0000 UTC m=+113.056088862"
Jul 07 07:00:34.265024 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:34.264991    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6b76b64dbb-fg77q"
Jul 07 07:00:34.265473 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:34.265291    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6b76b64dbb-fg77q"
Jul 07 07:00:34.269952 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:34.269929    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6b76b64dbb-fg77q"
Jul 07 07:00:34.498974 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:34.498934    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6b76b64dbb-fg77q"
Jul 07 07:00:35.453990 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:35.453946    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-bc9lh"]
Jul 07 07:00:35.457308 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:35.457291    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bc9lh"
Jul 07 07:00:35.460510 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:35.460488    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 07 07:00:35.461064 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:35.461041    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 07 07:00:35.461169 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:35.461069    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 07 07:00:35.461169 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:35.461041    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-bjwds\""
Jul 07 07:00:35.498020 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:35.497994    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2f31a127-490d-41ff-a0ab-f24c4aec47f3-sys\") pod \"node-exporter-bc9lh\" (UID: \"2f31a127-490d-41ff-a0ab-f24c4aec47f3\") " pod="openshift-monitoring/node-exporter-bc9lh"
Jul 07 07:00:35.498020 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:35.498023    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2f31a127-490d-41ff-a0ab-f24c4aec47f3-node-exporter-tls\") pod \"node-exporter-bc9lh\" (UID: \"2f31a127-490d-41ff-a0ab-f24c4aec47f3\") " pod="openshift-monitoring/node-exporter-bc9lh"
Jul 07 07:00:35.498186 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:35.498046    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2f31a127-490d-41ff-a0ab-f24c4aec47f3-root\") pod \"node-exporter-bc9lh\" (UID: \"2f31a127-490d-41ff-a0ab-f24c4aec47f3\") " pod="openshift-monitoring/node-exporter-bc9lh"
Jul 07 07:00:35.498186 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:35.498065    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2f31a127-490d-41ff-a0ab-f24c4aec47f3-node-exporter-textfile\") pod \"node-exporter-bc9lh\" (UID: \"2f31a127-490d-41ff-a0ab-f24c4aec47f3\") " pod="openshift-monitoring/node-exporter-bc9lh"
Jul 07 07:00:35.498186 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:35.498123    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2f31a127-490d-41ff-a0ab-f24c4aec47f3-node-exporter-accelerators-collector-config\") pod \"node-exporter-bc9lh\" (UID: \"2f31a127-490d-41ff-a0ab-f24c4aec47f3\") " pod="openshift-monitoring/node-exporter-bc9lh"
Jul 07 07:00:35.498288 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:35.498235    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj4tm\" (UniqueName: \"kubernetes.io/projected/2f31a127-490d-41ff-a0ab-f24c4aec47f3-kube-api-access-kj4tm\") pod \"node-exporter-bc9lh\" (UID: \"2f31a127-490d-41ff-a0ab-f24c4aec47f3\") " pod="openshift-monitoring/node-exporter-bc9lh"
Jul 07 07:00:35.498288 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:35.498261    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2f31a127-490d-41ff-a0ab-f24c4aec47f3-node-exporter-wtmp\") pod \"node-exporter-bc9lh\" (UID: \"2f31a127-490d-41ff-a0ab-f24c4aec47f3\") " pod="openshift-monitoring/node-exporter-bc9lh"
Jul 07 07:00:35.498288 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:35.498283    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2f31a127-490d-41ff-a0ab-f24c4aec47f3-metrics-client-ca\") pod \"node-exporter-bc9lh\" (UID: \"2f31a127-490d-41ff-a0ab-f24c4aec47f3\") " pod="openshift-monitoring/node-exporter-bc9lh"
Jul 07 07:00:35.498404 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:35.498299    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2f31a127-490d-41ff-a0ab-f24c4aec47f3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bc9lh\" (UID: \"2f31a127-490d-41ff-a0ab-f24c4aec47f3\") " pod="openshift-monitoring/node-exporter-bc9lh"
Jul 07 07:00:35.599523 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:35.599490    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2f31a127-490d-41ff-a0ab-f24c4aec47f3-sys\") pod \"node-exporter-bc9lh\" (UID: \"2f31a127-490d-41ff-a0ab-f24c4aec47f3\") " pod="openshift-monitoring/node-exporter-bc9lh"
Jul 07 07:00:35.599683 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:35.599537    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2f31a127-490d-41ff-a0ab-f24c4aec47f3-node-exporter-tls\") pod \"node-exporter-bc9lh\" (UID: \"2f31a127-490d-41ff-a0ab-f24c4aec47f3\") " pod="openshift-monitoring/node-exporter-bc9lh"
Jul 07 07:00:35.599683 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:35.599560    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2f31a127-490d-41ff-a0ab-f24c4aec47f3-root\") pod \"node-exporter-bc9lh\" (UID: \"2f31a127-490d-41ff-a0ab-f24c4aec47f3\") " pod="openshift-monitoring/node-exporter-bc9lh"
Jul 07 07:00:35.599683 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:35.599577    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2f31a127-490d-41ff-a0ab-f24c4aec47f3-node-exporter-textfile\") pod \"node-exporter-bc9lh\" (UID: \"2f31a127-490d-41ff-a0ab-f24c4aec47f3\") " pod="openshift-monitoring/node-exporter-bc9lh"
Jul 07 07:00:35.599683 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:35.599597    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2f31a127-490d-41ff-a0ab-f24c4aec47f3-node-exporter-accelerators-collector-config\") pod \"node-exporter-bc9lh\" (UID: \"2f31a127-490d-41ff-a0ab-f24c4aec47f3\") " pod="openshift-monitoring/node-exporter-bc9lh"
Jul 07 07:00:35.599683 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:35.599601    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2f31a127-490d-41ff-a0ab-f24c4aec47f3-sys\") pod \"node-exporter-bc9lh\" (UID: \"2f31a127-490d-41ff-a0ab-f24c4aec47f3\") " pod="openshift-monitoring/node-exporter-bc9lh"
Jul 07 07:00:35.599683 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:35.599635    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj4tm\" (UniqueName: \"kubernetes.io/projected/2f31a127-490d-41ff-a0ab-f24c4aec47f3-kube-api-access-kj4tm\") pod \"node-exporter-bc9lh\" (UID: \"2f31a127-490d-41ff-a0ab-f24c4aec47f3\") " pod="openshift-monitoring/node-exporter-bc9lh"
Jul 07 07:00:35.599683 ip-10-0-138-97 kubenswrapper[2579]: E0707 07:00:35.599662    2579 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 07 07:00:35.600051 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:35.599697    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2f31a127-490d-41ff-a0ab-f24c4aec47f3-root\") pod \"node-exporter-bc9lh\" (UID: \"2f31a127-490d-41ff-a0ab-f24c4aec47f3\") " pod="openshift-monitoring/node-exporter-bc9lh"
Jul 07 07:00:35.600051 ip-10-0-138-97 kubenswrapper[2579]: E0707 07:00:35.599731    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f31a127-490d-41ff-a0ab-f24c4aec47f3-node-exporter-tls podName:2f31a127-490d-41ff-a0ab-f24c4aec47f3 nodeName:}" failed. No retries permitted until 2026-07-07 07:00:36.099710741 +0000 UTC m=+115.643472217 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/2f31a127-490d-41ff-a0ab-f24c4aec47f3-node-exporter-tls") pod "node-exporter-bc9lh" (UID: "2f31a127-490d-41ff-a0ab-f24c4aec47f3") : secret "node-exporter-tls" not found
Jul 07 07:00:35.600051 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:35.599787    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2f31a127-490d-41ff-a0ab-f24c4aec47f3-node-exporter-wtmp\") pod \"node-exporter-bc9lh\" (UID: \"2f31a127-490d-41ff-a0ab-f24c4aec47f3\") " pod="openshift-monitoring/node-exporter-bc9lh"
Jul 07 07:00:35.600051 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:35.599859    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2f31a127-490d-41ff-a0ab-f24c4aec47f3-metrics-client-ca\") pod \"node-exporter-bc9lh\" (UID: \"2f31a127-490d-41ff-a0ab-f24c4aec47f3\") " pod="openshift-monitoring/node-exporter-bc9lh"
Jul 07 07:00:35.600051 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:35.599884    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2f31a127-490d-41ff-a0ab-f24c4aec47f3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bc9lh\" (UID: \"2f31a127-490d-41ff-a0ab-f24c4aec47f3\") " pod="openshift-monitoring/node-exporter-bc9lh"
Jul 07 07:00:35.600051 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:35.599948    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2f31a127-490d-41ff-a0ab-f24c4aec47f3-node-exporter-wtmp\") pod \"node-exporter-bc9lh\" (UID: \"2f31a127-490d-41ff-a0ab-f24c4aec47f3\") " pod="openshift-monitoring/node-exporter-bc9lh"
Jul 07 07:00:35.600363 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:35.600255    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2f31a127-490d-41ff-a0ab-f24c4aec47f3-node-exporter-accelerators-collector-config\") pod \"node-exporter-bc9lh\" (UID: \"2f31a127-490d-41ff-a0ab-f24c4aec47f3\") " pod="openshift-monitoring/node-exporter-bc9lh"
Jul 07 07:00:35.600363 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:35.600295    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2f31a127-490d-41ff-a0ab-f24c4aec47f3-node-exporter-textfile\") pod \"node-exporter-bc9lh\" (UID: \"2f31a127-490d-41ff-a0ab-f24c4aec47f3\") " pod="openshift-monitoring/node-exporter-bc9lh"
Jul 07 07:00:35.600492 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:35.600469    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2f31a127-490d-41ff-a0ab-f24c4aec47f3-metrics-client-ca\") pod \"node-exporter-bc9lh\" (UID: \"2f31a127-490d-41ff-a0ab-f24c4aec47f3\") " pod="openshift-monitoring/node-exporter-bc9lh"
Jul 07 07:00:35.602224 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:35.602207    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2f31a127-490d-41ff-a0ab-f24c4aec47f3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bc9lh\" (UID: \"2f31a127-490d-41ff-a0ab-f24c4aec47f3\") " pod="openshift-monitoring/node-exporter-bc9lh"
Jul 07 07:00:35.610178 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:35.610159    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj4tm\" (UniqueName: \"kubernetes.io/projected/2f31a127-490d-41ff-a0ab-f24c4aec47f3-kube-api-access-kj4tm\") pod \"node-exporter-bc9lh\" (UID: \"2f31a127-490d-41ff-a0ab-f24c4aec47f3\") " pod="openshift-monitoring/node-exporter-bc9lh"
Jul 07 07:00:36.104568 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:36.104534    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2f31a127-490d-41ff-a0ab-f24c4aec47f3-node-exporter-tls\") pod \"node-exporter-bc9lh\" (UID: \"2f31a127-490d-41ff-a0ab-f24c4aec47f3\") " pod="openshift-monitoring/node-exporter-bc9lh"
Jul 07 07:00:36.106861 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:36.106836    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2f31a127-490d-41ff-a0ab-f24c4aec47f3-node-exporter-tls\") pod \"node-exporter-bc9lh\" (UID: \"2f31a127-490d-41ff-a0ab-f24c4aec47f3\") " pod="openshift-monitoring/node-exporter-bc9lh"
Jul 07 07:00:36.366176 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:36.366095    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bc9lh"
Jul 07 07:00:36.379607 ip-10-0-138-97 kubenswrapper[2579]: W0707 07:00:36.379570    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f31a127_490d_41ff_a0ab_f24c4aec47f3.slice/crio-7aa05757633b25bb31aee3dd14a4cbb9d1161d341b9b7412b67c23555af48511 WatchSource:0}: Error finding container 7aa05757633b25bb31aee3dd14a4cbb9d1161d341b9b7412b67c23555af48511: Status 404 returned error can't find the container with id 7aa05757633b25bb31aee3dd14a4cbb9d1161d341b9b7412b67c23555af48511
Jul 07 07:00:36.502386 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:36.502342    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bc9lh" event={"ID":"2f31a127-490d-41ff-a0ab-f24c4aec47f3","Type":"ContainerStarted","Data":"7aa05757633b25bb31aee3dd14a4cbb9d1161d341b9b7412b67c23555af48511"}
Jul 07 07:00:38.511632 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:38.511599    2579 generic.go:356] "Generic (PLEG): container finished" podID="2f31a127-490d-41ff-a0ab-f24c4aec47f3" containerID="3d8f7341d49502773b01978133242acff268f8e38e87729a9fce0b799d04736a" exitCode=0
Jul 07 07:00:38.512161 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:38.511693    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bc9lh" event={"ID":"2f31a127-490d-41ff-a0ab-f24c4aec47f3","Type":"ContainerDied","Data":"3d8f7341d49502773b01978133242acff268f8e38e87729a9fce0b799d04736a"}
Jul 07 07:00:39.517027 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:39.516996    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bc9lh" event={"ID":"2f31a127-490d-41ff-a0ab-f24c4aec47f3","Type":"ContainerStarted","Data":"7eadba1f801a3ad459bb1b4697aa07bdaefc196043cccf238caf0fdb46d35642"}
Jul 07 07:00:39.517407 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:39.517050    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bc9lh" event={"ID":"2f31a127-490d-41ff-a0ab-f24c4aec47f3","Type":"ContainerStarted","Data":"77e1a946de3db1f75fb164ff708ff0f6eba237cf47ba2d1276149780459aa62f"}
Jul 07 07:00:39.541739 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:39.541691    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-bc9lh" podStartSLOduration=3.344694579 podStartE2EDuration="4.541673452s" totalImagesPullingTime="1.134585265s" totalInitContainerRuntime="62.393608ms" isStatefulPod=true podCreationTimestamp="2026-07-07 07:00:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 07:00:39.539759083 +0000 UTC m=+119.083520563" watchObservedRunningTime="2026-07-07 07:00:39.541673452 +0000 UTC m=+119.085434935"
Jul 07 07:00:41.612118 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.612081    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 07:00:41.629433 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.629399    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 07:00:41.632232 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.632202    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 07 07:00:41.632669 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.632640    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 07 07:00:41.634028 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.633989    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 07 07:00:41.634524 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.634502    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 07 07:00:41.635118 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.635096    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 07 07:00:41.635434 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.635410    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 07 07:00:41.635617 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.635598    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 07 07:00:41.635783 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.635762    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-5rkb9pqhmst8\""
Jul 07 07:00:41.636095 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.636075    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-ttls8\""
Jul 07 07:00:41.636223 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.636195    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 07 07:00:41.636466 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.636451    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 07 07:00:41.638584 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.638565    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 07 07:00:41.639327 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.639307    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 07 07:00:41.640049 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.640008    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 07 07:00:41.640183 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.640164    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 07 07:00:41.641465 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.641441    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 07:00:41.642717 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.642695    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 07 07:00:41.644401 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.644381    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 07 07:00:41.751307 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.751274    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/560793a0-f73a-4192-9898-82202b318714-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"560793a0-f73a-4192-9898-82202b318714\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 07:00:41.751307 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.751307    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/560793a0-f73a-4192-9898-82202b318714-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"560793a0-f73a-4192-9898-82202b318714\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 07:00:41.751524 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.751332    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/560793a0-f73a-4192-9898-82202b318714-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"560793a0-f73a-4192-9898-82202b318714\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 07:00:41.751524 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.751389    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/560793a0-f73a-4192-9898-82202b318714-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"560793a0-f73a-4192-9898-82202b318714\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 07:00:41.751524 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.751417    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/560793a0-f73a-4192-9898-82202b318714-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"560793a0-f73a-4192-9898-82202b318714\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 07:00:41.751524 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.751440    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/560793a0-f73a-4192-9898-82202b318714-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"560793a0-f73a-4192-9898-82202b318714\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 07:00:41.751524 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.751461    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/560793a0-f73a-4192-9898-82202b318714-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"560793a0-f73a-4192-9898-82202b318714\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 07:00:41.751524 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.751478    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/560793a0-f73a-4192-9898-82202b318714-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"560793a0-f73a-4192-9898-82202b318714\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 07:00:41.751725 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.751525    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/560793a0-f73a-4192-9898-82202b318714-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"560793a0-f73a-4192-9898-82202b318714\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 07:00:41.751725 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.751618    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/560793a0-f73a-4192-9898-82202b318714-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"560793a0-f73a-4192-9898-82202b318714\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 07:00:41.751725 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.751651    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/560793a0-f73a-4192-9898-82202b318714-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"560793a0-f73a-4192-9898-82202b318714\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 07:00:41.751725 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.751673    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb6wm\" (UniqueName: \"kubernetes.io/projected/560793a0-f73a-4192-9898-82202b318714-kube-api-access-vb6wm\") pod \"prometheus-k8s-0\" (UID: \"560793a0-f73a-4192-9898-82202b318714\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 07:00:41.751725 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.751706    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/560793a0-f73a-4192-9898-82202b318714-config\") pod \"prometheus-k8s-0\" (UID: \"560793a0-f73a-4192-9898-82202b318714\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 07:00:41.751884 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.751754    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/560793a0-f73a-4192-9898-82202b318714-config-out\") pod \"prometheus-k8s-0\" (UID: \"560793a0-f73a-4192-9898-82202b318714\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 07:00:41.751884 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.751769    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/560793a0-f73a-4192-9898-82202b318714-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"560793a0-f73a-4192-9898-82202b318714\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 07:00:41.751884 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.751786    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/560793a0-f73a-4192-9898-82202b318714-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"560793a0-f73a-4192-9898-82202b318714\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 07:00:41.751884 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.751810    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/560793a0-f73a-4192-9898-82202b318714-web-config\") pod \"prometheus-k8s-0\" (UID: \"560793a0-f73a-4192-9898-82202b318714\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 07:00:41.751884 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.751848    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/560793a0-f73a-4192-9898-82202b318714-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"560793a0-f73a-4192-9898-82202b318714\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 07:00:41.752077 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.751882    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/560793a0-f73a-4192-9898-82202b318714-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"560793a0-f73a-4192-9898-82202b318714\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 07:00:41.752077 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.751899    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/560793a0-f73a-4192-9898-82202b318714-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"560793a0-f73a-4192-9898-82202b318714\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 07:00:41.853270 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.853229    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/560793a0-f73a-4192-9898-82202b318714-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"560793a0-f73a-4192-9898-82202b318714\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 07:00:41.853270 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.853270    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/560793a0-f73a-4192-9898-82202b318714-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"560793a0-f73a-4192-9898-82202b318714\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 07:00:41.853519 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.853288    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/560793a0-f73a-4192-9898-82202b318714-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"560793a0-f73a-4192-9898-82202b318714\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 07:00:41.853519 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.853345    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/560793a0-f73a-4192-9898-82202b318714-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"560793a0-f73a-4192-9898-82202b318714\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 07:00:41.853519 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.853381    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/560793a0-f73a-4192-9898-82202b318714-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"560793a0-f73a-4192-9898-82202b318714\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 07:00:41.853519 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.853418    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/560793a0-f73a-4192-9898-82202b318714-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"560793a0-f73a-4192-9898-82202b318714\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 07:00:41.853519 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.853466    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/560793a0-f73a-4192-9898-82202b318714-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"560793a0-f73a-4192-9898-82202b318714\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 07:00:41.853519 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.853496    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/560793a0-f73a-4192-9898-82202b318714-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"560793a0-f73a-4192-9898-82202b318714\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 07:00:41.853817 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.853543    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/560793a0-f73a-4192-9898-82202b318714-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"560793a0-f73a-4192-9898-82202b318714\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 07:00:41.853817 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.853568    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/560793a0-f73a-4192-9898-82202b318714-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"560793a0-f73a-4192-9898-82202b318714\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 07:00:41.853817 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.853592    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/560793a0-f73a-4192-9898-82202b318714-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"560793a0-f73a-4192-9898-82202b318714\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 07:00:41.853817 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.853613    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/560793a0-f73a-4192-9898-82202b318714-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"560793a0-f73a-4192-9898-82202b318714\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 07:00:41.853817 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.853646    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/560793a0-f73a-4192-9898-82202b318714-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"560793a0-f73a-4192-9898-82202b318714\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 07:00:41.853817 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.853669    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/560793a0-f73a-4192-9898-82202b318714-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"560793a0-f73a-4192-9898-82202b318714\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 07:00:41.853817 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.853697    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb6wm\" (UniqueName: \"kubernetes.io/projected/560793a0-f73a-4192-9898-82202b318714-kube-api-access-vb6wm\") pod \"prometheus-k8s-0\" (UID: \"560793a0-f73a-4192-9898-82202b318714\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 07:00:41.853817 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.853719    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/560793a0-f73a-4192-9898-82202b318714-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"560793a0-f73a-4192-9898-82202b318714\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 07:00:41.853817 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.853768    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/560793a0-f73a-4192-9898-82202b318714-config\") pod \"prometheus-k8s-0\" (UID: \"560793a0-f73a-4192-9898-82202b318714\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 07:00:41.853817 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.853816    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/560793a0-f73a-4192-9898-82202b318714-config-out\") pod \"prometheus-k8s-0\" (UID: \"560793a0-f73a-4192-9898-82202b318714\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 07:00:41.854413 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.853843    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/560793a0-f73a-4192-9898-82202b318714-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"560793a0-f73a-4192-9898-82202b318714\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 07:00:41.854413 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.853871    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/560793a0-f73a-4192-9898-82202b318714-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"560793a0-f73a-4192-9898-82202b318714\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 07:00:41.854413 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.853899    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/560793a0-f73a-4192-9898-82202b318714-web-config\") pod \"prometheus-k8s-0\" (UID: \"560793a0-f73a-4192-9898-82202b318714\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 07:00:41.854413 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.854273    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/560793a0-f73a-4192-9898-82202b318714-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"560793a0-f73a-4192-9898-82202b318714\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 07:00:41.855518 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.854649    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/560793a0-f73a-4192-9898-82202b318714-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"560793a0-f73a-4192-9898-82202b318714\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 07:00:41.855518 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.855176    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/560793a0-f73a-4192-9898-82202b318714-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"560793a0-f73a-4192-9898-82202b318714\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 07:00:41.857560 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.856529    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/560793a0-f73a-4192-9898-82202b318714-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"560793a0-f73a-4192-9898-82202b318714\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 07:00:41.857560 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.856539    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/560793a0-f73a-4192-9898-82202b318714-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"560793a0-f73a-4192-9898-82202b318714\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 07:00:41.857560 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.856644    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/560793a0-f73a-4192-9898-82202b318714-web-config\") pod \"prometheus-k8s-0\" (UID: \"560793a0-f73a-4192-9898-82202b318714\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 07:00:41.857560 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.857202    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/560793a0-f73a-4192-9898-82202b318714-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"560793a0-f73a-4192-9898-82202b318714\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 07:00:41.858312 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.858287    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/560793a0-f73a-4192-9898-82202b318714-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"560793a0-f73a-4192-9898-82202b318714\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 07:00:41.858988 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.858947    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/560793a0-f73a-4192-9898-82202b318714-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"560793a0-f73a-4192-9898-82202b318714\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 07:00:41.859252 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.859229    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/560793a0-f73a-4192-9898-82202b318714-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"560793a0-f73a-4192-9898-82202b318714\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 07:00:41.859829 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.859802    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/560793a0-f73a-4192-9898-82202b318714-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"560793a0-f73a-4192-9898-82202b318714\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 07:00:41.859913 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.859884    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/560793a0-f73a-4192-9898-82202b318714-config\") pod \"prometheus-k8s-0\" (UID: \"560793a0-f73a-4192-9898-82202b318714\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 07:00:41.860014 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.859992    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/560793a0-f73a-4192-9898-82202b318714-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"560793a0-f73a-4192-9898-82202b318714\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 07:00:41.860109 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.860081    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/560793a0-f73a-4192-9898-82202b318714-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"560793a0-f73a-4192-9898-82202b318714\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 07:00:41.860223 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.860087    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/560793a0-f73a-4192-9898-82202b318714-config-out\") pod \"prometheus-k8s-0\" (UID: \"560793a0-f73a-4192-9898-82202b318714\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 07:00:41.860985 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.860943    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/560793a0-f73a-4192-9898-82202b318714-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"560793a0-f73a-4192-9898-82202b318714\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 07:00:41.861218 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.861195    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/560793a0-f73a-4192-9898-82202b318714-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"560793a0-f73a-4192-9898-82202b318714\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 07:00:41.861385 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.861367    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/560793a0-f73a-4192-9898-82202b318714-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"560793a0-f73a-4192-9898-82202b318714\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 07:00:41.869889 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.869837    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb6wm\" (UniqueName: \"kubernetes.io/projected/560793a0-f73a-4192-9898-82202b318714-kube-api-access-vb6wm\") pod \"prometheus-k8s-0\" (UID: \"560793a0-f73a-4192-9898-82202b318714\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 07:00:41.944825 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:41.944791    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 07:00:42.115690 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:42.115613    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 07:00:42.118336 ip-10-0-138-97 kubenswrapper[2579]: W0707 07:00:42.118301    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod560793a0_f73a_4192_9898_82202b318714.slice/crio-6d5143dc845d075f5b0357ceac3f677838334d62b211228e556237318b6713e3 WatchSource:0}: Error finding container 6d5143dc845d075f5b0357ceac3f677838334d62b211228e556237318b6713e3: Status 404 returned error can't find the container with id 6d5143dc845d075f5b0357ceac3f677838334d62b211228e556237318b6713e3
Jul 07 07:00:42.529088 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:42.529052    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"560793a0-f73a-4192-9898-82202b318714","Type":"ContainerStarted","Data":"6d5143dc845d075f5b0357ceac3f677838334d62b211228e556237318b6713e3"}
Jul 07 07:00:42.619862 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:42.619834    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-65b55697fd-4z2qz"
Jul 07 07:00:44.536809 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:44.536767    2579 generic.go:356] "Generic (PLEG): container finished" podID="560793a0-f73a-4192-9898-82202b318714" containerID="4541f76e1a23a8de15b92e43a75097212bc514f8f5f6395a4b841c44551917c9" exitCode=0
Jul 07 07:00:44.537282 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:44.536810    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"560793a0-f73a-4192-9898-82202b318714","Type":"ContainerDied","Data":"4541f76e1a23a8de15b92e43a75097212bc514f8f5f6395a4b841c44551917c9"}
Jul 07 07:00:48.554462 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:48.554430    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"560793a0-f73a-4192-9898-82202b318714","Type":"ContainerStarted","Data":"fdba9dcd42f737e097a426cc03be2df00d121c1f85edc52239dd16650130f2fa"}
Jul 07 07:00:48.554824 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:48.554503    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"560793a0-f73a-4192-9898-82202b318714","Type":"ContainerStarted","Data":"4ba05133997ed6e67292e8b805e20ecff4b3b2bbc095bd622d30efb0f15c2503"}
Jul 07 07:00:50.568944 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:50.568916    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"560793a0-f73a-4192-9898-82202b318714","Type":"ContainerStarted","Data":"ed3075c37c8bb068704c4f0f3afa20e22e7329e75b2d43b8e640e95543d46d6c"}
Jul 07 07:00:50.569344 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:50.568999    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"560793a0-f73a-4192-9898-82202b318714","Type":"ContainerStarted","Data":"b5d9e70cabffbdad7180859e08d0fb63d1ea81e28b9ec56a1240d04c6177e956"}
Jul 07 07:00:50.569344 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:50.569041    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"560793a0-f73a-4192-9898-82202b318714","Type":"ContainerStarted","Data":"3931efbf4b3128dd5316e6e12c3381df0ce6f9e30d961c0d7391b02cb766ab65"}
Jul 07 07:00:50.735346 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:50.735309    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0dba90e8-7ad4-46cb-bc27-718b6f644f5d-metrics-certs\") pod \"network-metrics-daemon-s9stz\" (UID: \"0dba90e8-7ad4-46cb-bc27-718b6f644f5d\") " pod="openshift-multus/network-metrics-daemon-s9stz"
Jul 07 07:00:50.737629 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:50.737604    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0dba90e8-7ad4-46cb-bc27-718b6f644f5d-metrics-certs\") pod \"network-metrics-daemon-s9stz\" (UID: \"0dba90e8-7ad4-46cb-bc27-718b6f644f5d\") " pod="openshift-multus/network-metrics-daemon-s9stz"
Jul 07 07:00:50.832219 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:50.832182    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b76b64dbb-fg77q"]
Jul 07 07:00:50.979774 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:50.979747    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-59nzl\""
Jul 07 07:00:50.987388 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:50.987337    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s9stz"
Jul 07 07:00:51.117604 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:51.117565    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-s9stz"]
Jul 07 07:00:51.120823 ip-10-0-138-97 kubenswrapper[2579]: W0707 07:00:51.120765    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dba90e8_7ad4_46cb_bc27_718b6f644f5d.slice/crio-c49ab9dc2d93da36295af2c197671381e7431a87a1540db9088c695e7bbbd4f5 WatchSource:0}: Error finding container c49ab9dc2d93da36295af2c197671381e7431a87a1540db9088c695e7bbbd4f5: Status 404 returned error can't find the container with id c49ab9dc2d93da36295af2c197671381e7431a87a1540db9088c695e7bbbd4f5
Jul 07 07:00:51.575884 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:51.575835    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"560793a0-f73a-4192-9898-82202b318714","Type":"ContainerStarted","Data":"651538c49eb0d8b02d3218bc5d542eb98ded8040767d9b6a9a6513cd4bf7b1db"}
Jul 07 07:00:51.577000 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:51.576975    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-s9stz" event={"ID":"0dba90e8-7ad4-46cb-bc27-718b6f644f5d","Type":"ContainerStarted","Data":"c49ab9dc2d93da36295af2c197671381e7431a87a1540db9088c695e7bbbd4f5"}
Jul 07 07:00:51.617325 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:51.617274    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.6697921300000003 podStartE2EDuration="10.617259339s" totalImagesPullingTime="6.922594239s" totalInitContainerRuntime="24.87297ms" isStatefulPod=false podCreationTimestamp="2026-07-07 07:00:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 07:00:51.615497272 +0000 UTC m=+131.159258753" watchObservedRunningTime="2026-07-07 07:00:51.617259339 +0000 UTC m=+131.161020820"
Jul 07 07:00:51.945611 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:51.945530    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 07:00:52.583299 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:52.583264    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-s9stz" event={"ID":"0dba90e8-7ad4-46cb-bc27-718b6f644f5d","Type":"ContainerStarted","Data":"e7df2a3f838c91a81eec2cd65474110b23ae9d217345c4653c30ca591c824cbe"}
Jul 07 07:00:53.589351 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:53.589319    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-s9stz" event={"ID":"0dba90e8-7ad4-46cb-bc27-718b6f644f5d","Type":"ContainerStarted","Data":"856ed095198afe20d8fdaf282d83c1fee8b412a4c42942f6d566c9a53ce0c6e9"}
Jul 07 07:00:53.611988 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:53.611927    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-s9stz" podStartSLOduration=130.264631667 podStartE2EDuration="2m11.611913157s" totalImagesPullingTime="1.34728149s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 06:58:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 07:00:53.610567891 +0000 UTC m=+133.154329371" watchObservedRunningTime="2026-07-07 07:00:53.611913157 +0000 UTC m=+133.155674638"
Jul 07 07:00:57.634266 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:57.634220    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-65b55697fd-4z2qz" podUID="8b86dea1-145b-4f19-b0f8-ae53d29815bb" containerName="registry" containerID="cri-o://37d2a8efbcc4e0d2a0fc735ab0bf8bdec7407a5bd26f33eed60dfffd113023f8" gracePeriod=30
Jul 07 07:00:57.885428 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:57.885374    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65b55697fd-4z2qz"
Jul 07 07:00:58.003056 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:58.003018    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8b86dea1-145b-4f19-b0f8-ae53d29815bb-registry-certificates\") pod \"8b86dea1-145b-4f19-b0f8-ae53d29815bb\" (UID: \"8b86dea1-145b-4f19-b0f8-ae53d29815bb\") "
Jul 07 07:00:58.003225 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:58.003077    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8b86dea1-145b-4f19-b0f8-ae53d29815bb-bound-sa-token\") pod \"8b86dea1-145b-4f19-b0f8-ae53d29815bb\" (UID: \"8b86dea1-145b-4f19-b0f8-ae53d29815bb\") "
Jul 07 07:00:58.003225 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:58.003096    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8b86dea1-145b-4f19-b0f8-ae53d29815bb-registry-tls\") pod \"8b86dea1-145b-4f19-b0f8-ae53d29815bb\" (UID: \"8b86dea1-145b-4f19-b0f8-ae53d29815bb\") "
Jul 07 07:00:58.003225 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:58.003117    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8mlhx\" (UniqueName: \"kubernetes.io/projected/8b86dea1-145b-4f19-b0f8-ae53d29815bb-kube-api-access-8mlhx\") pod \"8b86dea1-145b-4f19-b0f8-ae53d29815bb\" (UID: \"8b86dea1-145b-4f19-b0f8-ae53d29815bb\") "
Jul 07 07:00:58.003225 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:58.003158    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8b86dea1-145b-4f19-b0f8-ae53d29815bb-trusted-ca\") pod \"8b86dea1-145b-4f19-b0f8-ae53d29815bb\" (UID: \"8b86dea1-145b-4f19-b0f8-ae53d29815bb\") "
Jul 07 07:00:58.003225 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:58.003174    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8b86dea1-145b-4f19-b0f8-ae53d29815bb-ca-trust-extracted\") pod \"8b86dea1-145b-4f19-b0f8-ae53d29815bb\" (UID: \"8b86dea1-145b-4f19-b0f8-ae53d29815bb\") "
Jul 07 07:00:58.003225 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:58.003198    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8b86dea1-145b-4f19-b0f8-ae53d29815bb-installation-pull-secrets\") pod \"8b86dea1-145b-4f19-b0f8-ae53d29815bb\" (UID: \"8b86dea1-145b-4f19-b0f8-ae53d29815bb\") "
Jul 07 07:00:58.003481 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:58.003228    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8b86dea1-145b-4f19-b0f8-ae53d29815bb-image-registry-private-configuration\") pod \"8b86dea1-145b-4f19-b0f8-ae53d29815bb\" (UID: \"8b86dea1-145b-4f19-b0f8-ae53d29815bb\") "
Jul 07 07:00:58.003532 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:58.003484    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b86dea1-145b-4f19-b0f8-ae53d29815bb-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "8b86dea1-145b-4f19-b0f8-ae53d29815bb" (UID: "8b86dea1-145b-4f19-b0f8-ae53d29815bb"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 07:00:58.003847 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:58.003816    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b86dea1-145b-4f19-b0f8-ae53d29815bb-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "8b86dea1-145b-4f19-b0f8-ae53d29815bb" (UID: "8b86dea1-145b-4f19-b0f8-ae53d29815bb"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 07:00:58.005691 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:58.005666    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b86dea1-145b-4f19-b0f8-ae53d29815bb-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "8b86dea1-145b-4f19-b0f8-ae53d29815bb" (UID: "8b86dea1-145b-4f19-b0f8-ae53d29815bb"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 07:00:58.005782 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:58.005705    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b86dea1-145b-4f19-b0f8-ae53d29815bb-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "8b86dea1-145b-4f19-b0f8-ae53d29815bb" (UID: "8b86dea1-145b-4f19-b0f8-ae53d29815bb"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 07:00:58.005820 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:58.005808    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b86dea1-145b-4f19-b0f8-ae53d29815bb-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "8b86dea1-145b-4f19-b0f8-ae53d29815bb" (UID: "8b86dea1-145b-4f19-b0f8-ae53d29815bb"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 07:00:58.005863 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:58.005832    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b86dea1-145b-4f19-b0f8-ae53d29815bb-kube-api-access-8mlhx" (OuterVolumeSpecName: "kube-api-access-8mlhx") pod "8b86dea1-145b-4f19-b0f8-ae53d29815bb" (UID: "8b86dea1-145b-4f19-b0f8-ae53d29815bb"). InnerVolumeSpecName "kube-api-access-8mlhx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 07:00:58.006094 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:58.006071    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b86dea1-145b-4f19-b0f8-ae53d29815bb-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "8b86dea1-145b-4f19-b0f8-ae53d29815bb" (UID: "8b86dea1-145b-4f19-b0f8-ae53d29815bb"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 07:00:58.011761 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:58.011735    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b86dea1-145b-4f19-b0f8-ae53d29815bb-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "8b86dea1-145b-4f19-b0f8-ae53d29815bb" (UID: "8b86dea1-145b-4f19-b0f8-ae53d29815bb"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 07:00:58.104801 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:58.104748    2579 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8b86dea1-145b-4f19-b0f8-ae53d29815bb-bound-sa-token\") on node \"ip-10-0-138-97.ec2.internal\" DevicePath \"\""
Jul 07 07:00:58.104801 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:58.104794    2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8b86dea1-145b-4f19-b0f8-ae53d29815bb-registry-tls\") on node \"ip-10-0-138-97.ec2.internal\" DevicePath \"\""
Jul 07 07:00:58.104801 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:58.104804    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mlhx\" (UniqueName: \"kubernetes.io/projected/8b86dea1-145b-4f19-b0f8-ae53d29815bb-kube-api-access-8mlhx\") on node \"ip-10-0-138-97.ec2.internal\" DevicePath \"\""
Jul 07 07:00:58.104801 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:58.104814    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8b86dea1-145b-4f19-b0f8-ae53d29815bb-trusted-ca\") on node \"ip-10-0-138-97.ec2.internal\" DevicePath \"\""
Jul 07 07:00:58.104801 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:58.104823    2579 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8b86dea1-145b-4f19-b0f8-ae53d29815bb-ca-trust-extracted\") on node \"ip-10-0-138-97.ec2.internal\" DevicePath \"\""
Jul 07 07:00:58.105118 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:58.104831    2579 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8b86dea1-145b-4f19-b0f8-ae53d29815bb-installation-pull-secrets\") on node \"ip-10-0-138-97.ec2.internal\" DevicePath \"\""
Jul 07 07:00:58.105118 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:58.104841    2579 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8b86dea1-145b-4f19-b0f8-ae53d29815bb-image-registry-private-configuration\") on node \"ip-10-0-138-97.ec2.internal\" DevicePath \"\""
Jul 07 07:00:58.105118 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:58.104850    2579 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8b86dea1-145b-4f19-b0f8-ae53d29815bb-registry-certificates\") on node \"ip-10-0-138-97.ec2.internal\" DevicePath \"\""
Jul 07 07:00:58.610753 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:58.610719    2579 generic.go:356] "Generic (PLEG): container finished" podID="8b86dea1-145b-4f19-b0f8-ae53d29815bb" containerID="37d2a8efbcc4e0d2a0fc735ab0bf8bdec7407a5bd26f33eed60dfffd113023f8" exitCode=0
Jul 07 07:00:58.610924 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:58.610763    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65b55697fd-4z2qz" event={"ID":"8b86dea1-145b-4f19-b0f8-ae53d29815bb","Type":"ContainerDied","Data":"37d2a8efbcc4e0d2a0fc735ab0bf8bdec7407a5bd26f33eed60dfffd113023f8"}
Jul 07 07:00:58.610924 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:58.610771    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65b55697fd-4z2qz"
Jul 07 07:00:58.610924 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:58.610808    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65b55697fd-4z2qz" event={"ID":"8b86dea1-145b-4f19-b0f8-ae53d29815bb","Type":"ContainerDied","Data":"913d786f77ce72006ece3099fee753ad4fed09af5f439b225e17cd4df5ba5e28"}
Jul 07 07:00:58.610924 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:58.610839    2579 scope.go:117] "RemoveContainer" containerID="37d2a8efbcc4e0d2a0fc735ab0bf8bdec7407a5bd26f33eed60dfffd113023f8"
Jul 07 07:00:58.619065 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:58.619045    2579 scope.go:117] "RemoveContainer" containerID="37d2a8efbcc4e0d2a0fc735ab0bf8bdec7407a5bd26f33eed60dfffd113023f8"
Jul 07 07:00:58.619336 ip-10-0-138-97 kubenswrapper[2579]: E0707 07:00:58.619313    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37d2a8efbcc4e0d2a0fc735ab0bf8bdec7407a5bd26f33eed60dfffd113023f8\": container with ID starting with 37d2a8efbcc4e0d2a0fc735ab0bf8bdec7407a5bd26f33eed60dfffd113023f8 not found: ID does not exist" containerID="37d2a8efbcc4e0d2a0fc735ab0bf8bdec7407a5bd26f33eed60dfffd113023f8"
Jul 07 07:00:58.619416 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:58.619344    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37d2a8efbcc4e0d2a0fc735ab0bf8bdec7407a5bd26f33eed60dfffd113023f8"} err="failed to get container status \"37d2a8efbcc4e0d2a0fc735ab0bf8bdec7407a5bd26f33eed60dfffd113023f8\": rpc error: code = NotFound desc = could not find container \"37d2a8efbcc4e0d2a0fc735ab0bf8bdec7407a5bd26f33eed60dfffd113023f8\": container with ID starting with 37d2a8efbcc4e0d2a0fc735ab0bf8bdec7407a5bd26f33eed60dfffd113023f8 not found: ID does not exist"
Jul 07 07:00:58.635415 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:58.635390    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-65b55697fd-4z2qz"]
Jul 07 07:00:58.637066 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:58.637046    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-65b55697fd-4z2qz"]
Jul 07 07:00:58.966203 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:00:58.964530    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b86dea1-145b-4f19-b0f8-ae53d29815bb" path="/var/lib/kubelet/pods/8b86dea1-145b-4f19-b0f8-ae53d29815bb/volumes"
Jul 07 07:01:05.638405 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:01:05.638366    2579 generic.go:356] "Generic (PLEG): container finished" podID="565f2b59-eb10-4b8c-a103-8e2e52dcb823" containerID="40440345f29150189f2acb35def60eeb4e560e4bc733bf30f3a9d4269f1a67f4" exitCode=0
Jul 07 07:01:05.638775 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:01:05.638457    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-8h4xr" event={"ID":"565f2b59-eb10-4b8c-a103-8e2e52dcb823","Type":"ContainerDied","Data":"40440345f29150189f2acb35def60eeb4e560e4bc733bf30f3a9d4269f1a67f4"}
Jul 07 07:01:05.638775 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:01:05.638760    2579 scope.go:117] "RemoveContainer" containerID="40440345f29150189f2acb35def60eeb4e560e4bc733bf30f3a9d4269f1a67f4"
Jul 07 07:01:06.643415 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:01:06.643380    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-8h4xr" event={"ID":"565f2b59-eb10-4b8c-a103-8e2e52dcb823","Type":"ContainerStarted","Data":"2d56e0f10e249e55d9f943f99dba7150e67eff0e23821d170dad6544d49f5ca4"}
Jul 07 07:01:15.675894 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:01:15.675860    2579 generic.go:356] "Generic (PLEG): container finished" podID="deb1b79a-dcc1-49e3-aaaf-6fd19b663de1" containerID="25e06c3a05158a5a1929cbf640e4f05ed7561af1bf9a2e3a9d0fe329e375f670" exitCode=0
Jul 07 07:01:15.676268 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:01:15.675918    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-5d9gv" event={"ID":"deb1b79a-dcc1-49e3-aaaf-6fd19b663de1","Type":"ContainerDied","Data":"25e06c3a05158a5a1929cbf640e4f05ed7561af1bf9a2e3a9d0fe329e375f670"}
Jul 07 07:01:15.676268 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:01:15.676229    2579 scope.go:117] "RemoveContainer" containerID="25e06c3a05158a5a1929cbf640e4f05ed7561af1bf9a2e3a9d0fe329e375f670"
Jul 07 07:01:15.852621 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:01:15.852524    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-6b76b64dbb-fg77q" podUID="f1df7ac9-1934-4203-a6a9-3d2db27fb21b" containerName="console" containerID="cri-o://eb1ea754d98e7ec2522241e2adac3433217ab7b56de6b875da660455e27dc200" gracePeriod=15
Jul 07 07:01:16.094379 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:01:16.094353    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b76b64dbb-fg77q"
Jul 07 07:01:16.142923 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:01:16.142841    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f1df7ac9-1934-4203-a6a9-3d2db27fb21b-console-config\") pod \"f1df7ac9-1934-4203-a6a9-3d2db27fb21b\" (UID: \"f1df7ac9-1934-4203-a6a9-3d2db27fb21b\") "
Jul 07 07:01:16.142923 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:01:16.142889    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f1df7ac9-1934-4203-a6a9-3d2db27fb21b-console-serving-cert\") pod \"f1df7ac9-1934-4203-a6a9-3d2db27fb21b\" (UID: \"f1df7ac9-1934-4203-a6a9-3d2db27fb21b\") "
Jul 07 07:01:16.142923 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:01:16.142921    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f1df7ac9-1934-4203-a6a9-3d2db27fb21b-console-oauth-config\") pod \"f1df7ac9-1934-4203-a6a9-3d2db27fb21b\" (UID: \"f1df7ac9-1934-4203-a6a9-3d2db27fb21b\") "
Jul 07 07:01:16.143184 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:01:16.142979    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f1df7ac9-1934-4203-a6a9-3d2db27fb21b-oauth-serving-cert\") pod \"f1df7ac9-1934-4203-a6a9-3d2db27fb21b\" (UID: \"f1df7ac9-1934-4203-a6a9-3d2db27fb21b\") "
Jul 07 07:01:16.143184 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:01:16.143001    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1df7ac9-1934-4203-a6a9-3d2db27fb21b-trusted-ca-bundle\") pod \"f1df7ac9-1934-4203-a6a9-3d2db27fb21b\" (UID: \"f1df7ac9-1934-4203-a6a9-3d2db27fb21b\") "
Jul 07 07:01:16.143184 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:01:16.143027    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w2zhq\" (UniqueName: \"kubernetes.io/projected/f1df7ac9-1934-4203-a6a9-3d2db27fb21b-kube-api-access-w2zhq\") pod \"f1df7ac9-1934-4203-a6a9-3d2db27fb21b\" (UID: \"f1df7ac9-1934-4203-a6a9-3d2db27fb21b\") "
Jul 07 07:01:16.143184 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:01:16.143063    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f1df7ac9-1934-4203-a6a9-3d2db27fb21b-service-ca\") pod \"f1df7ac9-1934-4203-a6a9-3d2db27fb21b\" (UID: \"f1df7ac9-1934-4203-a6a9-3d2db27fb21b\") "
Jul 07 07:01:16.143378 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:01:16.143228    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1df7ac9-1934-4203-a6a9-3d2db27fb21b-console-config" (OuterVolumeSpecName: "console-config") pod "f1df7ac9-1934-4203-a6a9-3d2db27fb21b" (UID: "f1df7ac9-1934-4203-a6a9-3d2db27fb21b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 07:01:16.143478 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:01:16.143449    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1df7ac9-1934-4203-a6a9-3d2db27fb21b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "f1df7ac9-1934-4203-a6a9-3d2db27fb21b" (UID: "f1df7ac9-1934-4203-a6a9-3d2db27fb21b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 07:01:16.143538 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:01:16.143458    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1df7ac9-1934-4203-a6a9-3d2db27fb21b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f1df7ac9-1934-4203-a6a9-3d2db27fb21b" (UID: "f1df7ac9-1934-4203-a6a9-3d2db27fb21b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 07:01:16.143538 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:01:16.143498    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1df7ac9-1934-4203-a6a9-3d2db27fb21b-service-ca" (OuterVolumeSpecName: "service-ca") pod "f1df7ac9-1934-4203-a6a9-3d2db27fb21b" (UID: "f1df7ac9-1934-4203-a6a9-3d2db27fb21b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 07:01:16.145257 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:01:16.145232    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1df7ac9-1934-4203-a6a9-3d2db27fb21b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f1df7ac9-1934-4203-a6a9-3d2db27fb21b" (UID: "f1df7ac9-1934-4203-a6a9-3d2db27fb21b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 07:01:16.145369 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:01:16.145237    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1df7ac9-1934-4203-a6a9-3d2db27fb21b-kube-api-access-w2zhq" (OuterVolumeSpecName: "kube-api-access-w2zhq") pod "f1df7ac9-1934-4203-a6a9-3d2db27fb21b" (UID: "f1df7ac9-1934-4203-a6a9-3d2db27fb21b"). InnerVolumeSpecName "kube-api-access-w2zhq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 07:01:16.145369 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:01:16.145329    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1df7ac9-1934-4203-a6a9-3d2db27fb21b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f1df7ac9-1934-4203-a6a9-3d2db27fb21b" (UID: "f1df7ac9-1934-4203-a6a9-3d2db27fb21b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 07:01:16.243591 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:01:16.243554    2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f1df7ac9-1934-4203-a6a9-3d2db27fb21b-console-oauth-config\") on node \"ip-10-0-138-97.ec2.internal\" DevicePath \"\""
Jul 07 07:01:16.243591 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:01:16.243587    2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f1df7ac9-1934-4203-a6a9-3d2db27fb21b-oauth-serving-cert\") on node \"ip-10-0-138-97.ec2.internal\" DevicePath \"\""
Jul 07 07:01:16.243591 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:01:16.243599    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1df7ac9-1934-4203-a6a9-3d2db27fb21b-trusted-ca-bundle\") on node \"ip-10-0-138-97.ec2.internal\" DevicePath \"\""
Jul 07 07:01:16.243807 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:01:16.243607    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2zhq\" (UniqueName: \"kubernetes.io/projected/f1df7ac9-1934-4203-a6a9-3d2db27fb21b-kube-api-access-w2zhq\") on node \"ip-10-0-138-97.ec2.internal\" DevicePath \"\""
Jul 07 07:01:16.243807 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:01:16.243631    2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f1df7ac9-1934-4203-a6a9-3d2db27fb21b-service-ca\") on node \"ip-10-0-138-97.ec2.internal\" DevicePath \"\""
Jul 07 07:01:16.243807 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:01:16.243639    2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f1df7ac9-1934-4203-a6a9-3d2db27fb21b-console-config\") on node \"ip-10-0-138-97.ec2.internal\" DevicePath \"\""
Jul 07 07:01:16.243807 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:01:16.243646    2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f1df7ac9-1934-4203-a6a9-3d2db27fb21b-console-serving-cert\") on node \"ip-10-0-138-97.ec2.internal\" DevicePath \"\""
Jul 07 07:01:16.680109 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:01:16.680074    2579 generic.go:356] "Generic (PLEG): container finished" podID="f1df7ac9-1934-4203-a6a9-3d2db27fb21b" containerID="eb1ea754d98e7ec2522241e2adac3433217ab7b56de6b875da660455e27dc200" exitCode=0
Jul 07 07:01:16.680500 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:01:16.680122    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b76b64dbb-fg77q"
Jul 07 07:01:16.680500 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:01:16.680163    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b76b64dbb-fg77q" event={"ID":"f1df7ac9-1934-4203-a6a9-3d2db27fb21b","Type":"ContainerDied","Data":"eb1ea754d98e7ec2522241e2adac3433217ab7b56de6b875da660455e27dc200"}
Jul 07 07:01:16.680500 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:01:16.680221    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b76b64dbb-fg77q" event={"ID":"f1df7ac9-1934-4203-a6a9-3d2db27fb21b","Type":"ContainerDied","Data":"fc71b47dc885e8b3a89e415ad689557dc594f9bbf54e790bccfae1c4c1fd1fde"}
Jul 07 07:01:16.680500 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:01:16.680249    2579 scope.go:117] "RemoveContainer" containerID="eb1ea754d98e7ec2522241e2adac3433217ab7b56de6b875da660455e27dc200"
Jul 07 07:01:16.682135 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:01:16.682110    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-5d9gv" event={"ID":"deb1b79a-dcc1-49e3-aaaf-6fd19b663de1","Type":"ContainerStarted","Data":"469a10ec80222f525b42cc92b00c2a7745a3b53af60ba69466299f02abfe896d"}
Jul 07 07:01:16.688835 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:01:16.688819    2579 scope.go:117] "RemoveContainer" containerID="eb1ea754d98e7ec2522241e2adac3433217ab7b56de6b875da660455e27dc200"
Jul 07 07:01:16.689112 ip-10-0-138-97 kubenswrapper[2579]: E0707 07:01:16.689098    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb1ea754d98e7ec2522241e2adac3433217ab7b56de6b875da660455e27dc200\": container with ID starting with eb1ea754d98e7ec2522241e2adac3433217ab7b56de6b875da660455e27dc200 not found: ID does not exist" containerID="eb1ea754d98e7ec2522241e2adac3433217ab7b56de6b875da660455e27dc200"
Jul 07 07:01:16.689182 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:01:16.689118    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb1ea754d98e7ec2522241e2adac3433217ab7b56de6b875da660455e27dc200"} err="failed to get container status \"eb1ea754d98e7ec2522241e2adac3433217ab7b56de6b875da660455e27dc200\": rpc error: code = NotFound desc = could not find container \"eb1ea754d98e7ec2522241e2adac3433217ab7b56de6b875da660455e27dc200\": container with ID starting with eb1ea754d98e7ec2522241e2adac3433217ab7b56de6b875da660455e27dc200 not found: ID does not exist"
Jul 07 07:01:16.722105 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:01:16.722073    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b76b64dbb-fg77q"]
Jul 07 07:01:16.728533 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:01:16.728513    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6b76b64dbb-fg77q"]
Jul 07 07:01:16.963278 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:01:16.963201    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1df7ac9-1934-4203-a6a9-3d2db27fb21b" path="/var/lib/kubelet/pods/f1df7ac9-1934-4203-a6a9-3d2db27fb21b/volumes"
Jul 07 07:01:41.945129 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:01:41.945079    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 07:01:41.964594 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:01:41.964563    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 07:01:42.790310 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:01:42.790279    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 07:03:40.897253 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:03:40.897226    2579 kubelet.go:1648] "Image garbage collection succeeded"
Jul 07 07:04:10.414623 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:04:10.414589    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-wl4cp"]
Jul 07 07:04:10.420570 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:04:10.414895    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b86dea1-145b-4f19-b0f8-ae53d29815bb" containerName="registry"
Jul 07 07:04:10.420570 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:04:10.414908    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b86dea1-145b-4f19-b0f8-ae53d29815bb" containerName="registry"
Jul 07 07:04:10.420570 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:04:10.414925    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1df7ac9-1934-4203-a6a9-3d2db27fb21b" containerName="console"
Jul 07 07:04:10.420570 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:04:10.414932    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1df7ac9-1934-4203-a6a9-3d2db27fb21b" containerName="console"
Jul 07 07:04:10.420570 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:04:10.415002    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1df7ac9-1934-4203-a6a9-3d2db27fb21b" containerName="console"
Jul 07 07:04:10.420570 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:04:10.415010    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b86dea1-145b-4f19-b0f8-ae53d29815bb" containerName="registry"
Jul 07 07:04:10.420570 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:04:10.418307    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-wl4cp"
Jul 07 07:04:10.421616 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:04:10.421593    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 07 07:04:10.422765 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:04:10.422747    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 07 07:04:10.422765 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:04:10.422749    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-d92x8\""
Jul 07 07:04:10.428629 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:04:10.428607    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-wl4cp"]
Jul 07 07:04:10.510347 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:04:10.510308    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg2nr\" (UniqueName: \"kubernetes.io/projected/9268ea6c-bae5-4c1d-86e5-3e8a692e61c5-kube-api-access-xg2nr\") pod \"cert-manager-webhook-6486ffd997-wl4cp\" (UID: \"9268ea6c-bae5-4c1d-86e5-3e8a692e61c5\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wl4cp"
Jul 07 07:04:10.510347 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:04:10.510361    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9268ea6c-bae5-4c1d-86e5-3e8a692e61c5-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-wl4cp\" (UID: \"9268ea6c-bae5-4c1d-86e5-3e8a692e61c5\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wl4cp"
Jul 07 07:04:10.611396 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:04:10.611358    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg2nr\" (UniqueName: \"kubernetes.io/projected/9268ea6c-bae5-4c1d-86e5-3e8a692e61c5-kube-api-access-xg2nr\") pod \"cert-manager-webhook-6486ffd997-wl4cp\" (UID: \"9268ea6c-bae5-4c1d-86e5-3e8a692e61c5\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wl4cp"
Jul 07 07:04:10.611590 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:04:10.611408    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9268ea6c-bae5-4c1d-86e5-3e8a692e61c5-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-wl4cp\" (UID: \"9268ea6c-bae5-4c1d-86e5-3e8a692e61c5\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wl4cp"
Jul 07 07:04:10.623035 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:04:10.623006    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9268ea6c-bae5-4c1d-86e5-3e8a692e61c5-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-wl4cp\" (UID: \"9268ea6c-bae5-4c1d-86e5-3e8a692e61c5\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wl4cp"
Jul 07 07:04:10.623167 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:04:10.623076    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg2nr\" (UniqueName: \"kubernetes.io/projected/9268ea6c-bae5-4c1d-86e5-3e8a692e61c5-kube-api-access-xg2nr\") pod \"cert-manager-webhook-6486ffd997-wl4cp\" (UID: \"9268ea6c-bae5-4c1d-86e5-3e8a692e61c5\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wl4cp"
Jul 07 07:04:10.729473 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:04:10.729447    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-wl4cp"
Jul 07 07:04:10.863379 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:04:10.863352    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-wl4cp"]
Jul 07 07:04:10.865723 ip-10-0-138-97 kubenswrapper[2579]: W0707 07:04:10.865697    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9268ea6c_bae5_4c1d_86e5_3e8a692e61c5.slice/crio-0753f03bc29ad6583f48b345d749beafddcc18777bdb0d2b88d8345d5590ec47 WatchSource:0}: Error finding container 0753f03bc29ad6583f48b345d749beafddcc18777bdb0d2b88d8345d5590ec47: Status 404 returned error can't find the container with id 0753f03bc29ad6583f48b345d749beafddcc18777bdb0d2b88d8345d5590ec47
Jul 07 07:04:10.868081 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:04:10.868064    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 07:04:11.296446 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:04:11.296413    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-wl4cp" event={"ID":"9268ea6c-bae5-4c1d-86e5-3e8a692e61c5","Type":"ContainerStarted","Data":"0753f03bc29ad6583f48b345d749beafddcc18777bdb0d2b88d8345d5590ec47"}
Jul 07 07:04:14.311061 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:04:14.311028    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-wl4cp" event={"ID":"9268ea6c-bae5-4c1d-86e5-3e8a692e61c5","Type":"ContainerStarted","Data":"9a45b9043f923b2a406b17da6f16f9e023d3e29512c517e729197fbd43726ec4"}
Jul 07 07:04:14.311458 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:04:14.311084    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-wl4cp"
Jul 07 07:04:14.329669 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:04:14.329617    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-wl4cp" podStartSLOduration=1.392266802 podStartE2EDuration="4.329601309s" totalImagesPullingTime="2.937334507s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 07:04:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 07:04:14.328839092 +0000 UTC m=+333.872600573" watchObservedRunningTime="2026-07-07 07:04:14.329601309 +0000 UTC m=+333.873362790"
Jul 07 07:04:20.317285 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:04:20.317248    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-wl4cp"
Jul 07 07:04:41.548354 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:04:41.548270    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-dbf9cc589-qfksk"]
Jul 07 07:04:41.550420 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:04:41.550402    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-dbf9cc589-qfksk"
Jul 07 07:04:41.553175 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:04:41.553151    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 07 07:04:41.553175 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:04:41.553140    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 07 07:04:41.553435 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:04:41.553422    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 07 07:04:41.553528 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:04:41.553510    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-lpw58\""
Jul 07 07:04:41.555097 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:04:41.555080    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 07 07:04:41.564647 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:04:41.564620    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-dbf9cc589-qfksk"]
Jul 07 07:04:41.570273 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:04:41.570246    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/50a43470-6929-4b81-a4c8-fa225d250335-webhook-cert\") pod \"opendatahub-operator-controller-manager-dbf9cc589-qfksk\" (UID: \"50a43470-6929-4b81-a4c8-fa225d250335\") " pod="opendatahub/opendatahub-operator-controller-manager-dbf9cc589-qfksk"
Jul 07 07:04:41.570394 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:04:41.570306    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/50a43470-6929-4b81-a4c8-fa225d250335-manifests\") pod \"opendatahub-operator-controller-manager-dbf9cc589-qfksk\" (UID: \"50a43470-6929-4b81-a4c8-fa225d250335\") " pod="opendatahub/opendatahub-operator-controller-manager-dbf9cc589-qfksk"
Jul 07 07:04:41.570394 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:04:41.570328    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lks9j\" (UniqueName: \"kubernetes.io/projected/50a43470-6929-4b81-a4c8-fa225d250335-kube-api-access-lks9j\") pod \"opendatahub-operator-controller-manager-dbf9cc589-qfksk\" (UID: \"50a43470-6929-4b81-a4c8-fa225d250335\") " pod="opendatahub/opendatahub-operator-controller-manager-dbf9cc589-qfksk"
Jul 07 07:04:41.570394 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:04:41.570347    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50a43470-6929-4b81-a4c8-fa225d250335-tmp\") pod \"opendatahub-operator-controller-manager-dbf9cc589-qfksk\" (UID: \"50a43470-6929-4b81-a4c8-fa225d250335\") " pod="opendatahub/opendatahub-operator-controller-manager-dbf9cc589-qfksk"
Jul 07 07:04:41.570518 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:04:41.570405    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/50a43470-6929-4b81-a4c8-fa225d250335-apiservice-cert\") pod \"opendatahub-operator-controller-manager-dbf9cc589-qfksk\" (UID: \"50a43470-6929-4b81-a4c8-fa225d250335\") " pod="opendatahub/opendatahub-operator-controller-manager-dbf9cc589-qfksk"
Jul 07 07:04:41.671257 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:04:41.671220    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/50a43470-6929-4b81-a4c8-fa225d250335-webhook-cert\") pod \"opendatahub-operator-controller-manager-dbf9cc589-qfksk\" (UID: \"50a43470-6929-4b81-a4c8-fa225d250335\") " pod="opendatahub/opendatahub-operator-controller-manager-dbf9cc589-qfksk"
Jul 07 07:04:41.671451 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:04:41.671320    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/50a43470-6929-4b81-a4c8-fa225d250335-manifests\") pod \"opendatahub-operator-controller-manager-dbf9cc589-qfksk\" (UID: \"50a43470-6929-4b81-a4c8-fa225d250335\") " pod="opendatahub/opendatahub-operator-controller-manager-dbf9cc589-qfksk"
Jul 07 07:04:41.671451 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:04:41.671343    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lks9j\" (UniqueName: \"kubernetes.io/projected/50a43470-6929-4b81-a4c8-fa225d250335-kube-api-access-lks9j\") pod \"opendatahub-operator-controller-manager-dbf9cc589-qfksk\" (UID: \"50a43470-6929-4b81-a4c8-fa225d250335\") " pod="opendatahub/opendatahub-operator-controller-manager-dbf9cc589-qfksk"
Jul 07 07:04:41.671451 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:04:41.671363    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50a43470-6929-4b81-a4c8-fa225d250335-tmp\") pod \"opendatahub-operator-controller-manager-dbf9cc589-qfksk\" (UID: \"50a43470-6929-4b81-a4c8-fa225d250335\") " pod="opendatahub/opendatahub-operator-controller-manager-dbf9cc589-qfksk"
Jul 07 07:04:41.671451 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:04:41.671387    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/50a43470-6929-4b81-a4c8-fa225d250335-apiservice-cert\") pod \"opendatahub-operator-controller-manager-dbf9cc589-qfksk\" (UID: \"50a43470-6929-4b81-a4c8-fa225d250335\") " pod="opendatahub/opendatahub-operator-controller-manager-dbf9cc589-qfksk"
Jul 07 07:04:41.671839 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:04:41.671817    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/50a43470-6929-4b81-a4c8-fa225d250335-manifests\") pod \"opendatahub-operator-controller-manager-dbf9cc589-qfksk\" (UID: \"50a43470-6929-4b81-a4c8-fa225d250335\") " pod="opendatahub/opendatahub-operator-controller-manager-dbf9cc589-qfksk"
Jul 07 07:04:41.673702 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:04:41.673672    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50a43470-6929-4b81-a4c8-fa225d250335-tmp\") pod \"opendatahub-operator-controller-manager-dbf9cc589-qfksk\" (UID: \"50a43470-6929-4b81-a4c8-fa225d250335\") " pod="opendatahub/opendatahub-operator-controller-manager-dbf9cc589-qfksk"
Jul 07 07:04:41.673820 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:04:41.673809    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/50a43470-6929-4b81-a4c8-fa225d250335-webhook-cert\") pod \"opendatahub-operator-controller-manager-dbf9cc589-qfksk\" (UID: \"50a43470-6929-4b81-a4c8-fa225d250335\") " pod="opendatahub/opendatahub-operator-controller-manager-dbf9cc589-qfksk"
Jul 07 07:04:41.673986 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:04:41.673946    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/50a43470-6929-4b81-a4c8-fa225d250335-apiservice-cert\") pod \"opendatahub-operator-controller-manager-dbf9cc589-qfksk\" (UID: \"50a43470-6929-4b81-a4c8-fa225d250335\") " pod="opendatahub/opendatahub-operator-controller-manager-dbf9cc589-qfksk"
Jul 07 07:04:41.681742 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:04:41.681715    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lks9j\" (UniqueName: \"kubernetes.io/projected/50a43470-6929-4b81-a4c8-fa225d250335-kube-api-access-lks9j\") pod \"opendatahub-operator-controller-manager-dbf9cc589-qfksk\" (UID: \"50a43470-6929-4b81-a4c8-fa225d250335\") " pod="opendatahub/opendatahub-operator-controller-manager-dbf9cc589-qfksk"
Jul 07 07:04:41.861250 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:04:41.861157    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-dbf9cc589-qfksk"
Jul 07 07:04:42.037562 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:04:42.037526    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-dbf9cc589-qfksk"]
Jul 07 07:04:42.044004 ip-10-0-138-97 kubenswrapper[2579]: W0707 07:04:42.043973    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50a43470_6929_4b81_a4c8_fa225d250335.slice/crio-1a05e58c0b19835dd501f1f30786489f3be29a7b6be6744e7ad43cd03fe1ba96 WatchSource:0}: Error finding container 1a05e58c0b19835dd501f1f30786489f3be29a7b6be6744e7ad43cd03fe1ba96: Status 404 returned error can't find the container with id 1a05e58c0b19835dd501f1f30786489f3be29a7b6be6744e7ad43cd03fe1ba96
Jul 07 07:04:42.412553 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:04:42.412516    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-dbf9cc589-qfksk" event={"ID":"50a43470-6929-4b81-a4c8-fa225d250335","Type":"ContainerStarted","Data":"1a05e58c0b19835dd501f1f30786489f3be29a7b6be6744e7ad43cd03fe1ba96"}
Jul 07 07:04:45.427090 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:04:45.427005    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-dbf9cc589-qfksk" event={"ID":"50a43470-6929-4b81-a4c8-fa225d250335","Type":"ContainerStarted","Data":"a31ca74b7bbfa561672a84d64cd50f6e461548adbe73d1e00033585c5a8143d5"}
Jul 07 07:04:46.431900 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:04:46.431859    2579 generic.go:356] "Generic (PLEG): container finished" podID="50a43470-6929-4b81-a4c8-fa225d250335" containerID="a31ca74b7bbfa561672a84d64cd50f6e461548adbe73d1e00033585c5a8143d5" exitCode=0
Jul 07 07:04:46.432294 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:04:46.431948    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-dbf9cc589-qfksk" event={"ID":"50a43470-6929-4b81-a4c8-fa225d250335","Type":"ContainerDied","Data":"a31ca74b7bbfa561672a84d64cd50f6e461548adbe73d1e00033585c5a8143d5"}
Jul 07 07:04:47.437491 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:04:47.437401    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-dbf9cc589-qfksk" event={"ID":"50a43470-6929-4b81-a4c8-fa225d250335","Type":"ContainerStarted","Data":"d0b97edbec36ce48a92cba7ca4d2d56832c7c3ba5dd966e1485d9dd674a192ce"}
Jul 07 07:04:47.437840 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:04:47.437542    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-dbf9cc589-qfksk"
Jul 07 07:04:47.462781 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:04:47.462721    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-dbf9cc589-qfksk" podStartSLOduration=1.7357188159999999 podStartE2EDuration="6.462700217s" totalImagesPullingTime="3.663985183s" totalInitContainerRuntime="1.062996218s" isStatefulPod=false podCreationTimestamp="2026-07-07 07:04:41 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 07:04:47.462531155 +0000 UTC m=+367.006292639" watchObservedRunningTime="2026-07-07 07:04:47.462700217 +0000 UTC m=+367.006461700"
Jul 07 07:04:58.444128 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:04:58.444093    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-dbf9cc589-qfksk"
Jul 07 07:05:01.164154 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:05:01.164117    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-54bd8c98d7-jw7hc"]
Jul 07 07:05:01.169822 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:05:01.169796    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-54bd8c98d7-jw7hc"
Jul 07 07:05:01.174279 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:05:01.174253    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 07 07:05:01.174455 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:05:01.174425    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-sbczf\""
Jul 07 07:05:01.175909 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:05:01.175602    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 07 07:05:01.175909 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:05:01.175809    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 07 07:05:01.176101 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:05:01.175997    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 07 07:05:01.185065 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:05:01.185034    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-54bd8c98d7-jw7hc"]
Jul 07 07:05:01.237065 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:05:01.237034    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9e42bb72-5874-40c2-b79d-a24298bb8586-tmp\") pod \"kube-auth-proxy-54bd8c98d7-jw7hc\" (UID: \"9e42bb72-5874-40c2-b79d-a24298bb8586\") " pod="openshift-ingress/kube-auth-proxy-54bd8c98d7-jw7hc"
Jul 07 07:05:01.237257 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:05:01.237073    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e42bb72-5874-40c2-b79d-a24298bb8586-tls-certs\") pod \"kube-auth-proxy-54bd8c98d7-jw7hc\" (UID: \"9e42bb72-5874-40c2-b79d-a24298bb8586\") " pod="openshift-ingress/kube-auth-proxy-54bd8c98d7-jw7hc"
Jul 07 07:05:01.237257 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:05:01.237121    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvbrw\" (UniqueName: \"kubernetes.io/projected/9e42bb72-5874-40c2-b79d-a24298bb8586-kube-api-access-gvbrw\") pod \"kube-auth-proxy-54bd8c98d7-jw7hc\" (UID: \"9e42bb72-5874-40c2-b79d-a24298bb8586\") " pod="openshift-ingress/kube-auth-proxy-54bd8c98d7-jw7hc"
Jul 07 07:05:01.338343 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:05:01.338310    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvbrw\" (UniqueName: \"kubernetes.io/projected/9e42bb72-5874-40c2-b79d-a24298bb8586-kube-api-access-gvbrw\") pod \"kube-auth-proxy-54bd8c98d7-jw7hc\" (UID: \"9e42bb72-5874-40c2-b79d-a24298bb8586\") " pod="openshift-ingress/kube-auth-proxy-54bd8c98d7-jw7hc"
Jul 07 07:05:01.338540 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:05:01.338384    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9e42bb72-5874-40c2-b79d-a24298bb8586-tmp\") pod \"kube-auth-proxy-54bd8c98d7-jw7hc\" (UID: \"9e42bb72-5874-40c2-b79d-a24298bb8586\") " pod="openshift-ingress/kube-auth-proxy-54bd8c98d7-jw7hc"
Jul 07 07:05:01.338540 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:05:01.338421    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e42bb72-5874-40c2-b79d-a24298bb8586-tls-certs\") pod \"kube-auth-proxy-54bd8c98d7-jw7hc\" (UID: \"9e42bb72-5874-40c2-b79d-a24298bb8586\") " pod="openshift-ingress/kube-auth-proxy-54bd8c98d7-jw7hc"
Jul 07 07:05:01.338654 ip-10-0-138-97 kubenswrapper[2579]: E0707 07:05:01.338563    2579 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Jul 07 07:05:01.338654 ip-10-0-138-97 kubenswrapper[2579]: E0707 07:05:01.338637    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e42bb72-5874-40c2-b79d-a24298bb8586-tls-certs podName:9e42bb72-5874-40c2-b79d-a24298bb8586 nodeName:}" failed. No retries permitted until 2026-07-07 07:05:01.838613321 +0000 UTC m=+381.382374800 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/9e42bb72-5874-40c2-b79d-a24298bb8586-tls-certs") pod "kube-auth-proxy-54bd8c98d7-jw7hc" (UID: "9e42bb72-5874-40c2-b79d-a24298bb8586") : secret "kube-auth-proxy-tls" not found
Jul 07 07:05:01.341076 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:05:01.341049    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9e42bb72-5874-40c2-b79d-a24298bb8586-tmp\") pod \"kube-auth-proxy-54bd8c98d7-jw7hc\" (UID: \"9e42bb72-5874-40c2-b79d-a24298bb8586\") " pod="openshift-ingress/kube-auth-proxy-54bd8c98d7-jw7hc"
Jul 07 07:05:01.350533 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:05:01.350505    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvbrw\" (UniqueName: \"kubernetes.io/projected/9e42bb72-5874-40c2-b79d-a24298bb8586-kube-api-access-gvbrw\") pod \"kube-auth-proxy-54bd8c98d7-jw7hc\" (UID: \"9e42bb72-5874-40c2-b79d-a24298bb8586\") " pod="openshift-ingress/kube-auth-proxy-54bd8c98d7-jw7hc"
Jul 07 07:05:01.842803 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:05:01.842765    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e42bb72-5874-40c2-b79d-a24298bb8586-tls-certs\") pod \"kube-auth-proxy-54bd8c98d7-jw7hc\" (UID: \"9e42bb72-5874-40c2-b79d-a24298bb8586\") " pod="openshift-ingress/kube-auth-proxy-54bd8c98d7-jw7hc"
Jul 07 07:05:01.845328 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:05:01.845304    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e42bb72-5874-40c2-b79d-a24298bb8586-tls-certs\") pod \"kube-auth-proxy-54bd8c98d7-jw7hc\" (UID: \"9e42bb72-5874-40c2-b79d-a24298bb8586\") " pod="openshift-ingress/kube-auth-proxy-54bd8c98d7-jw7hc"
Jul 07 07:05:02.080911 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:05:02.080878    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-54bd8c98d7-jw7hc"
Jul 07 07:05:02.209884 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:05:02.209852    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5c6f759dd-gwbjb"]
Jul 07 07:05:02.212630 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:05:02.212612    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5c6f759dd-gwbjb"
Jul 07 07:05:02.216080 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:05:02.216054    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 07 07:05:02.216218 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:05:02.216125    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 07 07:05:02.216506 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:05:02.216490    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-pn7dq\""
Jul 07 07:05:02.217382 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:05:02.217324    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 07 07:05:02.217559 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:05:02.217402    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 07 07:05:02.218156 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:05:02.217973    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 07 07:05:02.219719 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:05:02.219698    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-54bd8c98d7-jw7hc"]
Jul 07 07:05:02.221148 ip-10-0-138-97 kubenswrapper[2579]: W0707 07:05:02.221115    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e42bb72_5874_40c2_b79d_a24298bb8586.slice/crio-2c7eef98c7c57f25d4c2b3908165800840f6290bbd9af066dd77eecebe9a7533 WatchSource:0}: Error finding container 2c7eef98c7c57f25d4c2b3908165800840f6290bbd9af066dd77eecebe9a7533: Status 404 returned error can't find the container with id 2c7eef98c7c57f25d4c2b3908165800840f6290bbd9af066dd77eecebe9a7533
Jul 07 07:05:02.231293 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:05:02.231269    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5c6f759dd-gwbjb"]
Jul 07 07:05:02.345828 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:05:02.345769    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/31a943d1-8ef0-4f15-9e3a-ba696db451b7-manager-config\") pod \"lws-controller-manager-5c6f759dd-gwbjb\" (UID: \"31a943d1-8ef0-4f15-9e3a-ba696db451b7\") " pod="openshift-lws-operator/lws-controller-manager-5c6f759dd-gwbjb"
Jul 07 07:05:02.345828 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:05:02.345807    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/31a943d1-8ef0-4f15-9e3a-ba696db451b7-cert\") pod \"lws-controller-manager-5c6f759dd-gwbjb\" (UID: \"31a943d1-8ef0-4f15-9e3a-ba696db451b7\") " pod="openshift-lws-operator/lws-controller-manager-5c6f759dd-gwbjb"
Jul 07 07:05:02.346101 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:05:02.345856    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzm9m\" (UniqueName: \"kubernetes.io/projected/31a943d1-8ef0-4f15-9e3a-ba696db451b7-kube-api-access-fzm9m\") pod \"lws-controller-manager-5c6f759dd-gwbjb\" (UID: \"31a943d1-8ef0-4f15-9e3a-ba696db451b7\") " pod="openshift-lws-operator/lws-controller-manager-5c6f759dd-gwbjb"
Jul 07 07:05:02.346101 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:05:02.345990    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/31a943d1-8ef0-4f15-9e3a-ba696db451b7-metrics-cert\") pod \"lws-controller-manager-5c6f759dd-gwbjb\" (UID: \"31a943d1-8ef0-4f15-9e3a-ba696db451b7\") " pod="openshift-lws-operator/lws-controller-manager-5c6f759dd-gwbjb"
Jul 07 07:05:02.446611 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:05:02.446518    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzm9m\" (UniqueName: \"kubernetes.io/projected/31a943d1-8ef0-4f15-9e3a-ba696db451b7-kube-api-access-fzm9m\") pod \"lws-controller-manager-5c6f759dd-gwbjb\" (UID: \"31a943d1-8ef0-4f15-9e3a-ba696db451b7\") " pod="openshift-lws-operator/lws-controller-manager-5c6f759dd-gwbjb"
Jul 07 07:05:02.446611 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:05:02.446582    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/31a943d1-8ef0-4f15-9e3a-ba696db451b7-metrics-cert\") pod \"lws-controller-manager-5c6f759dd-gwbjb\" (UID: \"31a943d1-8ef0-4f15-9e3a-ba696db451b7\") " pod="openshift-lws-operator/lws-controller-manager-5c6f759dd-gwbjb"
Jul 07 07:05:02.446828 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:05:02.446649    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/31a943d1-8ef0-4f15-9e3a-ba696db451b7-manager-config\") pod \"lws-controller-manager-5c6f759dd-gwbjb\" (UID: \"31a943d1-8ef0-4f15-9e3a-ba696db451b7\") " pod="openshift-lws-operator/lws-controller-manager-5c6f759dd-gwbjb"
Jul 07 07:05:02.446828 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:05:02.446763    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/31a943d1-8ef0-4f15-9e3a-ba696db451b7-cert\") pod \"lws-controller-manager-5c6f759dd-gwbjb\" (UID: \"31a943d1-8ef0-4f15-9e3a-ba696db451b7\") " pod="openshift-lws-operator/lws-controller-manager-5c6f759dd-gwbjb"
Jul 07 07:05:02.447321 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:05:02.447297    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/31a943d1-8ef0-4f15-9e3a-ba696db451b7-manager-config\") pod \"lws-controller-manager-5c6f759dd-gwbjb\" (UID: \"31a943d1-8ef0-4f15-9e3a-ba696db451b7\") " pod="openshift-lws-operator/lws-controller-manager-5c6f759dd-gwbjb"
Jul 07 07:05:02.449123 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:05:02.449099    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/31a943d1-8ef0-4f15-9e3a-ba696db451b7-cert\") pod \"lws-controller-manager-5c6f759dd-gwbjb\" (UID: \"31a943d1-8ef0-4f15-9e3a-ba696db451b7\") " pod="openshift-lws-operator/lws-controller-manager-5c6f759dd-gwbjb"
Jul 07 07:05:02.449309 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:05:02.449288    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/31a943d1-8ef0-4f15-9e3a-ba696db451b7-metrics-cert\") pod \"lws-controller-manager-5c6f759dd-gwbjb\" (UID: \"31a943d1-8ef0-4f15-9e3a-ba696db451b7\") " pod="openshift-lws-operator/lws-controller-manager-5c6f759dd-gwbjb"
Jul 07 07:05:02.462125 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:05:02.462094    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzm9m\" (UniqueName: \"kubernetes.io/projected/31a943d1-8ef0-4f15-9e3a-ba696db451b7-kube-api-access-fzm9m\") pod \"lws-controller-manager-5c6f759dd-gwbjb\" (UID: \"31a943d1-8ef0-4f15-9e3a-ba696db451b7\") " pod="openshift-lws-operator/lws-controller-manager-5c6f759dd-gwbjb"
Jul 07 07:05:02.494653 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:05:02.494615    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-54bd8c98d7-jw7hc" event={"ID":"9e42bb72-5874-40c2-b79d-a24298bb8586","Type":"ContainerStarted","Data":"2c7eef98c7c57f25d4c2b3908165800840f6290bbd9af066dd77eecebe9a7533"}
Jul 07 07:05:02.524088 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:05:02.524051    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5c6f759dd-gwbjb"
Jul 07 07:05:02.664852 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:05:02.664824    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5c6f759dd-gwbjb"]
Jul 07 07:05:02.667200 ip-10-0-138-97 kubenswrapper[2579]: W0707 07:05:02.667168    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31a943d1_8ef0_4f15_9e3a_ba696db451b7.slice/crio-7ebfdc6e998fc67277154e2e6d8f8cab9624a29526c5fb302ba400d40fd54575 WatchSource:0}: Error finding container 7ebfdc6e998fc67277154e2e6d8f8cab9624a29526c5fb302ba400d40fd54575: Status 404 returned error can't find the container with id 7ebfdc6e998fc67277154e2e6d8f8cab9624a29526c5fb302ba400d40fd54575
Jul 07 07:05:03.501420 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:05:03.501376    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5c6f759dd-gwbjb" event={"ID":"31a943d1-8ef0-4f15-9e3a-ba696db451b7","Type":"ContainerStarted","Data":"7ebfdc6e998fc67277154e2e6d8f8cab9624a29526c5fb302ba400d40fd54575"}
Jul 07 07:05:06.515523 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:05:06.515481    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-54bd8c98d7-jw7hc" event={"ID":"9e42bb72-5874-40c2-b79d-a24298bb8586","Type":"ContainerStarted","Data":"a626a47cc9c38d792f3e3ee1604f7135e4e0d52fd54714784527e90fbf61a096"}
Jul 07 07:05:06.516944 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:05:06.516916    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5c6f759dd-gwbjb" event={"ID":"31a943d1-8ef0-4f15-9e3a-ba696db451b7","Type":"ContainerStarted","Data":"d5a0c611a4f66d0d33c036fed1568448de77632e7a998807c2f9c88de023d64e"}
Jul 07 07:05:06.517041 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:05:06.516995    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5c6f759dd-gwbjb"
Jul 07 07:05:06.536292 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:05:06.536240    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-54bd8c98d7-jw7hc" podStartSLOduration=2.081459867 podStartE2EDuration="5.536219126s" totalImagesPullingTime="3.454759259s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 07:05:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 07:05:06.53458784 +0000 UTC m=+386.078349321" watchObservedRunningTime="2026-07-07 07:05:06.536219126 +0000 UTC m=+386.079980606"
Jul 07 07:05:06.553752 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:05:06.553702    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5c6f759dd-gwbjb" podStartSLOduration=1.5463031 podStartE2EDuration="4.553684855s" totalImagesPullingTime="3.007381755s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 07:05:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 07:05:06.552458528 +0000 UTC m=+386.096220014" watchObservedRunningTime="2026-07-07 07:05:06.553684855 +0000 UTC m=+386.097446335"
Jul 07 07:05:17.523433 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:05:17.523395    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5c6f759dd-gwbjb"
Jul 07 07:06:59.373530 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:06:59.373500    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dmq79"]
Jul 07 07:06:59.376514 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:06:59.376497    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dmq79"
Jul 07 07:06:59.384042 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:06:59.379759    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-zvpzq\""
Jul 07 07:06:59.384042 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:06:59.381168    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 07 07:06:59.384042 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:06:59.381434    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 07 07:06:59.393886 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:06:59.393862    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dmq79"]
Jul 07 07:06:59.506302 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:06:59.506257    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hlmr\" (UniqueName: \"kubernetes.io/projected/d8b2175d-87fe-4164-bdec-c36e4923701b-kube-api-access-5hlmr\") pod \"limitador-operator-controller-manager-859d7b78bd-dmq79\" (UID: \"d8b2175d-87fe-4164-bdec-c36e4923701b\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dmq79"
Jul 07 07:06:59.607694 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:06:59.607658    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hlmr\" (UniqueName: \"kubernetes.io/projected/d8b2175d-87fe-4164-bdec-c36e4923701b-kube-api-access-5hlmr\") pod \"limitador-operator-controller-manager-859d7b78bd-dmq79\" (UID: \"d8b2175d-87fe-4164-bdec-c36e4923701b\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dmq79"
Jul 07 07:06:59.620464 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:06:59.620437    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hlmr\" (UniqueName: \"kubernetes.io/projected/d8b2175d-87fe-4164-bdec-c36e4923701b-kube-api-access-5hlmr\") pod \"limitador-operator-controller-manager-859d7b78bd-dmq79\" (UID: \"d8b2175d-87fe-4164-bdec-c36e4923701b\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dmq79"
Jul 07 07:06:59.691355 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:06:59.691264    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dmq79"
Jul 07 07:06:59.831538 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:06:59.831508    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dmq79"]
Jul 07 07:06:59.834059 ip-10-0-138-97 kubenswrapper[2579]: W0707 07:06:59.834036    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8b2175d_87fe_4164_bdec_c36e4923701b.slice/crio-f6424b68198fbb304e249264ef799957ceb3dd02e3a47b3c593e4291da7778ed WatchSource:0}: Error finding container f6424b68198fbb304e249264ef799957ceb3dd02e3a47b3c593e4291da7778ed: Status 404 returned error can't find the container with id f6424b68198fbb304e249264ef799957ceb3dd02e3a47b3c593e4291da7778ed
Jul 07 07:06:59.933636 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:06:59.933598    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dmq79" event={"ID":"d8b2175d-87fe-4164-bdec-c36e4923701b","Type":"ContainerStarted","Data":"f6424b68198fbb304e249264ef799957ceb3dd02e3a47b3c593e4291da7778ed"}
Jul 07 07:07:01.948003 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:01.947886    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dmq79" event={"ID":"d8b2175d-87fe-4164-bdec-c36e4923701b","Type":"ContainerStarted","Data":"e6e640b83793bf4cdd619e640377ec7d95cb6265d47e725281ab37267c908601"}
Jul 07 07:07:01.948003 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:01.947984    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dmq79"
Jul 07 07:07:01.970027 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:01.969979    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dmq79" podStartSLOduration=1.192613173 podStartE2EDuration="2.96993855s" totalImagesPullingTime="1.777325377s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 07:06:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 07:07:01.968274444 +0000 UTC m=+501.512035925" watchObservedRunningTime="2026-07-07 07:07:01.96993855 +0000 UTC m=+501.513700031"
Jul 07 07:07:09.375350 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:09.375310    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6dcfdb8b9-272nf"]
Jul 07 07:07:09.378510 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:09.378491    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6dcfdb8b9-272nf"
Jul 07 07:07:09.381647 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:09.381620    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 07 07:07:09.381755 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:09.381686    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 07 07:07:09.383014 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:09.382999    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 07 07:07:09.383077 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:09.383010    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 07 07:07:09.383917 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:09.383897    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 07 07:07:09.388983 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:09.388964    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 07 07:07:09.389065 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:09.388952    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-jh5lq\""
Jul 07 07:07:09.389108 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:09.388988    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 07 07:07:09.394424 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:09.394401    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6dcfdb8b9-272nf"]
Jul 07 07:07:09.402165 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:09.402143    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 07 07:07:09.488315 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:09.488280    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1894f384-040c-4d22-b4fd-a294b9cca8fb-console-oauth-config\") pod \"console-6dcfdb8b9-272nf\" (UID: \"1894f384-040c-4d22-b4fd-a294b9cca8fb\") " pod="openshift-console/console-6dcfdb8b9-272nf"
Jul 07 07:07:09.488507 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:09.488328    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1894f384-040c-4d22-b4fd-a294b9cca8fb-service-ca\") pod \"console-6dcfdb8b9-272nf\" (UID: \"1894f384-040c-4d22-b4fd-a294b9cca8fb\") " pod="openshift-console/console-6dcfdb8b9-272nf"
Jul 07 07:07:09.488507 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:09.488416    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1894f384-040c-4d22-b4fd-a294b9cca8fb-oauth-serving-cert\") pod \"console-6dcfdb8b9-272nf\" (UID: \"1894f384-040c-4d22-b4fd-a294b9cca8fb\") " pod="openshift-console/console-6dcfdb8b9-272nf"
Jul 07 07:07:09.488507 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:09.488453    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glnc6\" (UniqueName: \"kubernetes.io/projected/1894f384-040c-4d22-b4fd-a294b9cca8fb-kube-api-access-glnc6\") pod \"console-6dcfdb8b9-272nf\" (UID: \"1894f384-040c-4d22-b4fd-a294b9cca8fb\") " pod="openshift-console/console-6dcfdb8b9-272nf"
Jul 07 07:07:09.488644 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:09.488522    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1894f384-040c-4d22-b4fd-a294b9cca8fb-trusted-ca-bundle\") pod \"console-6dcfdb8b9-272nf\" (UID: \"1894f384-040c-4d22-b4fd-a294b9cca8fb\") " pod="openshift-console/console-6dcfdb8b9-272nf"
Jul 07 07:07:09.488644 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:09.488550    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1894f384-040c-4d22-b4fd-a294b9cca8fb-console-serving-cert\") pod \"console-6dcfdb8b9-272nf\" (UID: \"1894f384-040c-4d22-b4fd-a294b9cca8fb\") " pod="openshift-console/console-6dcfdb8b9-272nf"
Jul 07 07:07:09.488644 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:09.488584    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1894f384-040c-4d22-b4fd-a294b9cca8fb-console-config\") pod \"console-6dcfdb8b9-272nf\" (UID: \"1894f384-040c-4d22-b4fd-a294b9cca8fb\") " pod="openshift-console/console-6dcfdb8b9-272nf"
Jul 07 07:07:09.589428 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:09.589389    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1894f384-040c-4d22-b4fd-a294b9cca8fb-trusted-ca-bundle\") pod \"console-6dcfdb8b9-272nf\" (UID: \"1894f384-040c-4d22-b4fd-a294b9cca8fb\") " pod="openshift-console/console-6dcfdb8b9-272nf"
Jul 07 07:07:09.589428 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:09.589429    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1894f384-040c-4d22-b4fd-a294b9cca8fb-console-serving-cert\") pod \"console-6dcfdb8b9-272nf\" (UID: \"1894f384-040c-4d22-b4fd-a294b9cca8fb\") " pod="openshift-console/console-6dcfdb8b9-272nf"
Jul 07 07:07:09.589684 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:09.589457    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1894f384-040c-4d22-b4fd-a294b9cca8fb-console-config\") pod \"console-6dcfdb8b9-272nf\" (UID: \"1894f384-040c-4d22-b4fd-a294b9cca8fb\") " pod="openshift-console/console-6dcfdb8b9-272nf"
Jul 07 07:07:09.589684 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:09.589495    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1894f384-040c-4d22-b4fd-a294b9cca8fb-console-oauth-config\") pod \"console-6dcfdb8b9-272nf\" (UID: \"1894f384-040c-4d22-b4fd-a294b9cca8fb\") " pod="openshift-console/console-6dcfdb8b9-272nf"
Jul 07 07:07:09.589684 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:09.589536    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1894f384-040c-4d22-b4fd-a294b9cca8fb-service-ca\") pod \"console-6dcfdb8b9-272nf\" (UID: \"1894f384-040c-4d22-b4fd-a294b9cca8fb\") " pod="openshift-console/console-6dcfdb8b9-272nf"
Jul 07 07:07:09.589815 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:09.589681    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1894f384-040c-4d22-b4fd-a294b9cca8fb-oauth-serving-cert\") pod \"console-6dcfdb8b9-272nf\" (UID: \"1894f384-040c-4d22-b4fd-a294b9cca8fb\") " pod="openshift-console/console-6dcfdb8b9-272nf"
Jul 07 07:07:09.589815 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:09.589725    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glnc6\" (UniqueName: \"kubernetes.io/projected/1894f384-040c-4d22-b4fd-a294b9cca8fb-kube-api-access-glnc6\") pod \"console-6dcfdb8b9-272nf\" (UID: \"1894f384-040c-4d22-b4fd-a294b9cca8fb\") " pod="openshift-console/console-6dcfdb8b9-272nf"
Jul 07 07:07:09.590367 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:09.590345    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1894f384-040c-4d22-b4fd-a294b9cca8fb-service-ca\") pod \"console-6dcfdb8b9-272nf\" (UID: \"1894f384-040c-4d22-b4fd-a294b9cca8fb\") " pod="openshift-console/console-6dcfdb8b9-272nf"
Jul 07 07:07:09.590458 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:09.590389    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1894f384-040c-4d22-b4fd-a294b9cca8fb-console-config\") pod \"console-6dcfdb8b9-272nf\" (UID: \"1894f384-040c-4d22-b4fd-a294b9cca8fb\") " pod="openshift-console/console-6dcfdb8b9-272nf"
Jul 07 07:07:09.590458 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:09.590400    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1894f384-040c-4d22-b4fd-a294b9cca8fb-trusted-ca-bundle\") pod \"console-6dcfdb8b9-272nf\" (UID: \"1894f384-040c-4d22-b4fd-a294b9cca8fb\") " pod="openshift-console/console-6dcfdb8b9-272nf"
Jul 07 07:07:09.590458 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:09.590415    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1894f384-040c-4d22-b4fd-a294b9cca8fb-oauth-serving-cert\") pod \"console-6dcfdb8b9-272nf\" (UID: \"1894f384-040c-4d22-b4fd-a294b9cca8fb\") " pod="openshift-console/console-6dcfdb8b9-272nf"
Jul 07 07:07:09.592445 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:09.592421    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1894f384-040c-4d22-b4fd-a294b9cca8fb-console-oauth-config\") pod \"console-6dcfdb8b9-272nf\" (UID: \"1894f384-040c-4d22-b4fd-a294b9cca8fb\") " pod="openshift-console/console-6dcfdb8b9-272nf"
Jul 07 07:07:09.592676 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:09.592655    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1894f384-040c-4d22-b4fd-a294b9cca8fb-console-serving-cert\") pod \"console-6dcfdb8b9-272nf\" (UID: \"1894f384-040c-4d22-b4fd-a294b9cca8fb\") " pod="openshift-console/console-6dcfdb8b9-272nf"
Jul 07 07:07:09.600102 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:09.600082    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glnc6\" (UniqueName: \"kubernetes.io/projected/1894f384-040c-4d22-b4fd-a294b9cca8fb-kube-api-access-glnc6\") pod \"console-6dcfdb8b9-272nf\" (UID: \"1894f384-040c-4d22-b4fd-a294b9cca8fb\") " pod="openshift-console/console-6dcfdb8b9-272nf"
Jul 07 07:07:09.687676 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:09.687584    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6dcfdb8b9-272nf"
Jul 07 07:07:09.826017 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:09.825985    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6dcfdb8b9-272nf"]
Jul 07 07:07:09.828989 ip-10-0-138-97 kubenswrapper[2579]: W0707 07:07:09.828945    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1894f384_040c_4d22_b4fd_a294b9cca8fb.slice/crio-a57ca45cbd11ebcbe47c13630d6877fdd02b7f106e681eab4e200f52bb27a24e WatchSource:0}: Error finding container a57ca45cbd11ebcbe47c13630d6877fdd02b7f106e681eab4e200f52bb27a24e: Status 404 returned error can't find the container with id a57ca45cbd11ebcbe47c13630d6877fdd02b7f106e681eab4e200f52bb27a24e
Jul 07 07:07:09.983395 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:09.983361    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6dcfdb8b9-272nf" event={"ID":"1894f384-040c-4d22-b4fd-a294b9cca8fb","Type":"ContainerStarted","Data":"2ab7b81971a5db82b64ee3886cd53f87b5719e5576e6b1726a02962ec907d6e8"}
Jul 07 07:07:09.983551 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:09.983411    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6dcfdb8b9-272nf" event={"ID":"1894f384-040c-4d22-b4fd-a294b9cca8fb","Type":"ContainerStarted","Data":"a57ca45cbd11ebcbe47c13630d6877fdd02b7f106e681eab4e200f52bb27a24e"}
Jul 07 07:07:10.006448 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:10.006397    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6dcfdb8b9-272nf" podStartSLOduration=1.006380551 podStartE2EDuration="1.006380551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 07:07:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 07:07:10.004772552 +0000 UTC m=+509.548534060" watchObservedRunningTime="2026-07-07 07:07:10.006380551 +0000 UTC m=+509.550142032"
Jul 07 07:07:12.955456 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:12.955422    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dmq79"
Jul 07 07:07:19.688735 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:19.688686    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6dcfdb8b9-272nf"
Jul 07 07:07:19.689228 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:19.688771    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6dcfdb8b9-272nf"
Jul 07 07:07:19.693605 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:19.693582    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6dcfdb8b9-272nf"
Jul 07 07:07:20.028286 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:20.028257    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6dcfdb8b9-272nf"
Jul 07 07:07:44.953817 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:44.953739    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-dtbdh"]
Jul 07 07:07:44.956150 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:44.956132    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-dtbdh"
Jul 07 07:07:44.958914 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:44.958889    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-9f64d\""
Jul 07 07:07:44.965578 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:44.965557    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-dtbdh"]
Jul 07 07:07:45.004380 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:45.004341    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg5hr\" (UniqueName: \"kubernetes.io/projected/8a836fc3-4043-46bf-86c8-1f1ba1b3cb07-kube-api-access-jg5hr\") pod \"authorino-58cd955675-dtbdh\" (UID: \"8a836fc3-4043-46bf-86c8-1f1ba1b3cb07\") " pod="kuadrant-system/authorino-58cd955675-dtbdh"
Jul 07 07:07:45.086549 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:45.086506    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-vqfmk"]
Jul 07 07:07:45.088611 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:45.088595    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-vqfmk"
Jul 07 07:07:45.098276 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:45.098250    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-vqfmk"]
Jul 07 07:07:45.105887 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:45.105859    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg5hr\" (UniqueName: \"kubernetes.io/projected/8a836fc3-4043-46bf-86c8-1f1ba1b3cb07-kube-api-access-jg5hr\") pod \"authorino-58cd955675-dtbdh\" (UID: \"8a836fc3-4043-46bf-86c8-1f1ba1b3cb07\") " pod="kuadrant-system/authorino-58cd955675-dtbdh"
Jul 07 07:07:45.114368 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:45.114346    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg5hr\" (UniqueName: \"kubernetes.io/projected/8a836fc3-4043-46bf-86c8-1f1ba1b3cb07-kube-api-access-jg5hr\") pod \"authorino-58cd955675-dtbdh\" (UID: \"8a836fc3-4043-46bf-86c8-1f1ba1b3cb07\") " pod="kuadrant-system/authorino-58cd955675-dtbdh"
Jul 07 07:07:45.206611 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:45.206513    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gpwd\" (UniqueName: \"kubernetes.io/projected/118744f0-f0d9-4353-98d5-2565bd283fd3-kube-api-access-7gpwd\") pod \"authorino-75d556bd77-vqfmk\" (UID: \"118744f0-f0d9-4353-98d5-2565bd283fd3\") " pod="kuadrant-system/authorino-75d556bd77-vqfmk"
Jul 07 07:07:45.267126 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:45.267097    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-dtbdh"
Jul 07 07:07:45.307644 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:45.307598    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gpwd\" (UniqueName: \"kubernetes.io/projected/118744f0-f0d9-4353-98d5-2565bd283fd3-kube-api-access-7gpwd\") pod \"authorino-75d556bd77-vqfmk\" (UID: \"118744f0-f0d9-4353-98d5-2565bd283fd3\") " pod="kuadrant-system/authorino-75d556bd77-vqfmk"
Jul 07 07:07:45.317745 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:45.317370    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gpwd\" (UniqueName: \"kubernetes.io/projected/118744f0-f0d9-4353-98d5-2565bd283fd3-kube-api-access-7gpwd\") pod \"authorino-75d556bd77-vqfmk\" (UID: \"118744f0-f0d9-4353-98d5-2565bd283fd3\") " pod="kuadrant-system/authorino-75d556bd77-vqfmk"
Jul 07 07:07:45.396842 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:45.396809    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-vqfmk"
Jul 07 07:07:45.410920 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:45.410751    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-dtbdh"]
Jul 07 07:07:45.413001 ip-10-0-138-97 kubenswrapper[2579]: W0707 07:07:45.412968    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8a836fc3_4043_46bf_86c8_1f1ba1b3cb07.slice/crio-e355572a784911685db3aa4221322a6f82919a109905a205f3aa895ee841d75d WatchSource:0}: Error finding container e355572a784911685db3aa4221322a6f82919a109905a205f3aa895ee841d75d: Status 404 returned error can't find the container with id e355572a784911685db3aa4221322a6f82919a109905a205f3aa895ee841d75d
Jul 07 07:07:45.524553 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:45.524365    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-vqfmk"]
Jul 07 07:07:45.527094 ip-10-0-138-97 kubenswrapper[2579]: W0707 07:07:45.527067    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod118744f0_f0d9_4353_98d5_2565bd283fd3.slice/crio-d41475bdb5e88f0cd673453f9f7f058c2b494a96a9d4ff679b948571eb4b36b6 WatchSource:0}: Error finding container d41475bdb5e88f0cd673453f9f7f058c2b494a96a9d4ff679b948571eb4b36b6: Status 404 returned error can't find the container with id d41475bdb5e88f0cd673453f9f7f058c2b494a96a9d4ff679b948571eb4b36b6
Jul 07 07:07:46.133478 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:46.133441    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-vqfmk" event={"ID":"118744f0-f0d9-4353-98d5-2565bd283fd3","Type":"ContainerStarted","Data":"d41475bdb5e88f0cd673453f9f7f058c2b494a96a9d4ff679b948571eb4b36b6"}
Jul 07 07:07:46.134947 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:46.134919    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-dtbdh" event={"ID":"8a836fc3-4043-46bf-86c8-1f1ba1b3cb07","Type":"ContainerStarted","Data":"e355572a784911685db3aa4221322a6f82919a109905a205f3aa895ee841d75d"}
Jul 07 07:07:49.158420 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:49.158324    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-dtbdh" event={"ID":"8a836fc3-4043-46bf-86c8-1f1ba1b3cb07","Type":"ContainerStarted","Data":"db2b17bfc8c37dc26674be6dc35f5b664298bf8cc20edb5f29676c47cf6110cb"}
Jul 07 07:07:49.161739 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:49.161713    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-vqfmk" event={"ID":"118744f0-f0d9-4353-98d5-2565bd283fd3","Type":"ContainerStarted","Data":"91e67807817f5d364c80e059ccd05fc0e0fb6f064e34ce6ba424ccb16929f4cc"}
Jul 07 07:07:49.180590 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:49.180522    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-dtbdh" podStartSLOduration=1.538048262 podStartE2EDuration="5.180501463s" totalImagesPullingTime="3.642453201s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 07:07:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 07:07:49.179874592 +0000 UTC m=+548.723636074" watchObservedRunningTime="2026-07-07 07:07:49.180501463 +0000 UTC m=+548.724262937"
Jul 07 07:07:49.197273 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:49.197208    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-vqfmk" podStartSLOduration=0.65762214 podStartE2EDuration="4.197185896s" totalImagesPullingTime="3.539563756s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 07:07:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 07:07:49.19608123 +0000 UTC m=+548.739842724" watchObservedRunningTime="2026-07-07 07:07:49.197185896 +0000 UTC m=+548.740947378"
Jul 07 07:07:49.226590 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:49.226549    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-dtbdh"]
Jul 07 07:07:51.171221 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:51.171181    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-dtbdh" podUID="8a836fc3-4043-46bf-86c8-1f1ba1b3cb07" containerName="authorino" containerID="cri-o://db2b17bfc8c37dc26674be6dc35f5b664298bf8cc20edb5f29676c47cf6110cb" gracePeriod=30
Jul 07 07:07:51.419217 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:51.419193    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-dtbdh"
Jul 07 07:07:51.462060 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:51.461981    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jg5hr\" (UniqueName: \"kubernetes.io/projected/8a836fc3-4043-46bf-86c8-1f1ba1b3cb07-kube-api-access-jg5hr\") pod \"8a836fc3-4043-46bf-86c8-1f1ba1b3cb07\" (UID: \"8a836fc3-4043-46bf-86c8-1f1ba1b3cb07\") "
Jul 07 07:07:51.464138 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:51.464111    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a836fc3-4043-46bf-86c8-1f1ba1b3cb07-kube-api-access-jg5hr" (OuterVolumeSpecName: "kube-api-access-jg5hr") pod "8a836fc3-4043-46bf-86c8-1f1ba1b3cb07" (UID: "8a836fc3-4043-46bf-86c8-1f1ba1b3cb07"). InnerVolumeSpecName "kube-api-access-jg5hr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 07:07:51.562736 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:51.562692    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jg5hr\" (UniqueName: \"kubernetes.io/projected/8a836fc3-4043-46bf-86c8-1f1ba1b3cb07-kube-api-access-jg5hr\") on node \"ip-10-0-138-97.ec2.internal\" DevicePath \"\""
Jul 07 07:07:52.176801 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:52.176760    2579 generic.go:356] "Generic (PLEG): container finished" podID="8a836fc3-4043-46bf-86c8-1f1ba1b3cb07" containerID="db2b17bfc8c37dc26674be6dc35f5b664298bf8cc20edb5f29676c47cf6110cb" exitCode=0
Jul 07 07:07:52.177225 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:52.176807    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-dtbdh"
Jul 07 07:07:52.177225 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:52.176809    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-dtbdh" event={"ID":"8a836fc3-4043-46bf-86c8-1f1ba1b3cb07","Type":"ContainerDied","Data":"db2b17bfc8c37dc26674be6dc35f5b664298bf8cc20edb5f29676c47cf6110cb"}
Jul 07 07:07:52.177225 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:52.176935    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-dtbdh" event={"ID":"8a836fc3-4043-46bf-86c8-1f1ba1b3cb07","Type":"ContainerDied","Data":"e355572a784911685db3aa4221322a6f82919a109905a205f3aa895ee841d75d"}
Jul 07 07:07:52.177225 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:52.176990    2579 scope.go:117] "RemoveContainer" containerID="db2b17bfc8c37dc26674be6dc35f5b664298bf8cc20edb5f29676c47cf6110cb"
Jul 07 07:07:52.185350 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:52.185326    2579 scope.go:117] "RemoveContainer" containerID="db2b17bfc8c37dc26674be6dc35f5b664298bf8cc20edb5f29676c47cf6110cb"
Jul 07 07:07:52.185622 ip-10-0-138-97 kubenswrapper[2579]: E0707 07:07:52.185602    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db2b17bfc8c37dc26674be6dc35f5b664298bf8cc20edb5f29676c47cf6110cb\": container with ID starting with db2b17bfc8c37dc26674be6dc35f5b664298bf8cc20edb5f29676c47cf6110cb not found: ID does not exist" containerID="db2b17bfc8c37dc26674be6dc35f5b664298bf8cc20edb5f29676c47cf6110cb"
Jul 07 07:07:52.185677 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:52.185633    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db2b17bfc8c37dc26674be6dc35f5b664298bf8cc20edb5f29676c47cf6110cb"} err="failed to get container status \"db2b17bfc8c37dc26674be6dc35f5b664298bf8cc20edb5f29676c47cf6110cb\": rpc error: code = NotFound desc = could not find container \"db2b17bfc8c37dc26674be6dc35f5b664298bf8cc20edb5f29676c47cf6110cb\": container with ID starting with db2b17bfc8c37dc26674be6dc35f5b664298bf8cc20edb5f29676c47cf6110cb not found: ID does not exist"
Jul 07 07:07:52.199829 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:52.199787    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-dtbdh"]
Jul 07 07:07:52.201559 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:52.201535    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-dtbdh"]
Jul 07 07:07:52.963396 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:07:52.963364    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a836fc3-4043-46bf-86c8-1f1ba1b3cb07" path="/var/lib/kubelet/pods/8a836fc3-4043-46bf-86c8-1f1ba1b3cb07/volumes"
Jul 07 07:08:13.276551 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:08:13.276516    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-69f5687f8b-fxns8"]
Jul 07 07:08:13.276968 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:08:13.276846    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a836fc3-4043-46bf-86c8-1f1ba1b3cb07" containerName="authorino"
Jul 07 07:08:13.276968 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:08:13.276860    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a836fc3-4043-46bf-86c8-1f1ba1b3cb07" containerName="authorino"
Jul 07 07:08:13.276968 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:08:13.276921    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a836fc3-4043-46bf-86c8-1f1ba1b3cb07" containerName="authorino"
Jul 07 07:08:13.279751 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:08:13.279733    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-69f5687f8b-fxns8"
Jul 07 07:08:13.289475 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:08:13.289451    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-69f5687f8b-fxns8"]
Jul 07 07:08:13.449595 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:08:13.449550    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skvbn\" (UniqueName: \"kubernetes.io/projected/cc9d2e66-5e0d-418a-99e5-d43f7bf93ba9-kube-api-access-skvbn\") pod \"authorino-69f5687f8b-fxns8\" (UID: \"cc9d2e66-5e0d-418a-99e5-d43f7bf93ba9\") " pod="kuadrant-system/authorino-69f5687f8b-fxns8"
Jul 07 07:08:13.454927 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:08:13.454897    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-69f5687f8b-fxns8"]
Jul 07 07:08:13.455170 ip-10-0-138-97 kubenswrapper[2579]: E0707 07:08:13.455148    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-skvbn], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-69f5687f8b-fxns8" podUID="cc9d2e66-5e0d-418a-99e5-d43f7bf93ba9"
Jul 07 07:08:13.485445 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:08:13.485409    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6bc8cd784b-cc8kh"]
Jul 07 07:08:13.488550 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:08:13.488532    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6bc8cd784b-cc8kh"
Jul 07 07:08:13.491055 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:08:13.491032    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 07 07:08:13.497464 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:08:13.497444    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6bc8cd784b-cc8kh"]
Jul 07 07:08:13.550388 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:08:13.550296    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skvbn\" (UniqueName: \"kubernetes.io/projected/cc9d2e66-5e0d-418a-99e5-d43f7bf93ba9-kube-api-access-skvbn\") pod \"authorino-69f5687f8b-fxns8\" (UID: \"cc9d2e66-5e0d-418a-99e5-d43f7bf93ba9\") " pod="kuadrant-system/authorino-69f5687f8b-fxns8"
Jul 07 07:08:13.559829 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:08:13.559800    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skvbn\" (UniqueName: \"kubernetes.io/projected/cc9d2e66-5e0d-418a-99e5-d43f7bf93ba9-kube-api-access-skvbn\") pod \"authorino-69f5687f8b-fxns8\" (UID: \"cc9d2e66-5e0d-418a-99e5-d43f7bf93ba9\") " pod="kuadrant-system/authorino-69f5687f8b-fxns8"
Jul 07 07:08:13.650762 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:08:13.650718    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7513a611-6905-46ee-861d-1577820b5973-tls-cert\") pod \"authorino-6bc8cd784b-cc8kh\" (UID: \"7513a611-6905-46ee-861d-1577820b5973\") " pod="kuadrant-system/authorino-6bc8cd784b-cc8kh"
Jul 07 07:08:13.650936 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:08:13.650768    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2dq5\" (UniqueName: \"kubernetes.io/projected/7513a611-6905-46ee-861d-1577820b5973-kube-api-access-x2dq5\") pod \"authorino-6bc8cd784b-cc8kh\" (UID: \"7513a611-6905-46ee-861d-1577820b5973\") " pod="kuadrant-system/authorino-6bc8cd784b-cc8kh"
Jul 07 07:08:13.751662 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:08:13.751622    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2dq5\" (UniqueName: \"kubernetes.io/projected/7513a611-6905-46ee-861d-1577820b5973-kube-api-access-x2dq5\") pod \"authorino-6bc8cd784b-cc8kh\" (UID: \"7513a611-6905-46ee-861d-1577820b5973\") " pod="kuadrant-system/authorino-6bc8cd784b-cc8kh"
Jul 07 07:08:13.751827 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:08:13.751716    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7513a611-6905-46ee-861d-1577820b5973-tls-cert\") pod \"authorino-6bc8cd784b-cc8kh\" (UID: \"7513a611-6905-46ee-861d-1577820b5973\") " pod="kuadrant-system/authorino-6bc8cd784b-cc8kh"
Jul 07 07:08:13.754109 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:08:13.754080    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7513a611-6905-46ee-861d-1577820b5973-tls-cert\") pod \"authorino-6bc8cd784b-cc8kh\" (UID: \"7513a611-6905-46ee-861d-1577820b5973\") " pod="kuadrant-system/authorino-6bc8cd784b-cc8kh"
Jul 07 07:08:13.761702 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:08:13.761678    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2dq5\" (UniqueName: \"kubernetes.io/projected/7513a611-6905-46ee-861d-1577820b5973-kube-api-access-x2dq5\") pod \"authorino-6bc8cd784b-cc8kh\" (UID: \"7513a611-6905-46ee-861d-1577820b5973\") " pod="kuadrant-system/authorino-6bc8cd784b-cc8kh"
Jul 07 07:08:13.798291 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:08:13.798260    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6bc8cd784b-cc8kh"
Jul 07 07:08:13.929710 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:08:13.929683    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6bc8cd784b-cc8kh"]
Jul 07 07:08:13.932287 ip-10-0-138-97 kubenswrapper[2579]: W0707 07:08:13.932258    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7513a611_6905_46ee_861d_1577820b5973.slice/crio-67ade16afad8863f518312d23414fd99ad4d942627cfaf53f89a3db27527e491 WatchSource:0}: Error finding container 67ade16afad8863f518312d23414fd99ad4d942627cfaf53f89a3db27527e491: Status 404 returned error can't find the container with id 67ade16afad8863f518312d23414fd99ad4d942627cfaf53f89a3db27527e491
Jul 07 07:08:14.284345 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:08:14.284318    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-69f5687f8b-fxns8"
Jul 07 07:08:14.284747 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:08:14.284317    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6bc8cd784b-cc8kh" event={"ID":"7513a611-6905-46ee-861d-1577820b5973","Type":"ContainerStarted","Data":"67ade16afad8863f518312d23414fd99ad4d942627cfaf53f89a3db27527e491"}
Jul 07 07:08:14.292164 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:08:14.292142    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-69f5687f8b-fxns8"
Jul 07 07:08:14.458468 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:08:14.458437    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-skvbn\" (UniqueName: \"kubernetes.io/projected/cc9d2e66-5e0d-418a-99e5-d43f7bf93ba9-kube-api-access-skvbn\") pod \"cc9d2e66-5e0d-418a-99e5-d43f7bf93ba9\" (UID: \"cc9d2e66-5e0d-418a-99e5-d43f7bf93ba9\") "
Jul 07 07:08:14.460481 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:08:14.460459    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc9d2e66-5e0d-418a-99e5-d43f7bf93ba9-kube-api-access-skvbn" (OuterVolumeSpecName: "kube-api-access-skvbn") pod "cc9d2e66-5e0d-418a-99e5-d43f7bf93ba9" (UID: "cc9d2e66-5e0d-418a-99e5-d43f7bf93ba9"). InnerVolumeSpecName "kube-api-access-skvbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 07:08:14.559360 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:08:14.559271    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skvbn\" (UniqueName: \"kubernetes.io/projected/cc9d2e66-5e0d-418a-99e5-d43f7bf93ba9-kube-api-access-skvbn\") on node \"ip-10-0-138-97.ec2.internal\" DevicePath \"\""
Jul 07 07:08:15.290028 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:08:15.289981    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6bc8cd784b-cc8kh" event={"ID":"7513a611-6905-46ee-861d-1577820b5973","Type":"ContainerStarted","Data":"34e0dd639da9c1949ed9548d80ba90bcf45fd6836779053f74e3d0e2c3554f9f"}
Jul 07 07:08:15.290420 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:08:15.290051    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-69f5687f8b-fxns8"
Jul 07 07:08:15.311918 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:08:15.311859    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6bc8cd784b-cc8kh" podStartSLOduration=1.881553724 podStartE2EDuration="2.311837986s" totalImagesPullingTime="430.284262ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 07:08:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 07:08:15.310629281 +0000 UTC m=+574.854390764" watchObservedRunningTime="2026-07-07 07:08:15.311837986 +0000 UTC m=+574.855599468"
Jul 07 07:08:15.337793 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:08:15.337761    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-69f5687f8b-fxns8"]
Jul 07 07:08:15.344540 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:08:15.344511    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-vqfmk"]
Jul 07 07:08:15.344706 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:08:15.344686    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-vqfmk" podUID="118744f0-f0d9-4353-98d5-2565bd283fd3" containerName="authorino" containerID="cri-o://91e67807817f5d364c80e059ccd05fc0e0fb6f064e34ce6ba424ccb16929f4cc" gracePeriod=30
Jul 07 07:08:15.346171 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:08:15.346146    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-69f5687f8b-fxns8"]
Jul 07 07:08:15.589661 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:08:15.589636    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-vqfmk"
Jul 07 07:08:15.769907 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:08:15.769867    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7gpwd\" (UniqueName: \"kubernetes.io/projected/118744f0-f0d9-4353-98d5-2565bd283fd3-kube-api-access-7gpwd\") pod \"118744f0-f0d9-4353-98d5-2565bd283fd3\" (UID: \"118744f0-f0d9-4353-98d5-2565bd283fd3\") "
Jul 07 07:08:15.772054 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:08:15.772027    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/118744f0-f0d9-4353-98d5-2565bd283fd3-kube-api-access-7gpwd" (OuterVolumeSpecName: "kube-api-access-7gpwd") pod "118744f0-f0d9-4353-98d5-2565bd283fd3" (UID: "118744f0-f0d9-4353-98d5-2565bd283fd3"). InnerVolumeSpecName "kube-api-access-7gpwd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 07:08:15.871077 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:08:15.870979    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gpwd\" (UniqueName: \"kubernetes.io/projected/118744f0-f0d9-4353-98d5-2565bd283fd3-kube-api-access-7gpwd\") on node \"ip-10-0-138-97.ec2.internal\" DevicePath \"\""
Jul 07 07:08:16.295502 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:08:16.295462    2579 generic.go:356] "Generic (PLEG): container finished" podID="118744f0-f0d9-4353-98d5-2565bd283fd3" containerID="91e67807817f5d364c80e059ccd05fc0e0fb6f064e34ce6ba424ccb16929f4cc" exitCode=0
Jul 07 07:08:16.295502 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:08:16.295505    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-vqfmk"
Jul 07 07:08:16.296002 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:08:16.295546    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-vqfmk" event={"ID":"118744f0-f0d9-4353-98d5-2565bd283fd3","Type":"ContainerDied","Data":"91e67807817f5d364c80e059ccd05fc0e0fb6f064e34ce6ba424ccb16929f4cc"}
Jul 07 07:08:16.296002 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:08:16.295590    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-vqfmk" event={"ID":"118744f0-f0d9-4353-98d5-2565bd283fd3","Type":"ContainerDied","Data":"d41475bdb5e88f0cd673453f9f7f058c2b494a96a9d4ff679b948571eb4b36b6"}
Jul 07 07:08:16.296002 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:08:16.295606    2579 scope.go:117] "RemoveContainer" containerID="91e67807817f5d364c80e059ccd05fc0e0fb6f064e34ce6ba424ccb16929f4cc"
Jul 07 07:08:16.304266 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:08:16.304249    2579 scope.go:117] "RemoveContainer" containerID="91e67807817f5d364c80e059ccd05fc0e0fb6f064e34ce6ba424ccb16929f4cc"
Jul 07 07:08:16.304517 ip-10-0-138-97 kubenswrapper[2579]: E0707 07:08:16.304498    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91e67807817f5d364c80e059ccd05fc0e0fb6f064e34ce6ba424ccb16929f4cc\": container with ID starting with 91e67807817f5d364c80e059ccd05fc0e0fb6f064e34ce6ba424ccb16929f4cc not found: ID does not exist" containerID="91e67807817f5d364c80e059ccd05fc0e0fb6f064e34ce6ba424ccb16929f4cc"
Jul 07 07:08:16.304563 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:08:16.304526    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91e67807817f5d364c80e059ccd05fc0e0fb6f064e34ce6ba424ccb16929f4cc"} err="failed to get container status \"91e67807817f5d364c80e059ccd05fc0e0fb6f064e34ce6ba424ccb16929f4cc\": rpc error: code = NotFound desc = could not find container \"91e67807817f5d364c80e059ccd05fc0e0fb6f064e34ce6ba424ccb16929f4cc\": container with ID starting with 91e67807817f5d364c80e059ccd05fc0e0fb6f064e34ce6ba424ccb16929f4cc not found: ID does not exist"
Jul 07 07:08:16.316573 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:08:16.316549    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-vqfmk"]
Jul 07 07:08:16.322931 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:08:16.322904    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-vqfmk"]
Jul 07 07:08:16.963431 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:08:16.963402    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="118744f0-f0d9-4353-98d5-2565bd283fd3" path="/var/lib/kubelet/pods/118744f0-f0d9-4353-98d5-2565bd283fd3/volumes"
Jul 07 07:08:16.963726 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:08:16.963713    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc9d2e66-5e0d-418a-99e5-d43f7bf93ba9" path="/var/lib/kubelet/pods/cc9d2e66-5e0d-418a-99e5-d43f7bf93ba9/volumes"
Jul 07 07:09:18.299091 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:09:18.299062    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-ct89n"]
Jul 07 07:09:18.300625 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:09:18.299375    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="118744f0-f0d9-4353-98d5-2565bd283fd3" containerName="authorino"
Jul 07 07:09:18.300625 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:09:18.299385    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="118744f0-f0d9-4353-98d5-2565bd283fd3" containerName="authorino"
Jul 07 07:09:18.300625 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:09:18.299441    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="118744f0-f0d9-4353-98d5-2565bd283fd3" containerName="authorino"
Jul 07 07:09:18.301138 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:09:18.301122    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ct89n"
Jul 07 07:09:18.303748 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:09:18.303730    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 07 07:09:18.304793 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:09:18.304773    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Jul 07 07:09:18.304923 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:09:18.304799    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-sml68\""
Jul 07 07:09:18.304923 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:09:18.304812    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 07 07:09:18.312671 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:09:18.312650    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-ct89n"]
Jul 07 07:09:18.399035 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:09:18.398998    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7368ec91-3412-4c08-b895-955fafebd42f-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ct89n\" (UID: \"7368ec91-3412-4c08-b895-955fafebd42f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ct89n"
Jul 07 07:09:18.399220 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:09:18.399061    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7368ec91-3412-4c08-b895-955fafebd42f-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ct89n\" (UID: \"7368ec91-3412-4c08-b895-955fafebd42f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ct89n"
Jul 07 07:09:18.399220 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:09:18.399112    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7368ec91-3412-4c08-b895-955fafebd42f-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ct89n\" (UID: \"7368ec91-3412-4c08-b895-955fafebd42f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ct89n"
Jul 07 07:09:18.399220 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:09:18.399127    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7368ec91-3412-4c08-b895-955fafebd42f-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ct89n\" (UID: \"7368ec91-3412-4c08-b895-955fafebd42f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ct89n"
Jul 07 07:09:18.399220 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:09:18.399201    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7368ec91-3412-4c08-b895-955fafebd42f-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ct89n\" (UID: \"7368ec91-3412-4c08-b895-955fafebd42f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ct89n"
Jul 07 07:09:18.399374 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:09:18.399254    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwvfz\" (UniqueName: \"kubernetes.io/projected/7368ec91-3412-4c08-b895-955fafebd42f-kube-api-access-hwvfz\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ct89n\" (UID: \"7368ec91-3412-4c08-b895-955fafebd42f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ct89n"
Jul 07 07:09:18.499817 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:09:18.499783    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwvfz\" (UniqueName: \"kubernetes.io/projected/7368ec91-3412-4c08-b895-955fafebd42f-kube-api-access-hwvfz\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ct89n\" (UID: \"7368ec91-3412-4c08-b895-955fafebd42f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ct89n"
Jul 07 07:09:18.499817 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:09:18.499824    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7368ec91-3412-4c08-b895-955fafebd42f-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ct89n\" (UID: \"7368ec91-3412-4c08-b895-955fafebd42f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ct89n"
Jul 07 07:09:18.500053 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:09:18.499868    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7368ec91-3412-4c08-b895-955fafebd42f-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ct89n\" (UID: \"7368ec91-3412-4c08-b895-955fafebd42f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ct89n"
Jul 07 07:09:18.500053 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:09:18.499892    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7368ec91-3412-4c08-b895-955fafebd42f-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ct89n\" (UID: \"7368ec91-3412-4c08-b895-955fafebd42f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ct89n"
Jul 07 07:09:18.500053 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:09:18.499906    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7368ec91-3412-4c08-b895-955fafebd42f-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ct89n\" (UID: \"7368ec91-3412-4c08-b895-955fafebd42f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ct89n"
Jul 07 07:09:18.500053 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:09:18.499977    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7368ec91-3412-4c08-b895-955fafebd42f-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ct89n\" (UID: \"7368ec91-3412-4c08-b895-955fafebd42f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ct89n"
Jul 07 07:09:18.500332 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:09:18.500308    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7368ec91-3412-4c08-b895-955fafebd42f-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ct89n\" (UID: \"7368ec91-3412-4c08-b895-955fafebd42f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ct89n"
Jul 07 07:09:18.500376 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:09:18.500322    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7368ec91-3412-4c08-b895-955fafebd42f-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ct89n\" (UID: \"7368ec91-3412-4c08-b895-955fafebd42f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ct89n"
Jul 07 07:09:18.500417 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:09:18.500374    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7368ec91-3412-4c08-b895-955fafebd42f-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ct89n\" (UID: \"7368ec91-3412-4c08-b895-955fafebd42f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ct89n"
Jul 07 07:09:18.502219 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:09:18.502189    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7368ec91-3412-4c08-b895-955fafebd42f-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ct89n\" (UID: \"7368ec91-3412-4c08-b895-955fafebd42f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ct89n"
Jul 07 07:09:18.502405 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:09:18.502389    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7368ec91-3412-4c08-b895-955fafebd42f-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ct89n\" (UID: \"7368ec91-3412-4c08-b895-955fafebd42f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ct89n"
Jul 07 07:09:18.510028 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:09:18.510003    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwvfz\" (UniqueName: \"kubernetes.io/projected/7368ec91-3412-4c08-b895-955fafebd42f-kube-api-access-hwvfz\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ct89n\" (UID: \"7368ec91-3412-4c08-b895-955fafebd42f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ct89n"
Jul 07 07:09:18.611600 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:09:18.611513    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ct89n"
Jul 07 07:09:18.739745 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:09:18.739714    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-ct89n"]
Jul 07 07:09:18.742698 ip-10-0-138-97 kubenswrapper[2579]: W0707 07:09:18.742669    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7368ec91_3412_4c08_b895_955fafebd42f.slice/crio-8eba8f4752888240bad8ba055385280e0d937b376fa2eddd1bd05b26391b40c3 WatchSource:0}: Error finding container 8eba8f4752888240bad8ba055385280e0d937b376fa2eddd1bd05b26391b40c3: Status 404 returned error can't find the container with id 8eba8f4752888240bad8ba055385280e0d937b376fa2eddd1bd05b26391b40c3
Jul 07 07:09:18.743935 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:09:18.743916    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 07:09:19.565120 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:09:19.565079    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ct89n" event={"ID":"7368ec91-3412-4c08-b895-955fafebd42f","Type":"ContainerStarted","Data":"8eba8f4752888240bad8ba055385280e0d937b376fa2eddd1bd05b26391b40c3"}
Jul 07 07:09:20.570928 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:09:20.570891    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ct89n" event={"ID":"7368ec91-3412-4c08-b895-955fafebd42f","Type":"ContainerStarted","Data":"e50ee33df9ca9d1a2e68acbc628a5f38daa16d43ba4b258a12baf6976574b05e"}
Jul 07 07:09:20.589867 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:09:20.589810    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ct89n" podStartSLOduration=1.497658019 podStartE2EDuration="2.589795828s" totalImagesPullingTime="1.092137809s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 07:09:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 07:09:20.588066294 +0000 UTC m=+640.131827773" watchObservedRunningTime="2026-07-07 07:09:20.589795828 +0000 UTC m=+640.133557306"
Jul 07 07:09:22.897782 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:09:22.897750    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-s265r"]
Jul 07 07:09:22.899945 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:09:22.899928    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s265r"
Jul 07 07:09:22.902831 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:09:22.902806    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Jul 07 07:09:22.916263 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:09:22.916243    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-s265r"]
Jul 07 07:09:23.040878 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:09:23.040840    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e5077c3-eff4-4f30-84dc-14614e08cca2-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s265r\" (UID: \"7e5077c3-eff4-4f30-84dc-14614e08cca2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s265r"
Jul 07 07:09:23.041064 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:09:23.040894    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e5077c3-eff4-4f30-84dc-14614e08cca2-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s265r\" (UID: \"7e5077c3-eff4-4f30-84dc-14614e08cca2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s265r"
Jul 07 07:09:23.041064 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:09:23.041012    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e5077c3-eff4-4f30-84dc-14614e08cca2-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s265r\" (UID: \"7e5077c3-eff4-4f30-84dc-14614e08cca2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s265r"
Jul 07 07:09:23.041064 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:09:23.041045    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w267\" (UniqueName: \"kubernetes.io/projected/7e5077c3-eff4-4f30-84dc-14614e08cca2-kube-api-access-4w267\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s265r\" (UID: \"7e5077c3-eff4-4f30-84dc-14614e08cca2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s265r"
Jul 07 07:09:23.041167 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:09:23.041110    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e5077c3-eff4-4f30-84dc-14614e08cca2-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s265r\" (UID: \"7e5077c3-eff4-4f30-84dc-14614e08cca2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s265r"
Jul 07 07:09:23.041167 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:09:23.041130    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e5077c3-eff4-4f30-84dc-14614e08cca2-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s265r\" (UID: \"7e5077c3-eff4-4f30-84dc-14614e08cca2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s265r"
Jul 07 07:09:23.141709 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:09:23.141669    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e5077c3-eff4-4f30-84dc-14614e08cca2-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s265r\" (UID: \"7e5077c3-eff4-4f30-84dc-14614e08cca2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s265r"
Jul 07 07:09:23.141904 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:09:23.141725    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e5077c3-eff4-4f30-84dc-14614e08cca2-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s265r\" (UID: \"7e5077c3-eff4-4f30-84dc-14614e08cca2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s265r"
Jul 07 07:09:23.141904 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:09:23.141762    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e5077c3-eff4-4f30-84dc-14614e08cca2-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s265r\" (UID: \"7e5077c3-eff4-4f30-84dc-14614e08cca2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s265r"
Jul 07 07:09:23.141904 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:09:23.141782    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w267\" (UniqueName: \"kubernetes.io/projected/7e5077c3-eff4-4f30-84dc-14614e08cca2-kube-api-access-4w267\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s265r\" (UID: \"7e5077c3-eff4-4f30-84dc-14614e08cca2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s265r"
Jul 07 07:09:23.141904 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:09:23.141813    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e5077c3-eff4-4f30-84dc-14614e08cca2-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s265r\" (UID: \"7e5077c3-eff4-4f30-84dc-14614e08cca2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s265r"
Jul 07 07:09:23.142134 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:09:23.141987    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e5077c3-eff4-4f30-84dc-14614e08cca2-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s265r\" (UID: \"7e5077c3-eff4-4f30-84dc-14614e08cca2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s265r"
Jul 07 07:09:23.142189 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:09:23.142146    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e5077c3-eff4-4f30-84dc-14614e08cca2-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s265r\" (UID: \"7e5077c3-eff4-4f30-84dc-14614e08cca2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s265r"
Jul 07 07:09:23.142275 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:09:23.142239    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e5077c3-eff4-4f30-84dc-14614e08cca2-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s265r\" (UID: \"7e5077c3-eff4-4f30-84dc-14614e08cca2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s265r"
Jul 07 07:09:23.142348 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:09:23.142330    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e5077c3-eff4-4f30-84dc-14614e08cca2-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s265r\" (UID: \"7e5077c3-eff4-4f30-84dc-14614e08cca2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s265r"
Jul 07 07:09:23.144157 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:09:23.144137    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e5077c3-eff4-4f30-84dc-14614e08cca2-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s265r\" (UID: \"7e5077c3-eff4-4f30-84dc-14614e08cca2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s265r"
Jul 07 07:09:23.144423 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:09:23.144409    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e5077c3-eff4-4f30-84dc-14614e08cca2-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s265r\" (UID: \"7e5077c3-eff4-4f30-84dc-14614e08cca2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s265r"
Jul 07 07:09:23.151255 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:09:23.151203    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w267\" (UniqueName: \"kubernetes.io/projected/7e5077c3-eff4-4f30-84dc-14614e08cca2-kube-api-access-4w267\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s265r\" (UID: \"7e5077c3-eff4-4f30-84dc-14614e08cca2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s265r"
Jul 07 07:09:23.209152 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:09:23.209113    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s265r"
Jul 07 07:09:23.340079 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:09:23.340050    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-s265r"]
Jul 07 07:09:23.342568 ip-10-0-138-97 kubenswrapper[2579]: W0707 07:09:23.342543    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e5077c3_eff4_4f30_84dc_14614e08cca2.slice/crio-2a26da547ddc237f58d642feacdc42a4b07eb5834e322e31816c627cd12c55a2 WatchSource:0}: Error finding container 2a26da547ddc237f58d642feacdc42a4b07eb5834e322e31816c627cd12c55a2: Status 404 returned error can't find the container with id 2a26da547ddc237f58d642feacdc42a4b07eb5834e322e31816c627cd12c55a2
Jul 07 07:09:23.585143 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:09:23.585116    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s265r" event={"ID":"7e5077c3-eff4-4f30-84dc-14614e08cca2","Type":"ContainerStarted","Data":"af96bc209f98b015034438f0bd0a3f5d1b844c0e4a1e6f5f29cb9033954277d2"}
Jul 07 07:09:23.585254 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:09:23.585169    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s265r" event={"ID":"7e5077c3-eff4-4f30-84dc-14614e08cca2","Type":"ContainerStarted","Data":"2a26da547ddc237f58d642feacdc42a4b07eb5834e322e31816c627cd12c55a2"}
Jul 07 07:09:24.610876 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:09:24.610816    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s265r" podStartSLOduration=2.446892219 podStartE2EDuration="2.610799387s" totalImagesPullingTime="163.907168ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 07:09:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 07:09:24.608805218 +0000 UTC m=+644.152566699" watchObservedRunningTime="2026-07-07 07:09:24.610799387 +0000 UTC m=+644.154560875"
Jul 07 07:09:28.612074 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:09:28.612040    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ct89n"
Jul 07 07:09:28.612475 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:09:28.612102    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ct89n"
Jul 07 07:09:28.624412 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:09:28.624384    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ct89n"
Jul 07 07:09:29.620784 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:09:29.620753    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ct89n"
Jul 07 07:09:33.209536 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:09:33.209497    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s265r"
Jul 07 07:09:33.209985 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:09:33.209566    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s265r"
Jul 07 07:09:33.222031 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:09:33.222003    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s265r"
Jul 07 07:09:33.636917 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:09:33.636888    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s265r"
Jul 07 07:10:08.089979 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:10:08.089918    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-kjkht"]
Jul 07 07:10:08.093412 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:10:08.093389    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-kjkht"
Jul 07 07:10:08.096142 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:10:08.096120    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 07 07:10:08.103434 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:10:08.103411    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-kjkht"]
Jul 07 07:10:08.230680 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:10:08.230635    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8t8r\" (UniqueName: \"kubernetes.io/projected/2db964dd-d624-4187-8126-8705719cfe18-kube-api-access-z8t8r\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-kjkht\" (UID: \"2db964dd-d624-4187-8126-8705719cfe18\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-kjkht"
Jul 07 07:10:08.230867 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:10:08.230726    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2db964dd-d624-4187-8126-8705719cfe18-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-kjkht\" (UID: \"2db964dd-d624-4187-8126-8705719cfe18\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-kjkht"
Jul 07 07:10:08.230867 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:10:08.230747    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2db964dd-d624-4187-8126-8705719cfe18-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-kjkht\" (UID: \"2db964dd-d624-4187-8126-8705719cfe18\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-kjkht"
Jul 07 07:10:08.230867 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:10:08.230792    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2db964dd-d624-4187-8126-8705719cfe18-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-kjkht\" (UID: \"2db964dd-d624-4187-8126-8705719cfe18\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-kjkht"
Jul 07 07:10:08.230867 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:10:08.230815    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2db964dd-d624-4187-8126-8705719cfe18-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-kjkht\" (UID: \"2db964dd-d624-4187-8126-8705719cfe18\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-kjkht"
Jul 07 07:10:08.230867 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:10:08.230847    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2db964dd-d624-4187-8126-8705719cfe18-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-kjkht\" (UID: \"2db964dd-d624-4187-8126-8705719cfe18\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-kjkht"
Jul 07 07:10:08.332104 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:10:08.332056    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2db964dd-d624-4187-8126-8705719cfe18-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-kjkht\" (UID: \"2db964dd-d624-4187-8126-8705719cfe18\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-kjkht"
Jul 07 07:10:08.332104 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:10:08.332105    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2db964dd-d624-4187-8126-8705719cfe18-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-kjkht\" (UID: \"2db964dd-d624-4187-8126-8705719cfe18\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-kjkht"
Jul 07 07:10:08.332355 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:10:08.332152    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2db964dd-d624-4187-8126-8705719cfe18-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-kjkht\" (UID: \"2db964dd-d624-4187-8126-8705719cfe18\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-kjkht"
Jul 07 07:10:08.332355 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:10:08.332187    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2db964dd-d624-4187-8126-8705719cfe18-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-kjkht\" (UID: \"2db964dd-d624-4187-8126-8705719cfe18\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-kjkht"
Jul 07 07:10:08.332355 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:10:08.332221    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2db964dd-d624-4187-8126-8705719cfe18-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-kjkht\" (UID: \"2db964dd-d624-4187-8126-8705719cfe18\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-kjkht"
Jul 07 07:10:08.332355 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:10:08.332250    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8t8r\" (UniqueName: \"kubernetes.io/projected/2db964dd-d624-4187-8126-8705719cfe18-kube-api-access-z8t8r\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-kjkht\" (UID: \"2db964dd-d624-4187-8126-8705719cfe18\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-kjkht"
Jul 07 07:10:08.332580 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:10:08.332555    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2db964dd-d624-4187-8126-8705719cfe18-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-kjkht\" (UID: \"2db964dd-d624-4187-8126-8705719cfe18\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-kjkht"
Jul 07 07:10:08.332641 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:10:08.332579    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2db964dd-d624-4187-8126-8705719cfe18-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-kjkht\" (UID: \"2db964dd-d624-4187-8126-8705719cfe18\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-kjkht"
Jul 07 07:10:08.332641 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:10:08.332621    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2db964dd-d624-4187-8126-8705719cfe18-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-kjkht\" (UID: \"2db964dd-d624-4187-8126-8705719cfe18\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-kjkht"
Jul 07 07:10:08.334419 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:10:08.334398    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2db964dd-d624-4187-8126-8705719cfe18-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-kjkht\" (UID: \"2db964dd-d624-4187-8126-8705719cfe18\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-kjkht"
Jul 07 07:10:08.334672 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:10:08.334650    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2db964dd-d624-4187-8126-8705719cfe18-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-kjkht\" (UID: \"2db964dd-d624-4187-8126-8705719cfe18\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-kjkht"
Jul 07 07:10:08.345246 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:10:08.345192    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8t8r\" (UniqueName: \"kubernetes.io/projected/2db964dd-d624-4187-8126-8705719cfe18-kube-api-access-z8t8r\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-kjkht\" (UID: \"2db964dd-d624-4187-8126-8705719cfe18\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-kjkht"
Jul 07 07:10:08.403147 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:10:08.403103    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-kjkht"
Jul 07 07:10:08.538662 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:10:08.538632    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-kjkht"]
Jul 07 07:10:08.541211 ip-10-0-138-97 kubenswrapper[2579]: W0707 07:10:08.541180    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2db964dd_d624_4187_8126_8705719cfe18.slice/crio-c2f322531e525eb22f140de635b2c04beb7a93887fc85fb7bd69c63aa6235a5b WatchSource:0}: Error finding container c2f322531e525eb22f140de635b2c04beb7a93887fc85fb7bd69c63aa6235a5b: Status 404 returned error can't find the container with id c2f322531e525eb22f140de635b2c04beb7a93887fc85fb7bd69c63aa6235a5b
Jul 07 07:10:08.778220 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:10:08.778170    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-kjkht" event={"ID":"2db964dd-d624-4187-8126-8705719cfe18","Type":"ContainerStarted","Data":"c2f322531e525eb22f140de635b2c04beb7a93887fc85fb7bd69c63aa6235a5b"}
Jul 07 07:10:09.784024 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:10:09.783984    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-kjkht" event={"ID":"2db964dd-d624-4187-8126-8705719cfe18","Type":"ContainerStarted","Data":"06e8190dd4b83dbe87accf9ef15c49468d3bde75dbf133e3d5cfaeff21f32aec"}
Jul 07 07:10:09.811698 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:10:09.811637    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-kjkht" podStartSLOduration=1.391817031 podStartE2EDuration="1.811621409s" totalImagesPullingTime="419.804378ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 07:10:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 07:10:09.810711505 +0000 UTC m=+689.354472988" watchObservedRunningTime="2026-07-07 07:10:09.811621409 +0000 UTC m=+689.355382889"
Jul 07 07:10:18.403524 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:10:18.403483    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-kjkht"
Jul 07 07:10:18.403524 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:10:18.403528    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-kjkht"
Jul 07 07:10:18.415853 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:10:18.415826    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-kjkht"
Jul 07 07:10:18.840517 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:10:18.840485    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-kjkht"
Jul 07 07:10:51.238937 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:10:51.238858    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-68575fcd4c-j27b8"]
Jul 07 07:10:51.242237 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:10:51.242217    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68575fcd4c-j27b8"
Jul 07 07:10:51.251505 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:10:51.251482    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-68575fcd4c-j27b8"]
Jul 07 07:10:51.423740 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:10:51.423705    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6a4ff039-bad1-4d23-8dc8-b8b7844b3034-tls-cert\") pod \"authorino-68575fcd4c-j27b8\" (UID: \"6a4ff039-bad1-4d23-8dc8-b8b7844b3034\") " pod="kuadrant-system/authorino-68575fcd4c-j27b8"
Jul 07 07:10:51.423914 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:10:51.423757    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8fm6\" (UniqueName: \"kubernetes.io/projected/6a4ff039-bad1-4d23-8dc8-b8b7844b3034-kube-api-access-m8fm6\") pod \"authorino-68575fcd4c-j27b8\" (UID: \"6a4ff039-bad1-4d23-8dc8-b8b7844b3034\") " pod="kuadrant-system/authorino-68575fcd4c-j27b8"
Jul 07 07:10:51.525208 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:10:51.525126    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6a4ff039-bad1-4d23-8dc8-b8b7844b3034-tls-cert\") pod \"authorino-68575fcd4c-j27b8\" (UID: \"6a4ff039-bad1-4d23-8dc8-b8b7844b3034\") " pod="kuadrant-system/authorino-68575fcd4c-j27b8"
Jul 07 07:10:51.525208 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:10:51.525179    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8fm6\" (UniqueName: \"kubernetes.io/projected/6a4ff039-bad1-4d23-8dc8-b8b7844b3034-kube-api-access-m8fm6\") pod \"authorino-68575fcd4c-j27b8\" (UID: \"6a4ff039-bad1-4d23-8dc8-b8b7844b3034\") " pod="kuadrant-system/authorino-68575fcd4c-j27b8"
Jul 07 07:10:51.527529 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:10:51.527500    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6a4ff039-bad1-4d23-8dc8-b8b7844b3034-tls-cert\") pod \"authorino-68575fcd4c-j27b8\" (UID: \"6a4ff039-bad1-4d23-8dc8-b8b7844b3034\") " pod="kuadrant-system/authorino-68575fcd4c-j27b8"
Jul 07 07:10:51.542889 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:10:51.542864    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8fm6\" (UniqueName: \"kubernetes.io/projected/6a4ff039-bad1-4d23-8dc8-b8b7844b3034-kube-api-access-m8fm6\") pod \"authorino-68575fcd4c-j27b8\" (UID: \"6a4ff039-bad1-4d23-8dc8-b8b7844b3034\") " pod="kuadrant-system/authorino-68575fcd4c-j27b8"
Jul 07 07:10:51.551658 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:10:51.551635    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68575fcd4c-j27b8"
Jul 07 07:10:51.691024 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:10:51.690993    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-68575fcd4c-j27b8"]
Jul 07 07:10:51.694333 ip-10-0-138-97 kubenswrapper[2579]: W0707 07:10:51.694299    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6a4ff039_bad1_4d23_8dc8_b8b7844b3034.slice/crio-f9463f914ddc307acd4bc5f79b7dc02509b3eb2e9ccf8de26086d5e7314ec94b WatchSource:0}: Error finding container f9463f914ddc307acd4bc5f79b7dc02509b3eb2e9ccf8de26086d5e7314ec94b: Status 404 returned error can't find the container with id f9463f914ddc307acd4bc5f79b7dc02509b3eb2e9ccf8de26086d5e7314ec94b
Jul 07 07:10:51.972028 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:10:51.971932    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68575fcd4c-j27b8" event={"ID":"6a4ff039-bad1-4d23-8dc8-b8b7844b3034","Type":"ContainerStarted","Data":"f9463f914ddc307acd4bc5f79b7dc02509b3eb2e9ccf8de26086d5e7314ec94b"}
Jul 07 07:10:52.979614 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:10:52.979567    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68575fcd4c-j27b8" event={"ID":"6a4ff039-bad1-4d23-8dc8-b8b7844b3034","Type":"ContainerStarted","Data":"d9d6bbb4ee9a9a6d3212c203d69fc271c1b9bfc34a9a0aa0a6014ddade63b732"}
Jul 07 07:10:53.004130 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:10:53.004065    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-68575fcd4c-j27b8" podStartSLOduration=1.523835979 podStartE2EDuration="2.004047719s" totalImagesPullingTime="480.21174ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 07:10:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 07:10:53.002602668 +0000 UTC m=+732.546364149" watchObservedRunningTime="2026-07-07 07:10:53.004047719 +0000 UTC m=+732.547809200"
Jul 07 07:10:53.049824 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:10:53.049787    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6bc8cd784b-cc8kh"]
Jul 07 07:10:53.050049 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:10:53.050020    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-6bc8cd784b-cc8kh" podUID="7513a611-6905-46ee-861d-1577820b5973" containerName="authorino" containerID="cri-o://34e0dd639da9c1949ed9548d80ba90bcf45fd6836779053f74e3d0e2c3554f9f" gracePeriod=30
Jul 07 07:10:53.306057 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:10:53.306030    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6bc8cd784b-cc8kh"
Jul 07 07:10:53.440632 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:10:53.440597    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7513a611-6905-46ee-861d-1577820b5973-tls-cert\") pod \"7513a611-6905-46ee-861d-1577820b5973\" (UID: \"7513a611-6905-46ee-861d-1577820b5973\") "
Jul 07 07:10:53.440813 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:10:53.440660    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x2dq5\" (UniqueName: \"kubernetes.io/projected/7513a611-6905-46ee-861d-1577820b5973-kube-api-access-x2dq5\") pod \"7513a611-6905-46ee-861d-1577820b5973\" (UID: \"7513a611-6905-46ee-861d-1577820b5973\") "
Jul 07 07:10:53.442915 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:10:53.442878    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7513a611-6905-46ee-861d-1577820b5973-kube-api-access-x2dq5" (OuterVolumeSpecName: "kube-api-access-x2dq5") pod "7513a611-6905-46ee-861d-1577820b5973" (UID: "7513a611-6905-46ee-861d-1577820b5973"). InnerVolumeSpecName "kube-api-access-x2dq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 07:10:53.450197 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:10:53.450167    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7513a611-6905-46ee-861d-1577820b5973-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "7513a611-6905-46ee-861d-1577820b5973" (UID: "7513a611-6905-46ee-861d-1577820b5973"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 07:10:53.541880 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:10:53.541800    2579 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7513a611-6905-46ee-861d-1577820b5973-tls-cert\") on node \"ip-10-0-138-97.ec2.internal\" DevicePath \"\""
Jul 07 07:10:53.541880 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:10:53.541846    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2dq5\" (UniqueName: \"kubernetes.io/projected/7513a611-6905-46ee-861d-1577820b5973-kube-api-access-x2dq5\") on node \"ip-10-0-138-97.ec2.internal\" DevicePath \"\""
Jul 07 07:10:53.984868 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:10:53.984827    2579 generic.go:356] "Generic (PLEG): container finished" podID="7513a611-6905-46ee-861d-1577820b5973" containerID="34e0dd639da9c1949ed9548d80ba90bcf45fd6836779053f74e3d0e2c3554f9f" exitCode=0
Jul 07 07:10:53.984868 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:10:53.984865    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6bc8cd784b-cc8kh"
Jul 07 07:10:53.985433 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:10:53.984901    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6bc8cd784b-cc8kh" event={"ID":"7513a611-6905-46ee-861d-1577820b5973","Type":"ContainerDied","Data":"34e0dd639da9c1949ed9548d80ba90bcf45fd6836779053f74e3d0e2c3554f9f"}
Jul 07 07:10:53.985433 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:10:53.984941    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6bc8cd784b-cc8kh" event={"ID":"7513a611-6905-46ee-861d-1577820b5973","Type":"ContainerDied","Data":"67ade16afad8863f518312d23414fd99ad4d942627cfaf53f89a3db27527e491"}
Jul 07 07:10:53.985433 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:10:53.984975    2579 scope.go:117] "RemoveContainer" containerID="34e0dd639da9c1949ed9548d80ba90bcf45fd6836779053f74e3d0e2c3554f9f"
Jul 07 07:10:53.993788 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:10:53.993615    2579 scope.go:117] "RemoveContainer" containerID="34e0dd639da9c1949ed9548d80ba90bcf45fd6836779053f74e3d0e2c3554f9f"
Jul 07 07:10:53.994037 ip-10-0-138-97 kubenswrapper[2579]: E0707 07:10:53.994006    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34e0dd639da9c1949ed9548d80ba90bcf45fd6836779053f74e3d0e2c3554f9f\": container with ID starting with 34e0dd639da9c1949ed9548d80ba90bcf45fd6836779053f74e3d0e2c3554f9f not found: ID does not exist" containerID="34e0dd639da9c1949ed9548d80ba90bcf45fd6836779053f74e3d0e2c3554f9f"
Jul 07 07:10:53.994131 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:10:53.994048    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34e0dd639da9c1949ed9548d80ba90bcf45fd6836779053f74e3d0e2c3554f9f"} err="failed to get container status \"34e0dd639da9c1949ed9548d80ba90bcf45fd6836779053f74e3d0e2c3554f9f\": rpc error: code = NotFound desc = could not find container \"34e0dd639da9c1949ed9548d80ba90bcf45fd6836779053f74e3d0e2c3554f9f\": container with ID starting with 34e0dd639da9c1949ed9548d80ba90bcf45fd6836779053f74e3d0e2c3554f9f not found: ID does not exist"
Jul 07 07:10:54.021069 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:10:54.021030    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6bc8cd784b-cc8kh"]
Jul 07 07:10:54.021755 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:10:54.021734    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6bc8cd784b-cc8kh"]
Jul 07 07:10:54.963364 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:10:54.963328    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7513a611-6905-46ee-861d-1577820b5973" path="/var/lib/kubelet/pods/7513a611-6905-46ee-861d-1577820b5973/volumes"
Jul 07 07:27:19.500904 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:27:19.500860    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-18ebf0cb/e2e-lifecycle-model-18ebf0cb-kserve-675955ffb7-b68wh"]
Jul 07 07:27:19.501553 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:27:19.501330    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7513a611-6905-46ee-861d-1577820b5973" containerName="authorino"
Jul 07 07:27:19.501553 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:27:19.501351    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7513a611-6905-46ee-861d-1577820b5973" containerName="authorino"
Jul 07 07:27:19.501553 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:27:19.501459    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7513a611-6905-46ee-861d-1577820b5973" containerName="authorino"
Jul 07 07:27:19.504468 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:27:19.504446    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-18ebf0cb/e2e-lifecycle-model-18ebf0cb-kserve-675955ffb7-b68wh"
Jul 07 07:27:19.508254 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:27:19.508229    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-18ebf0cb\"/\"kube-root-ca.crt\""
Jul 07 07:27:19.508254 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:27:19.508240    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-18ebf0cb\"/\"e2e-lifecycle-model-18ebf0cb-kserve-self-signed-certs\""
Jul 07 07:27:19.508439 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:27:19.508229    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-18ebf0cb\"/\"openshift-service-ca.crt\""
Jul 07 07:27:19.508439 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:27:19.508323    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-18ebf0cb\"/\"default-dockercfg-gq446\""
Jul 07 07:27:19.513475 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:27:19.513448    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-18ebf0cb/e2e-lifecycle-model-18ebf0cb-kserve-675955ffb7-b68wh"]
Jul 07 07:27:19.557911 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:27:19.557873    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eebb79c9-822f-40e7-9f09-a9dd01c90d47-model-cache\") pod \"e2e-lifecycle-model-18ebf0cb-kserve-675955ffb7-b68wh\" (UID: \"eebb79c9-822f-40e7-9f09-a9dd01c90d47\") " pod="ai-tenant-e2e-mt-18ebf0cb/e2e-lifecycle-model-18ebf0cb-kserve-675955ffb7-b68wh"
Jul 07 07:27:19.558122 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:27:19.557937    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eebb79c9-822f-40e7-9f09-a9dd01c90d47-tls-certs\") pod \"e2e-lifecycle-model-18ebf0cb-kserve-675955ffb7-b68wh\" (UID: \"eebb79c9-822f-40e7-9f09-a9dd01c90d47\") " pod="ai-tenant-e2e-mt-18ebf0cb/e2e-lifecycle-model-18ebf0cb-kserve-675955ffb7-b68wh"
Jul 07 07:27:19.558122 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:27:19.557976    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eebb79c9-822f-40e7-9f09-a9dd01c90d47-home\") pod \"e2e-lifecycle-model-18ebf0cb-kserve-675955ffb7-b68wh\" (UID: \"eebb79c9-822f-40e7-9f09-a9dd01c90d47\") " pod="ai-tenant-e2e-mt-18ebf0cb/e2e-lifecycle-model-18ebf0cb-kserve-675955ffb7-b68wh"
Jul 07 07:27:19.558122 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:27:19.558004    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eebb79c9-822f-40e7-9f09-a9dd01c90d47-dshm\") pod \"e2e-lifecycle-model-18ebf0cb-kserve-675955ffb7-b68wh\" (UID: \"eebb79c9-822f-40e7-9f09-a9dd01c90d47\") " pod="ai-tenant-e2e-mt-18ebf0cb/e2e-lifecycle-model-18ebf0cb-kserve-675955ffb7-b68wh"
Jul 07 07:27:19.558122 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:27:19.558063    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eebb79c9-822f-40e7-9f09-a9dd01c90d47-tmp-dir\") pod \"e2e-lifecycle-model-18ebf0cb-kserve-675955ffb7-b68wh\" (UID: \"eebb79c9-822f-40e7-9f09-a9dd01c90d47\") " pod="ai-tenant-e2e-mt-18ebf0cb/e2e-lifecycle-model-18ebf0cb-kserve-675955ffb7-b68wh"
Jul 07 07:27:19.558122 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:27:19.558110    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86vxf\" (UniqueName: \"kubernetes.io/projected/eebb79c9-822f-40e7-9f09-a9dd01c90d47-kube-api-access-86vxf\") pod \"e2e-lifecycle-model-18ebf0cb-kserve-675955ffb7-b68wh\" (UID: \"eebb79c9-822f-40e7-9f09-a9dd01c90d47\") " pod="ai-tenant-e2e-mt-18ebf0cb/e2e-lifecycle-model-18ebf0cb-kserve-675955ffb7-b68wh"
Jul 07 07:27:19.658773 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:27:19.658737    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86vxf\" (UniqueName: \"kubernetes.io/projected/eebb79c9-822f-40e7-9f09-a9dd01c90d47-kube-api-access-86vxf\") pod \"e2e-lifecycle-model-18ebf0cb-kserve-675955ffb7-b68wh\" (UID: \"eebb79c9-822f-40e7-9f09-a9dd01c90d47\") " pod="ai-tenant-e2e-mt-18ebf0cb/e2e-lifecycle-model-18ebf0cb-kserve-675955ffb7-b68wh"
Jul 07 07:27:19.658773 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:27:19.658777    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eebb79c9-822f-40e7-9f09-a9dd01c90d47-model-cache\") pod \"e2e-lifecycle-model-18ebf0cb-kserve-675955ffb7-b68wh\" (UID: \"eebb79c9-822f-40e7-9f09-a9dd01c90d47\") " pod="ai-tenant-e2e-mt-18ebf0cb/e2e-lifecycle-model-18ebf0cb-kserve-675955ffb7-b68wh"
Jul 07 07:27:19.659044 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:27:19.658810    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eebb79c9-822f-40e7-9f09-a9dd01c90d47-tls-certs\") pod \"e2e-lifecycle-model-18ebf0cb-kserve-675955ffb7-b68wh\" (UID: \"eebb79c9-822f-40e7-9f09-a9dd01c90d47\") " pod="ai-tenant-e2e-mt-18ebf0cb/e2e-lifecycle-model-18ebf0cb-kserve-675955ffb7-b68wh"
Jul 07 07:27:19.659044 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:27:19.658828    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eebb79c9-822f-40e7-9f09-a9dd01c90d47-home\") pod \"e2e-lifecycle-model-18ebf0cb-kserve-675955ffb7-b68wh\" (UID: \"eebb79c9-822f-40e7-9f09-a9dd01c90d47\") " pod="ai-tenant-e2e-mt-18ebf0cb/e2e-lifecycle-model-18ebf0cb-kserve-675955ffb7-b68wh"
Jul 07 07:27:19.659044 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:27:19.658846    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eebb79c9-822f-40e7-9f09-a9dd01c90d47-dshm\") pod \"e2e-lifecycle-model-18ebf0cb-kserve-675955ffb7-b68wh\" (UID: \"eebb79c9-822f-40e7-9f09-a9dd01c90d47\") " pod="ai-tenant-e2e-mt-18ebf0cb/e2e-lifecycle-model-18ebf0cb-kserve-675955ffb7-b68wh"
Jul 07 07:27:19.659044 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:27:19.658874    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eebb79c9-822f-40e7-9f09-a9dd01c90d47-tmp-dir\") pod \"e2e-lifecycle-model-18ebf0cb-kserve-675955ffb7-b68wh\" (UID: \"eebb79c9-822f-40e7-9f09-a9dd01c90d47\") " pod="ai-tenant-e2e-mt-18ebf0cb/e2e-lifecycle-model-18ebf0cb-kserve-675955ffb7-b68wh"
Jul 07 07:27:19.659312 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:27:19.659288    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eebb79c9-822f-40e7-9f09-a9dd01c90d47-model-cache\") pod \"e2e-lifecycle-model-18ebf0cb-kserve-675955ffb7-b68wh\" (UID: \"eebb79c9-822f-40e7-9f09-a9dd01c90d47\") " pod="ai-tenant-e2e-mt-18ebf0cb/e2e-lifecycle-model-18ebf0cb-kserve-675955ffb7-b68wh"
Jul 07 07:27:19.659312 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:27:19.659303    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eebb79c9-822f-40e7-9f09-a9dd01c90d47-tmp-dir\") pod \"e2e-lifecycle-model-18ebf0cb-kserve-675955ffb7-b68wh\" (UID: \"eebb79c9-822f-40e7-9f09-a9dd01c90d47\") " pod="ai-tenant-e2e-mt-18ebf0cb/e2e-lifecycle-model-18ebf0cb-kserve-675955ffb7-b68wh"
Jul 07 07:27:19.659413 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:27:19.659363    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eebb79c9-822f-40e7-9f09-a9dd01c90d47-home\") pod \"e2e-lifecycle-model-18ebf0cb-kserve-675955ffb7-b68wh\" (UID: \"eebb79c9-822f-40e7-9f09-a9dd01c90d47\") " pod="ai-tenant-e2e-mt-18ebf0cb/e2e-lifecycle-model-18ebf0cb-kserve-675955ffb7-b68wh"
Jul 07 07:27:19.661244 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:27:19.661215    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eebb79c9-822f-40e7-9f09-a9dd01c90d47-dshm\") pod \"e2e-lifecycle-model-18ebf0cb-kserve-675955ffb7-b68wh\" (UID: \"eebb79c9-822f-40e7-9f09-a9dd01c90d47\") " pod="ai-tenant-e2e-mt-18ebf0cb/e2e-lifecycle-model-18ebf0cb-kserve-675955ffb7-b68wh"
Jul 07 07:27:19.661490 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:27:19.661472    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eebb79c9-822f-40e7-9f09-a9dd01c90d47-tls-certs\") pod \"e2e-lifecycle-model-18ebf0cb-kserve-675955ffb7-b68wh\" (UID: \"eebb79c9-822f-40e7-9f09-a9dd01c90d47\") " pod="ai-tenant-e2e-mt-18ebf0cb/e2e-lifecycle-model-18ebf0cb-kserve-675955ffb7-b68wh"
Jul 07 07:27:19.667629 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:27:19.667606    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86vxf\" (UniqueName: \"kubernetes.io/projected/eebb79c9-822f-40e7-9f09-a9dd01c90d47-kube-api-access-86vxf\") pod \"e2e-lifecycle-model-18ebf0cb-kserve-675955ffb7-b68wh\" (UID: \"eebb79c9-822f-40e7-9f09-a9dd01c90d47\") " pod="ai-tenant-e2e-mt-18ebf0cb/e2e-lifecycle-model-18ebf0cb-kserve-675955ffb7-b68wh"
Jul 07 07:27:19.813893 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:27:19.813809    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-18ebf0cb/e2e-lifecycle-model-18ebf0cb-kserve-675955ffb7-b68wh"
Jul 07 07:27:19.962651 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:27:19.962629    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-18ebf0cb/e2e-lifecycle-model-18ebf0cb-kserve-675955ffb7-b68wh"]
Jul 07 07:27:19.964917 ip-10-0-138-97 kubenswrapper[2579]: W0707 07:27:19.964888    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeebb79c9_822f_40e7_9f09_a9dd01c90d47.slice/crio-ee442c533215c1f702daa9a9ccc28430b2d1b631f11112d16cdd559c090aca5c WatchSource:0}: Error finding container ee442c533215c1f702daa9a9ccc28430b2d1b631f11112d16cdd559c090aca5c: Status 404 returned error can't find the container with id ee442c533215c1f702daa9a9ccc28430b2d1b631f11112d16cdd559c090aca5c
Jul 07 07:27:20.244336 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:27:20.244303    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-18ebf0cb/e2e-lifecycle-model-18ebf0cb-kserve-675955ffb7-b68wh" event={"ID":"eebb79c9-822f-40e7-9f09-a9dd01c90d47","Type":"ContainerStarted","Data":"c964bcb6d02ff4a05b88811c9e12008aed9fc96479eca9c499c3939c26b0ec8c"}
Jul 07 07:27:20.244515 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:27:20.244346    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-18ebf0cb/e2e-lifecycle-model-18ebf0cb-kserve-675955ffb7-b68wh" event={"ID":"eebb79c9-822f-40e7-9f09-a9dd01c90d47","Type":"ContainerStarted","Data":"ee442c533215c1f702daa9a9ccc28430b2d1b631f11112d16cdd559c090aca5c"}
Jul 07 07:27:20.266487 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:27:20.266434    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-18ebf0cb/e2e-lifecycle-model-18ebf0cb-kserve-675955ffb7-b68wh" podStartSLOduration=1.266419547 podStartE2EDuration="1.266419547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 07:27:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 07:27:20.26527126 +0000 UTC m=+1719.809032742" watchObservedRunningTime="2026-07-07 07:27:20.266419547 +0000 UTC m=+1719.810181028"
Jul 07 07:27:29.814285 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:27:29.814246    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-18ebf0cb/e2e-lifecycle-model-18ebf0cb-kserve-675955ffb7-b68wh"
Jul 07 07:27:29.814285 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:27:29.814296    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-18ebf0cb/e2e-lifecycle-model-18ebf0cb-kserve-675955ffb7-b68wh"
Jul 07 07:27:29.827140 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:27:29.827112    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-18ebf0cb/e2e-lifecycle-model-18ebf0cb-kserve-675955ffb7-b68wh"
Jul 07 07:27:30.300068 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:27:30.300033    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-18ebf0cb/e2e-lifecycle-model-18ebf0cb-kserve-675955ffb7-b68wh"
Jul 07 07:28:05.166926 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:28:05.166892    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-18ebf0cb/e2e-lifecycle-model-18ebf0cb-kserve-675955ffb7-b68wh"]
Jul 07 07:28:20.187414 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:28:20.187364    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-mt-18ebf0cb/e2e-lifecycle-model-18ebf0cb-kserve-675955ffb7-b68wh" podUID="eebb79c9-822f-40e7-9f09-a9dd01c90d47" containerName="main" containerID="cri-o://c964bcb6d02ff4a05b88811c9e12008aed9fc96479eca9c499c3939c26b0ec8c" gracePeriod=45
Jul 07 07:28:20.289049 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:28:20.288995    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-18ebf0cb/e2e-lifecycle-model-18ebf0cb-kserve-675955ffb7-b68wh" podUID="eebb79c9-822f-40e7-9f09-a9dd01c90d47" containerName="main" probeResult="failure" output="Get \"https://10.132.0.32:8000/ready\": dial tcp 10.132.0.32:8000: connect: connection refused"
Jul 07 07:28:20.430334 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:28:20.430305    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-18ebf0cb/e2e-lifecycle-model-18ebf0cb-kserve-675955ffb7-b68wh"
Jul 07 07:28:20.505775 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:28:20.505736    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-86vxf\" (UniqueName: \"kubernetes.io/projected/eebb79c9-822f-40e7-9f09-a9dd01c90d47-kube-api-access-86vxf\") pod \"eebb79c9-822f-40e7-9f09-a9dd01c90d47\" (UID: \"eebb79c9-822f-40e7-9f09-a9dd01c90d47\") "
Jul 07 07:28:20.505951 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:28:20.505830    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eebb79c9-822f-40e7-9f09-a9dd01c90d47-home\") pod \"eebb79c9-822f-40e7-9f09-a9dd01c90d47\" (UID: \"eebb79c9-822f-40e7-9f09-a9dd01c90d47\") "
Jul 07 07:28:20.505951 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:28:20.505866    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eebb79c9-822f-40e7-9f09-a9dd01c90d47-tls-certs\") pod \"eebb79c9-822f-40e7-9f09-a9dd01c90d47\" (UID: \"eebb79c9-822f-40e7-9f09-a9dd01c90d47\") "
Jul 07 07:28:20.506086 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:28:20.505968    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eebb79c9-822f-40e7-9f09-a9dd01c90d47-model-cache\") pod \"eebb79c9-822f-40e7-9f09-a9dd01c90d47\" (UID: \"eebb79c9-822f-40e7-9f09-a9dd01c90d47\") "
Jul 07 07:28:20.506086 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:28:20.505994    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eebb79c9-822f-40e7-9f09-a9dd01c90d47-tmp-dir\") pod \"eebb79c9-822f-40e7-9f09-a9dd01c90d47\" (UID: \"eebb79c9-822f-40e7-9f09-a9dd01c90d47\") "
Jul 07 07:28:20.506086 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:28:20.506028    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eebb79c9-822f-40e7-9f09-a9dd01c90d47-dshm\") pod \"eebb79c9-822f-40e7-9f09-a9dd01c90d47\" (UID: \"eebb79c9-822f-40e7-9f09-a9dd01c90d47\") "
Jul 07 07:28:20.506256 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:28:20.506092    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eebb79c9-822f-40e7-9f09-a9dd01c90d47-home" (OuterVolumeSpecName: "home") pod "eebb79c9-822f-40e7-9f09-a9dd01c90d47" (UID: "eebb79c9-822f-40e7-9f09-a9dd01c90d47"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 07:28:20.506326 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:28:20.506296    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eebb79c9-822f-40e7-9f09-a9dd01c90d47-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "eebb79c9-822f-40e7-9f09-a9dd01c90d47" (UID: "eebb79c9-822f-40e7-9f09-a9dd01c90d47"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 07:28:20.506326 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:28:20.506310    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eebb79c9-822f-40e7-9f09-a9dd01c90d47-model-cache" (OuterVolumeSpecName: "model-cache") pod "eebb79c9-822f-40e7-9f09-a9dd01c90d47" (UID: "eebb79c9-822f-40e7-9f09-a9dd01c90d47"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 07:28:20.506428 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:28:20.506395    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eebb79c9-822f-40e7-9f09-a9dd01c90d47-model-cache\") on node \"ip-10-0-138-97.ec2.internal\" DevicePath \"\""
Jul 07 07:28:20.506428 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:28:20.506414    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eebb79c9-822f-40e7-9f09-a9dd01c90d47-tmp-dir\") on node \"ip-10-0-138-97.ec2.internal\" DevicePath \"\""
Jul 07 07:28:20.506541 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:28:20.506429    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eebb79c9-822f-40e7-9f09-a9dd01c90d47-home\") on node \"ip-10-0-138-97.ec2.internal\" DevicePath \"\""
Jul 07 07:28:20.508536 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:28:20.508507    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eebb79c9-822f-40e7-9f09-a9dd01c90d47-kube-api-access-86vxf" (OuterVolumeSpecName: "kube-api-access-86vxf") pod "eebb79c9-822f-40e7-9f09-a9dd01c90d47" (UID: "eebb79c9-822f-40e7-9f09-a9dd01c90d47"). InnerVolumeSpecName "kube-api-access-86vxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 07:28:20.508668 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:28:20.508538    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eebb79c9-822f-40e7-9f09-a9dd01c90d47-dshm" (OuterVolumeSpecName: "dshm") pod "eebb79c9-822f-40e7-9f09-a9dd01c90d47" (UID: "eebb79c9-822f-40e7-9f09-a9dd01c90d47"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 07:28:20.508668 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:28:20.508606    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eebb79c9-822f-40e7-9f09-a9dd01c90d47-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "eebb79c9-822f-40e7-9f09-a9dd01c90d47" (UID: "eebb79c9-822f-40e7-9f09-a9dd01c90d47"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 07:28:20.511364 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:28:20.511336    2579 generic.go:356] "Generic (PLEG): container finished" podID="eebb79c9-822f-40e7-9f09-a9dd01c90d47" containerID="c964bcb6d02ff4a05b88811c9e12008aed9fc96479eca9c499c3939c26b0ec8c" exitCode=0
Jul 07 07:28:20.511468 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:28:20.511382    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-18ebf0cb/e2e-lifecycle-model-18ebf0cb-kserve-675955ffb7-b68wh"
Jul 07 07:28:20.511468 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:28:20.511393    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-18ebf0cb/e2e-lifecycle-model-18ebf0cb-kserve-675955ffb7-b68wh" event={"ID":"eebb79c9-822f-40e7-9f09-a9dd01c90d47","Type":"ContainerDied","Data":"c964bcb6d02ff4a05b88811c9e12008aed9fc96479eca9c499c3939c26b0ec8c"}
Jul 07 07:28:20.511468 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:28:20.511432    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-18ebf0cb/e2e-lifecycle-model-18ebf0cb-kserve-675955ffb7-b68wh" event={"ID":"eebb79c9-822f-40e7-9f09-a9dd01c90d47","Type":"ContainerDied","Data":"ee442c533215c1f702daa9a9ccc28430b2d1b631f11112d16cdd559c090aca5c"}
Jul 07 07:28:20.511468 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:28:20.511455    2579 scope.go:117] "RemoveContainer" containerID="c964bcb6d02ff4a05b88811c9e12008aed9fc96479eca9c499c3939c26b0ec8c"
Jul 07 07:28:20.529512 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:28:20.529490    2579 scope.go:117] "RemoveContainer" containerID="c964bcb6d02ff4a05b88811c9e12008aed9fc96479eca9c499c3939c26b0ec8c"
Jul 07 07:28:20.529850 ip-10-0-138-97 kubenswrapper[2579]: E0707 07:28:20.529831    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c964bcb6d02ff4a05b88811c9e12008aed9fc96479eca9c499c3939c26b0ec8c\": container with ID starting with c964bcb6d02ff4a05b88811c9e12008aed9fc96479eca9c499c3939c26b0ec8c not found: ID does not exist" containerID="c964bcb6d02ff4a05b88811c9e12008aed9fc96479eca9c499c3939c26b0ec8c"
Jul 07 07:28:20.529916 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:28:20.529860    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c964bcb6d02ff4a05b88811c9e12008aed9fc96479eca9c499c3939c26b0ec8c"} err="failed to get container status \"c964bcb6d02ff4a05b88811c9e12008aed9fc96479eca9c499c3939c26b0ec8c\": rpc error: code = NotFound desc = could not find container \"c964bcb6d02ff4a05b88811c9e12008aed9fc96479eca9c499c3939c26b0ec8c\": container with ID starting with c964bcb6d02ff4a05b88811c9e12008aed9fc96479eca9c499c3939c26b0ec8c not found: ID does not exist"
Jul 07 07:28:20.544511 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:28:20.544483    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-18ebf0cb/e2e-lifecycle-model-18ebf0cb-kserve-675955ffb7-b68wh"]
Jul 07 07:28:20.547169 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:28:20.547147    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-18ebf0cb/e2e-lifecycle-model-18ebf0cb-kserve-675955ffb7-b68wh"]
Jul 07 07:28:20.606977 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:28:20.606925    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86vxf\" (UniqueName: \"kubernetes.io/projected/eebb79c9-822f-40e7-9f09-a9dd01c90d47-kube-api-access-86vxf\") on node \"ip-10-0-138-97.ec2.internal\" DevicePath \"\""
Jul 07 07:28:20.607139 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:28:20.606990    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eebb79c9-822f-40e7-9f09-a9dd01c90d47-tls-certs\") on node \"ip-10-0-138-97.ec2.internal\" DevicePath \"\""
Jul 07 07:28:20.607139 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:28:20.607006    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eebb79c9-822f-40e7-9f09-a9dd01c90d47-dshm\") on node \"ip-10-0-138-97.ec2.internal\" DevicePath \"\""
Jul 07 07:28:20.964008 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:28:20.963899    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eebb79c9-822f-40e7-9f09-a9dd01c90d47" path="/var/lib/kubelet/pods/eebb79c9-822f-40e7-9f09-a9dd01c90d47/volumes"
Jul 07 07:30:00.135534 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:30:00.135499    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29723490-7crxf"]
Jul 07 07:30:00.136024 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:30:00.135828    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eebb79c9-822f-40e7-9f09-a9dd01c90d47" containerName="main"
Jul 07 07:30:00.136024 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:30:00.135839    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="eebb79c9-822f-40e7-9f09-a9dd01c90d47" containerName="main"
Jul 07 07:30:00.136024 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:30:00.135898    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="eebb79c9-822f-40e7-9f09-a9dd01c90d47" containerName="main"
Jul 07 07:30:00.138827 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:30:00.138809    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29723490-7crxf"
Jul 07 07:30:00.141812 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:30:00.141795    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-z6nck\""
Jul 07 07:30:00.150338 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:30:00.150314    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29723490-7crxf"]
Jul 07 07:30:00.287973 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:30:00.287930    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zb96\" (UniqueName: \"kubernetes.io/projected/87666c82-44bb-4505-a97f-870202bd331e-kube-api-access-9zb96\") pod \"maas-api-key-cleanup-29723490-7crxf\" (UID: \"87666c82-44bb-4505-a97f-870202bd331e\") " pod="opendatahub/maas-api-key-cleanup-29723490-7crxf"
Jul 07 07:30:00.388558 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:30:00.388469    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zb96\" (UniqueName: \"kubernetes.io/projected/87666c82-44bb-4505-a97f-870202bd331e-kube-api-access-9zb96\") pod \"maas-api-key-cleanup-29723490-7crxf\" (UID: \"87666c82-44bb-4505-a97f-870202bd331e\") " pod="opendatahub/maas-api-key-cleanup-29723490-7crxf"
Jul 07 07:30:00.398650 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:30:00.398625    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zb96\" (UniqueName: \"kubernetes.io/projected/87666c82-44bb-4505-a97f-870202bd331e-kube-api-access-9zb96\") pod \"maas-api-key-cleanup-29723490-7crxf\" (UID: \"87666c82-44bb-4505-a97f-870202bd331e\") " pod="opendatahub/maas-api-key-cleanup-29723490-7crxf"
Jul 07 07:30:00.449582 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:30:00.449547    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29723490-7crxf"
Jul 07 07:30:00.578116 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:30:00.578094    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29723490-7crxf"]
Jul 07 07:30:00.580291 ip-10-0-138-97 kubenswrapper[2579]: W0707 07:30:00.580261    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87666c82_44bb_4505_a97f_870202bd331e.slice/crio-85c15a183e9f60c992f55cff577cf06f29213401e669b1ccd19602559d5e6143 WatchSource:0}: Error finding container 85c15a183e9f60c992f55cff577cf06f29213401e669b1ccd19602559d5e6143: Status 404 returned error can't find the container with id 85c15a183e9f60c992f55cff577cf06f29213401e669b1ccd19602559d5e6143
Jul 07 07:30:00.582150 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:30:00.582134    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 07:30:00.944114 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:30:00.944078    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29723490-7crxf" event={"ID":"87666c82-44bb-4505-a97f-870202bd331e","Type":"ContainerStarted","Data":"85c15a183e9f60c992f55cff577cf06f29213401e669b1ccd19602559d5e6143"}
Jul 07 07:30:03.960597 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:30:03.960503    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29723490-7crxf" event={"ID":"87666c82-44bb-4505-a97f-870202bd331e","Type":"ContainerStarted","Data":"b755c22182f35ddb658dd308bc48b41147de7cdaea6969021f741c47eea80043"}
Jul 07 07:30:03.980808 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:30:03.980749    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-key-cleanup-29723490-7crxf" podStartSLOduration=1.332870309 podStartE2EDuration="3.980734623s" totalImagesPullingTime="2.647864314s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 07:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 07:30:03.978992553 +0000 UTC m=+1883.522754038" watchObservedRunningTime="2026-07-07 07:30:03.980734623 +0000 UTC m=+1883.524496103"
Jul 07 07:30:04.965888 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:30:04.965849    2579 generic.go:356] "Generic (PLEG): container finished" podID="87666c82-44bb-4505-a97f-870202bd331e" containerID="b755c22182f35ddb658dd308bc48b41147de7cdaea6969021f741c47eea80043" exitCode=0
Jul 07 07:30:04.966322 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:30:04.965918    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29723490-7crxf" event={"ID":"87666c82-44bb-4505-a97f-870202bd331e","Type":"ContainerDied","Data":"b755c22182f35ddb658dd308bc48b41147de7cdaea6969021f741c47eea80043"}
Jul 07 07:30:06.098842 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:30:06.098812    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29723490-7crxf"
Jul 07 07:30:06.241189 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:30:06.241151    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9zb96\" (UniqueName: \"kubernetes.io/projected/87666c82-44bb-4505-a97f-870202bd331e-kube-api-access-9zb96\") pod \"87666c82-44bb-4505-a97f-870202bd331e\" (UID: \"87666c82-44bb-4505-a97f-870202bd331e\") "
Jul 07 07:30:06.243369 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:30:06.243346    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87666c82-44bb-4505-a97f-870202bd331e-kube-api-access-9zb96" (OuterVolumeSpecName: "kube-api-access-9zb96") pod "87666c82-44bb-4505-a97f-870202bd331e" (UID: "87666c82-44bb-4505-a97f-870202bd331e"). InnerVolumeSpecName "kube-api-access-9zb96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 07:30:06.342447 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:30:06.342407    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zb96\" (UniqueName: \"kubernetes.io/projected/87666c82-44bb-4505-a97f-870202bd331e-kube-api-access-9zb96\") on node \"ip-10-0-138-97.ec2.internal\" DevicePath \"\""
Jul 07 07:30:06.976094 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:30:06.976062    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29723490-7crxf" event={"ID":"87666c82-44bb-4505-a97f-870202bd331e","Type":"ContainerDied","Data":"85c15a183e9f60c992f55cff577cf06f29213401e669b1ccd19602559d5e6143"}
Jul 07 07:30:06.976094 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:30:06.976083    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29723490-7crxf"
Jul 07 07:30:06.976268 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:30:06.976113    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85c15a183e9f60c992f55cff577cf06f29213401e669b1ccd19602559d5e6143"
Jul 07 07:30:17.700001 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:30:17.699944    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-ebf5a0/test-model-ebf5a0-kserve-5bc47945f-vmckb"]
Jul 07 07:30:17.700432 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:30:17.700275    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87666c82-44bb-4505-a97f-870202bd331e" containerName="cleanup"
Jul 07 07:30:17.700432 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:30:17.700287    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="87666c82-44bb-4505-a97f-870202bd331e" containerName="cleanup"
Jul 07 07:30:17.700432 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:30:17.700347    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="87666c82-44bb-4505-a97f-870202bd331e" containerName="cleanup"
Jul 07 07:30:17.703192 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:30:17.703175    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ebf5a0/test-model-ebf5a0-kserve-5bc47945f-vmckb"
Jul 07 07:30:17.709050 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:30:17.709023    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-ebf5a0\"/\"kube-root-ca.crt\""
Jul 07 07:30:17.709182 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:30:17.709050    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-ebf5a0\"/\"default-dockercfg-j9wx7\""
Jul 07 07:30:17.709182 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:30:17.709036    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-ebf5a0\"/\"openshift-service-ca.crt\""
Jul 07 07:30:17.709182 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:30:17.709034    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-ebf5a0\"/\"test-model-ebf5a0-kserve-self-signed-certs\""
Jul 07 07:30:17.718418 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:30:17.718395    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-ebf5a0/test-model-ebf5a0-kserve-5bc47945f-vmckb"]
Jul 07 07:30:17.746007 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:30:17.745981    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fe198f6-279f-4436-b07b-d4cd0e149bcb-dshm\") pod \"test-model-ebf5a0-kserve-5bc47945f-vmckb\" (UID: \"5fe198f6-279f-4436-b07b-d4cd0e149bcb\") " pod="ai-tenant-e2e-inf-a-ebf5a0/test-model-ebf5a0-kserve-5bc47945f-vmckb"
Jul 07 07:30:17.746127 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:30:17.746021    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fe198f6-279f-4436-b07b-d4cd0e149bcb-tmp-dir\") pod \"test-model-ebf5a0-kserve-5bc47945f-vmckb\" (UID: \"5fe198f6-279f-4436-b07b-d4cd0e149bcb\") " pod="ai-tenant-e2e-inf-a-ebf5a0/test-model-ebf5a0-kserve-5bc47945f-vmckb"
Jul 07 07:30:17.746127 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:30:17.746037    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fe198f6-279f-4436-b07b-d4cd0e149bcb-model-cache\") pod \"test-model-ebf5a0-kserve-5bc47945f-vmckb\" (UID: \"5fe198f6-279f-4436-b07b-d4cd0e149bcb\") " pod="ai-tenant-e2e-inf-a-ebf5a0/test-model-ebf5a0-kserve-5bc47945f-vmckb"
Jul 07 07:30:17.746127 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:30:17.746103    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fe198f6-279f-4436-b07b-d4cd0e149bcb-tls-certs\") pod \"test-model-ebf5a0-kserve-5bc47945f-vmckb\" (UID: \"5fe198f6-279f-4436-b07b-d4cd0e149bcb\") " pod="ai-tenant-e2e-inf-a-ebf5a0/test-model-ebf5a0-kserve-5bc47945f-vmckb"
Jul 07 07:30:17.746286 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:30:17.746143    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fe198f6-279f-4436-b07b-d4cd0e149bcb-home\") pod \"test-model-ebf5a0-kserve-5bc47945f-vmckb\" (UID: \"5fe198f6-279f-4436-b07b-d4cd0e149bcb\") " pod="ai-tenant-e2e-inf-a-ebf5a0/test-model-ebf5a0-kserve-5bc47945f-vmckb"
Jul 07 07:30:17.746286 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:30:17.746160    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj454\" (UniqueName: \"kubernetes.io/projected/5fe198f6-279f-4436-b07b-d4cd0e149bcb-kube-api-access-zj454\") pod \"test-model-ebf5a0-kserve-5bc47945f-vmckb\" (UID: \"5fe198f6-279f-4436-b07b-d4cd0e149bcb\") " pod="ai-tenant-e2e-inf-a-ebf5a0/test-model-ebf5a0-kserve-5bc47945f-vmckb"
Jul 07 07:30:17.846623 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:30:17.846584    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fe198f6-279f-4436-b07b-d4cd0e149bcb-tmp-dir\") pod \"test-model-ebf5a0-kserve-5bc47945f-vmckb\" (UID: \"5fe198f6-279f-4436-b07b-d4cd0e149bcb\") " pod="ai-tenant-e2e-inf-a-ebf5a0/test-model-ebf5a0-kserve-5bc47945f-vmckb"
Jul 07 07:30:17.846623 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:30:17.846622    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fe198f6-279f-4436-b07b-d4cd0e149bcb-model-cache\") pod \"test-model-ebf5a0-kserve-5bc47945f-vmckb\" (UID: \"5fe198f6-279f-4436-b07b-d4cd0e149bcb\") " pod="ai-tenant-e2e-inf-a-ebf5a0/test-model-ebf5a0-kserve-5bc47945f-vmckb"
Jul 07 07:30:17.846874 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:30:17.846644    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fe198f6-279f-4436-b07b-d4cd0e149bcb-tls-certs\") pod \"test-model-ebf5a0-kserve-5bc47945f-vmckb\" (UID: \"5fe198f6-279f-4436-b07b-d4cd0e149bcb\") " pod="ai-tenant-e2e-inf-a-ebf5a0/test-model-ebf5a0-kserve-5bc47945f-vmckb"
Jul 07 07:30:17.846874 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:30:17.846675    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fe198f6-279f-4436-b07b-d4cd0e149bcb-home\") pod \"test-model-ebf5a0-kserve-5bc47945f-vmckb\" (UID: \"5fe198f6-279f-4436-b07b-d4cd0e149bcb\") " pod="ai-tenant-e2e-inf-a-ebf5a0/test-model-ebf5a0-kserve-5bc47945f-vmckb"
Jul 07 07:30:17.846874 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:30:17.846694    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj454\" (UniqueName: \"kubernetes.io/projected/5fe198f6-279f-4436-b07b-d4cd0e149bcb-kube-api-access-zj454\") pod \"test-model-ebf5a0-kserve-5bc47945f-vmckb\" (UID: \"5fe198f6-279f-4436-b07b-d4cd0e149bcb\") " pod="ai-tenant-e2e-inf-a-ebf5a0/test-model-ebf5a0-kserve-5bc47945f-vmckb"
Jul 07 07:30:17.846874 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:30:17.846757    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fe198f6-279f-4436-b07b-d4cd0e149bcb-dshm\") pod \"test-model-ebf5a0-kserve-5bc47945f-vmckb\" (UID: \"5fe198f6-279f-4436-b07b-d4cd0e149bcb\") " pod="ai-tenant-e2e-inf-a-ebf5a0/test-model-ebf5a0-kserve-5bc47945f-vmckb"
Jul 07 07:30:17.847091 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:30:17.847075    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fe198f6-279f-4436-b07b-d4cd0e149bcb-model-cache\") pod \"test-model-ebf5a0-kserve-5bc47945f-vmckb\" (UID: \"5fe198f6-279f-4436-b07b-d4cd0e149bcb\") " pod="ai-tenant-e2e-inf-a-ebf5a0/test-model-ebf5a0-kserve-5bc47945f-vmckb"
Jul 07 07:30:17.847150 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:30:17.847091    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fe198f6-279f-4436-b07b-d4cd0e149bcb-tmp-dir\") pod \"test-model-ebf5a0-kserve-5bc47945f-vmckb\" (UID: \"5fe198f6-279f-4436-b07b-d4cd0e149bcb\") " pod="ai-tenant-e2e-inf-a-ebf5a0/test-model-ebf5a0-kserve-5bc47945f-vmckb"
Jul 07 07:30:17.847150 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:30:17.847127    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fe198f6-279f-4436-b07b-d4cd0e149bcb-home\") pod \"test-model-ebf5a0-kserve-5bc47945f-vmckb\" (UID: \"5fe198f6-279f-4436-b07b-d4cd0e149bcb\") " pod="ai-tenant-e2e-inf-a-ebf5a0/test-model-ebf5a0-kserve-5bc47945f-vmckb"
Jul 07 07:30:17.848980 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:30:17.848931    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fe198f6-279f-4436-b07b-d4cd0e149bcb-dshm\") pod \"test-model-ebf5a0-kserve-5bc47945f-vmckb\" (UID: \"5fe198f6-279f-4436-b07b-d4cd0e149bcb\") " pod="ai-tenant-e2e-inf-a-ebf5a0/test-model-ebf5a0-kserve-5bc47945f-vmckb"
Jul 07 07:30:17.849227 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:30:17.849202    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fe198f6-279f-4436-b07b-d4cd0e149bcb-tls-certs\") pod \"test-model-ebf5a0-kserve-5bc47945f-vmckb\" (UID: \"5fe198f6-279f-4436-b07b-d4cd0e149bcb\") " pod="ai-tenant-e2e-inf-a-ebf5a0/test-model-ebf5a0-kserve-5bc47945f-vmckb"
Jul 07 07:30:17.866698 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:30:17.866674    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj454\" (UniqueName: \"kubernetes.io/projected/5fe198f6-279f-4436-b07b-d4cd0e149bcb-kube-api-access-zj454\") pod \"test-model-ebf5a0-kserve-5bc47945f-vmckb\" (UID: \"5fe198f6-279f-4436-b07b-d4cd0e149bcb\") " pod="ai-tenant-e2e-inf-a-ebf5a0/test-model-ebf5a0-kserve-5bc47945f-vmckb"
Jul 07 07:30:18.012666 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:30:18.012631    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ebf5a0/test-model-ebf5a0-kserve-5bc47945f-vmckb"
Jul 07 07:30:18.147798 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:30:18.147651    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-ebf5a0/test-model-ebf5a0-kserve-5bc47945f-vmckb"]
Jul 07 07:30:18.150526 ip-10-0-138-97 kubenswrapper[2579]: W0707 07:30:18.150502    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5fe198f6_279f_4436_b07b_d4cd0e149bcb.slice/crio-e29a7952b9476cfa2db58fc611ebbf7a610f5c85956b037758773baef59d399a WatchSource:0}: Error finding container e29a7952b9476cfa2db58fc611ebbf7a610f5c85956b037758773baef59d399a: Status 404 returned error can't find the container with id e29a7952b9476cfa2db58fc611ebbf7a610f5c85956b037758773baef59d399a
Jul 07 07:30:19.027594 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:30:19.027559    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ebf5a0/test-model-ebf5a0-kserve-5bc47945f-vmckb" event={"ID":"5fe198f6-279f-4436-b07b-d4cd0e149bcb","Type":"ContainerStarted","Data":"0dc619cc8578f31184944905574c5e99edb9649179cdd20233e41fdc36a34fe8"}
Jul 07 07:30:19.028006 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:30:19.027601    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ebf5a0/test-model-ebf5a0-kserve-5bc47945f-vmckb" event={"ID":"5fe198f6-279f-4436-b07b-d4cd0e149bcb","Type":"ContainerStarted","Data":"e29a7952b9476cfa2db58fc611ebbf7a610f5c85956b037758773baef59d399a"}
Jul 07 07:30:19.049073 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:30:19.049019    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-ebf5a0/test-model-ebf5a0-kserve-5bc47945f-vmckb" podStartSLOduration=2.049004053 podStartE2EDuration="2.049004053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 07:30:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 07:30:19.047609524 +0000 UTC m=+1898.591371006" watchObservedRunningTime="2026-07-07 07:30:19.049004053 +0000 UTC m=+1898.592765533"
Jul 07 07:30:28.013297 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:30:28.013254    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-ebf5a0/test-model-ebf5a0-kserve-5bc47945f-vmckb"
Jul 07 07:30:28.013748 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:30:28.013319    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-ebf5a0/test-model-ebf5a0-kserve-5bc47945f-vmckb"
Jul 07 07:30:28.025793 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:30:28.025763    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-ebf5a0/test-model-ebf5a0-kserve-5bc47945f-vmckb"
Jul 07 07:30:28.078721 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:30:28.078688    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-ebf5a0/test-model-ebf5a0-kserve-5bc47945f-vmckb"
Jul 07 07:31:04.304661 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:04.304624    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-e309b3/test-model-e309b3-kserve-6474954cb5-ff6wz"]
Jul 07 07:31:04.308089 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:04.308072    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e309b3/test-model-e309b3-kserve-6474954cb5-ff6wz"
Jul 07 07:31:04.312145 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:04.312125    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-e309b3\"/\"default-dockercfg-sb8bj\""
Jul 07 07:31:04.312278 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:04.312157    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-e309b3\"/\"openshift-service-ca.crt\""
Jul 07 07:31:04.312278 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:04.312131    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-e309b3\"/\"test-model-e309b3-kserve-self-signed-certs\""
Jul 07 07:31:04.312278 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:04.312208    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-e309b3\"/\"kube-root-ca.crt\""
Jul 07 07:31:04.318540 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:04.318519    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-e309b3/test-model-e309b3-kserve-6474954cb5-ff6wz"]
Jul 07 07:31:04.367787 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:04.367752    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed0fe067-71e2-4583-a156-96e6539c756b-dshm\") pod \"test-model-e309b3-kserve-6474954cb5-ff6wz\" (UID: \"ed0fe067-71e2-4583-a156-96e6539c756b\") " pod="ai-tenant-e2e-inf-b-e309b3/test-model-e309b3-kserve-6474954cb5-ff6wz"
Jul 07 07:31:04.367787 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:04.367788    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed0fe067-71e2-4583-a156-96e6539c756b-model-cache\") pod \"test-model-e309b3-kserve-6474954cb5-ff6wz\" (UID: \"ed0fe067-71e2-4583-a156-96e6539c756b\") " pod="ai-tenant-e2e-inf-b-e309b3/test-model-e309b3-kserve-6474954cb5-ff6wz"
Jul 07 07:31:04.368062 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:04.367809    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed0fe067-71e2-4583-a156-96e6539c756b-home\") pod \"test-model-e309b3-kserve-6474954cb5-ff6wz\" (UID: \"ed0fe067-71e2-4583-a156-96e6539c756b\") " pod="ai-tenant-e2e-inf-b-e309b3/test-model-e309b3-kserve-6474954cb5-ff6wz"
Jul 07 07:31:04.368062 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:04.367828    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nmq8\" (UniqueName: \"kubernetes.io/projected/ed0fe067-71e2-4583-a156-96e6539c756b-kube-api-access-6nmq8\") pod \"test-model-e309b3-kserve-6474954cb5-ff6wz\" (UID: \"ed0fe067-71e2-4583-a156-96e6539c756b\") " pod="ai-tenant-e2e-inf-b-e309b3/test-model-e309b3-kserve-6474954cb5-ff6wz"
Jul 07 07:31:04.368062 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:04.367935    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed0fe067-71e2-4583-a156-96e6539c756b-tmp-dir\") pod \"test-model-e309b3-kserve-6474954cb5-ff6wz\" (UID: \"ed0fe067-71e2-4583-a156-96e6539c756b\") " pod="ai-tenant-e2e-inf-b-e309b3/test-model-e309b3-kserve-6474954cb5-ff6wz"
Jul 07 07:31:04.368062 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:04.368030    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed0fe067-71e2-4583-a156-96e6539c756b-tls-certs\") pod \"test-model-e309b3-kserve-6474954cb5-ff6wz\" (UID: \"ed0fe067-71e2-4583-a156-96e6539c756b\") " pod="ai-tenant-e2e-inf-b-e309b3/test-model-e309b3-kserve-6474954cb5-ff6wz"
Jul 07 07:31:04.469390 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:04.469356    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed0fe067-71e2-4583-a156-96e6539c756b-tmp-dir\") pod \"test-model-e309b3-kserve-6474954cb5-ff6wz\" (UID: \"ed0fe067-71e2-4583-a156-96e6539c756b\") " pod="ai-tenant-e2e-inf-b-e309b3/test-model-e309b3-kserve-6474954cb5-ff6wz"
Jul 07 07:31:04.469561 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:04.469411    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed0fe067-71e2-4583-a156-96e6539c756b-tls-certs\") pod \"test-model-e309b3-kserve-6474954cb5-ff6wz\" (UID: \"ed0fe067-71e2-4583-a156-96e6539c756b\") " pod="ai-tenant-e2e-inf-b-e309b3/test-model-e309b3-kserve-6474954cb5-ff6wz"
Jul 07 07:31:04.469561 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:04.469463    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed0fe067-71e2-4583-a156-96e6539c756b-dshm\") pod \"test-model-e309b3-kserve-6474954cb5-ff6wz\" (UID: \"ed0fe067-71e2-4583-a156-96e6539c756b\") " pod="ai-tenant-e2e-inf-b-e309b3/test-model-e309b3-kserve-6474954cb5-ff6wz"
Jul 07 07:31:04.469561 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:04.469493    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed0fe067-71e2-4583-a156-96e6539c756b-model-cache\") pod \"test-model-e309b3-kserve-6474954cb5-ff6wz\" (UID: \"ed0fe067-71e2-4583-a156-96e6539c756b\") " pod="ai-tenant-e2e-inf-b-e309b3/test-model-e309b3-kserve-6474954cb5-ff6wz"
Jul 07 07:31:04.469561 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:04.469519    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed0fe067-71e2-4583-a156-96e6539c756b-home\") pod \"test-model-e309b3-kserve-6474954cb5-ff6wz\" (UID: \"ed0fe067-71e2-4583-a156-96e6539c756b\") " pod="ai-tenant-e2e-inf-b-e309b3/test-model-e309b3-kserve-6474954cb5-ff6wz"
Jul 07 07:31:04.469561 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:04.469541    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nmq8\" (UniqueName: \"kubernetes.io/projected/ed0fe067-71e2-4583-a156-96e6539c756b-kube-api-access-6nmq8\") pod \"test-model-e309b3-kserve-6474954cb5-ff6wz\" (UID: \"ed0fe067-71e2-4583-a156-96e6539c756b\") " pod="ai-tenant-e2e-inf-b-e309b3/test-model-e309b3-kserve-6474954cb5-ff6wz"
Jul 07 07:31:04.469882 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:04.469850    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed0fe067-71e2-4583-a156-96e6539c756b-tmp-dir\") pod \"test-model-e309b3-kserve-6474954cb5-ff6wz\" (UID: \"ed0fe067-71e2-4583-a156-96e6539c756b\") " pod="ai-tenant-e2e-inf-b-e309b3/test-model-e309b3-kserve-6474954cb5-ff6wz"
Jul 07 07:31:04.470023 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:04.469884    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed0fe067-71e2-4583-a156-96e6539c756b-model-cache\") pod \"test-model-e309b3-kserve-6474954cb5-ff6wz\" (UID: \"ed0fe067-71e2-4583-a156-96e6539c756b\") " pod="ai-tenant-e2e-inf-b-e309b3/test-model-e309b3-kserve-6474954cb5-ff6wz"
Jul 07 07:31:04.470023 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:04.469933    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed0fe067-71e2-4583-a156-96e6539c756b-home\") pod \"test-model-e309b3-kserve-6474954cb5-ff6wz\" (UID: \"ed0fe067-71e2-4583-a156-96e6539c756b\") " pod="ai-tenant-e2e-inf-b-e309b3/test-model-e309b3-kserve-6474954cb5-ff6wz"
Jul 07 07:31:04.471734 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:04.471709    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed0fe067-71e2-4583-a156-96e6539c756b-dshm\") pod \"test-model-e309b3-kserve-6474954cb5-ff6wz\" (UID: \"ed0fe067-71e2-4583-a156-96e6539c756b\") " pod="ai-tenant-e2e-inf-b-e309b3/test-model-e309b3-kserve-6474954cb5-ff6wz"
Jul 07 07:31:04.472025 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:04.472009    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed0fe067-71e2-4583-a156-96e6539c756b-tls-certs\") pod \"test-model-e309b3-kserve-6474954cb5-ff6wz\" (UID: \"ed0fe067-71e2-4583-a156-96e6539c756b\") " pod="ai-tenant-e2e-inf-b-e309b3/test-model-e309b3-kserve-6474954cb5-ff6wz"
Jul 07 07:31:04.479360 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:04.479333    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nmq8\" (UniqueName: \"kubernetes.io/projected/ed0fe067-71e2-4583-a156-96e6539c756b-kube-api-access-6nmq8\") pod \"test-model-e309b3-kserve-6474954cb5-ff6wz\" (UID: \"ed0fe067-71e2-4583-a156-96e6539c756b\") " pod="ai-tenant-e2e-inf-b-e309b3/test-model-e309b3-kserve-6474954cb5-ff6wz"
Jul 07 07:31:04.617722 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:04.617596    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e309b3/test-model-e309b3-kserve-6474954cb5-ff6wz"
Jul 07 07:31:04.750482 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:04.750451    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-e309b3/test-model-e309b3-kserve-6474954cb5-ff6wz"]
Jul 07 07:31:04.753688 ip-10-0-138-97 kubenswrapper[2579]: W0707 07:31:04.753662    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poded0fe067_71e2_4583_a156_96e6539c756b.slice/crio-bd462d25cd8b38cb4215a1ff1304a1ec1a76ff3ce51fe42b288890267df1128e WatchSource:0}: Error finding container bd462d25cd8b38cb4215a1ff1304a1ec1a76ff3ce51fe42b288890267df1128e: Status 404 returned error can't find the container with id bd462d25cd8b38cb4215a1ff1304a1ec1a76ff3ce51fe42b288890267df1128e
Jul 07 07:31:05.232530 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:05.232496    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e309b3/test-model-e309b3-kserve-6474954cb5-ff6wz" event={"ID":"ed0fe067-71e2-4583-a156-96e6539c756b","Type":"ContainerStarted","Data":"3bc69adb986c611b059fe124750b57b235452c8147237e1235ee4cebaa89df6a"}
Jul 07 07:31:05.232530 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:05.232534    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e309b3/test-model-e309b3-kserve-6474954cb5-ff6wz" event={"ID":"ed0fe067-71e2-4583-a156-96e6539c756b","Type":"ContainerStarted","Data":"bd462d25cd8b38cb4215a1ff1304a1ec1a76ff3ce51fe42b288890267df1128e"}
Jul 07 07:31:05.253542 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:05.253491    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-e309b3/test-model-e309b3-kserve-6474954cb5-ff6wz" podStartSLOduration=1.25345774 podStartE2EDuration="1.25345774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 07:31:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 07:31:05.252420061 +0000 UTC m=+1944.796181541" watchObservedRunningTime="2026-07-07 07:31:05.25345774 +0000 UTC m=+1944.797219219"
Jul 07 07:31:14.618701 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:14.618655    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-e309b3/test-model-e309b3-kserve-6474954cb5-ff6wz"
Jul 07 07:31:14.619200 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:14.618727    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-e309b3/test-model-e309b3-kserve-6474954cb5-ff6wz"
Jul 07 07:31:14.631142 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:14.631116    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-e309b3/test-model-e309b3-kserve-6474954cb5-ff6wz"
Jul 07 07:31:15.291664 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:15.291635    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-e309b3/test-model-e309b3-kserve-6474954cb5-ff6wz"
Jul 07 07:31:25.301271 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:25.301236    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-ebf5a0/test-model-ebf5a0-kserve-5bc47945f-vmckb"]
Jul 07 07:31:26.224325 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:26.224291    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-e309b3/test-model-e309b3-kserve-6474954cb5-ff6wz"]
Jul 07 07:31:40.321551 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:40.321448    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-ebf5a0/test-model-ebf5a0-kserve-5bc47945f-vmckb" podUID="5fe198f6-279f-4436-b07b-d4cd0e149bcb" containerName="main" containerID="cri-o://0dc619cc8578f31184944905574c5e99edb9649179cdd20233e41fdc36a34fe8" gracePeriod=45
Jul 07 07:31:40.571404 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:40.571381    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ebf5a0/test-model-ebf5a0-kserve-5bc47945f-vmckb"
Jul 07 07:31:40.700757 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:40.700670    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fe198f6-279f-4436-b07b-d4cd0e149bcb-tls-certs\") pod \"5fe198f6-279f-4436-b07b-d4cd0e149bcb\" (UID: \"5fe198f6-279f-4436-b07b-d4cd0e149bcb\") "
Jul 07 07:31:40.700757 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:40.700722    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fe198f6-279f-4436-b07b-d4cd0e149bcb-model-cache\") pod \"5fe198f6-279f-4436-b07b-d4cd0e149bcb\" (UID: \"5fe198f6-279f-4436-b07b-d4cd0e149bcb\") "
Jul 07 07:31:40.700757 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:40.700757    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fe198f6-279f-4436-b07b-d4cd0e149bcb-home\") pod \"5fe198f6-279f-4436-b07b-d4cd0e149bcb\" (UID: \"5fe198f6-279f-4436-b07b-d4cd0e149bcb\") "
Jul 07 07:31:40.701100 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:40.700831    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fe198f6-279f-4436-b07b-d4cd0e149bcb-tmp-dir\") pod \"5fe198f6-279f-4436-b07b-d4cd0e149bcb\" (UID: \"5fe198f6-279f-4436-b07b-d4cd0e149bcb\") "
Jul 07 07:31:40.701100 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:40.700862    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zj454\" (UniqueName: \"kubernetes.io/projected/5fe198f6-279f-4436-b07b-d4cd0e149bcb-kube-api-access-zj454\") pod \"5fe198f6-279f-4436-b07b-d4cd0e149bcb\" (UID: \"5fe198f6-279f-4436-b07b-d4cd0e149bcb\") "
Jul 07 07:31:40.701100 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:40.700982    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fe198f6-279f-4436-b07b-d4cd0e149bcb-dshm\") pod \"5fe198f6-279f-4436-b07b-d4cd0e149bcb\" (UID: \"5fe198f6-279f-4436-b07b-d4cd0e149bcb\") "
Jul 07 07:31:40.701231 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:40.701128    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fe198f6-279f-4436-b07b-d4cd0e149bcb-home" (OuterVolumeSpecName: "home") pod "5fe198f6-279f-4436-b07b-d4cd0e149bcb" (UID: "5fe198f6-279f-4436-b07b-d4cd0e149bcb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 07:31:40.701231 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:40.701112    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fe198f6-279f-4436-b07b-d4cd0e149bcb-model-cache" (OuterVolumeSpecName: "model-cache") pod "5fe198f6-279f-4436-b07b-d4cd0e149bcb" (UID: "5fe198f6-279f-4436-b07b-d4cd0e149bcb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 07:31:40.701231 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:40.701183    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fe198f6-279f-4436-b07b-d4cd0e149bcb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5fe198f6-279f-4436-b07b-d4cd0e149bcb" (UID: "5fe198f6-279f-4436-b07b-d4cd0e149bcb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 07:31:40.701343 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:40.701286    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fe198f6-279f-4436-b07b-d4cd0e149bcb-tmp-dir\") on node \"ip-10-0-138-97.ec2.internal\" DevicePath \"\""
Jul 07 07:31:40.701343 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:40.701309    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fe198f6-279f-4436-b07b-d4cd0e149bcb-model-cache\") on node \"ip-10-0-138-97.ec2.internal\" DevicePath \"\""
Jul 07 07:31:40.701343 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:40.701322    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fe198f6-279f-4436-b07b-d4cd0e149bcb-home\") on node \"ip-10-0-138-97.ec2.internal\" DevicePath \"\""
Jul 07 07:31:40.703193 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:40.703166    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fe198f6-279f-4436-b07b-d4cd0e149bcb-dshm" (OuterVolumeSpecName: "dshm") pod "5fe198f6-279f-4436-b07b-d4cd0e149bcb" (UID: "5fe198f6-279f-4436-b07b-d4cd0e149bcb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 07:31:40.703297 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:40.703197    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fe198f6-279f-4436-b07b-d4cd0e149bcb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5fe198f6-279f-4436-b07b-d4cd0e149bcb" (UID: "5fe198f6-279f-4436-b07b-d4cd0e149bcb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 07:31:40.703337 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:40.703294    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fe198f6-279f-4436-b07b-d4cd0e149bcb-kube-api-access-zj454" (OuterVolumeSpecName: "kube-api-access-zj454") pod "5fe198f6-279f-4436-b07b-d4cd0e149bcb" (UID: "5fe198f6-279f-4436-b07b-d4cd0e149bcb"). InnerVolumeSpecName "kube-api-access-zj454". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 07:31:40.802394 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:40.802353    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fe198f6-279f-4436-b07b-d4cd0e149bcb-dshm\") on node \"ip-10-0-138-97.ec2.internal\" DevicePath \"\""
Jul 07 07:31:40.802394 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:40.802386    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fe198f6-279f-4436-b07b-d4cd0e149bcb-tls-certs\") on node \"ip-10-0-138-97.ec2.internal\" DevicePath \"\""
Jul 07 07:31:40.802394 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:40.802400    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zj454\" (UniqueName: \"kubernetes.io/projected/5fe198f6-279f-4436-b07b-d4cd0e149bcb-kube-api-access-zj454\") on node \"ip-10-0-138-97.ec2.internal\" DevicePath \"\""
Jul 07 07:31:41.243996 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:41.243936    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-e309b3/test-model-e309b3-kserve-6474954cb5-ff6wz" podUID="ed0fe067-71e2-4583-a156-96e6539c756b" containerName="main" containerID="cri-o://3bc69adb986c611b059fe124750b57b235452c8147237e1235ee4cebaa89df6a" gracePeriod=45
Jul 07 07:31:41.399887 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:41.399853    2579 generic.go:356] "Generic (PLEG): container finished" podID="5fe198f6-279f-4436-b07b-d4cd0e149bcb" containerID="0dc619cc8578f31184944905574c5e99edb9649179cdd20233e41fdc36a34fe8" exitCode=0
Jul 07 07:31:41.400359 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:41.399898    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ebf5a0/test-model-ebf5a0-kserve-5bc47945f-vmckb"
Jul 07 07:31:41.400359 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:41.399936    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ebf5a0/test-model-ebf5a0-kserve-5bc47945f-vmckb" event={"ID":"5fe198f6-279f-4436-b07b-d4cd0e149bcb","Type":"ContainerDied","Data":"0dc619cc8578f31184944905574c5e99edb9649179cdd20233e41fdc36a34fe8"}
Jul 07 07:31:41.400359 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:41.400015    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ebf5a0/test-model-ebf5a0-kserve-5bc47945f-vmckb" event={"ID":"5fe198f6-279f-4436-b07b-d4cd0e149bcb","Type":"ContainerDied","Data":"e29a7952b9476cfa2db58fc611ebbf7a610f5c85956b037758773baef59d399a"}
Jul 07 07:31:41.400359 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:41.400042    2579 scope.go:117] "RemoveContainer" containerID="0dc619cc8578f31184944905574c5e99edb9649179cdd20233e41fdc36a34fe8"
Jul 07 07:31:41.401720 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:41.401681    2579 generic.go:356] "Generic (PLEG): container finished" podID="ed0fe067-71e2-4583-a156-96e6539c756b" containerID="3bc69adb986c611b059fe124750b57b235452c8147237e1235ee4cebaa89df6a" exitCode=0
Jul 07 07:31:41.401720 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:41.401718    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e309b3/test-model-e309b3-kserve-6474954cb5-ff6wz" event={"ID":"ed0fe067-71e2-4583-a156-96e6539c756b","Type":"ContainerDied","Data":"3bc69adb986c611b059fe124750b57b235452c8147237e1235ee4cebaa89df6a"}
Jul 07 07:31:41.408821 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:41.408803    2579 scope.go:117] "RemoveContainer" containerID="0dc619cc8578f31184944905574c5e99edb9649179cdd20233e41fdc36a34fe8"
Jul 07 07:31:41.409108 ip-10-0-138-97 kubenswrapper[2579]: E0707 07:31:41.409091    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dc619cc8578f31184944905574c5e99edb9649179cdd20233e41fdc36a34fe8\": container with ID starting with 0dc619cc8578f31184944905574c5e99edb9649179cdd20233e41fdc36a34fe8 not found: ID does not exist" containerID="0dc619cc8578f31184944905574c5e99edb9649179cdd20233e41fdc36a34fe8"
Jul 07 07:31:41.409173 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:41.409117    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dc619cc8578f31184944905574c5e99edb9649179cdd20233e41fdc36a34fe8"} err="failed to get container status \"0dc619cc8578f31184944905574c5e99edb9649179cdd20233e41fdc36a34fe8\": rpc error: code = NotFound desc = could not find container \"0dc619cc8578f31184944905574c5e99edb9649179cdd20233e41fdc36a34fe8\": container with ID starting with 0dc619cc8578f31184944905574c5e99edb9649179cdd20233e41fdc36a34fe8 not found: ID does not exist"
Jul 07 07:31:41.419731 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:41.419699    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-ebf5a0/test-model-ebf5a0-kserve-5bc47945f-vmckb"]
Jul 07 07:31:41.424254 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:41.424232    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-ebf5a0/test-model-ebf5a0-kserve-5bc47945f-vmckb"]
Jul 07 07:31:41.496711 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:41.496648    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e309b3/test-model-e309b3-kserve-6474954cb5-ff6wz"
Jul 07 07:31:41.609700 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:41.609660    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed0fe067-71e2-4583-a156-96e6539c756b-model-cache\") pod \"ed0fe067-71e2-4583-a156-96e6539c756b\" (UID: \"ed0fe067-71e2-4583-a156-96e6539c756b\") "
Jul 07 07:31:41.609700 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:41.609699    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed0fe067-71e2-4583-a156-96e6539c756b-tmp-dir\") pod \"ed0fe067-71e2-4583-a156-96e6539c756b\" (UID: \"ed0fe067-71e2-4583-a156-96e6539c756b\") "
Jul 07 07:31:41.609931 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:41.609722    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed0fe067-71e2-4583-a156-96e6539c756b-home\") pod \"ed0fe067-71e2-4583-a156-96e6539c756b\" (UID: \"ed0fe067-71e2-4583-a156-96e6539c756b\") "
Jul 07 07:31:41.609931 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:41.609782    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed0fe067-71e2-4583-a156-96e6539c756b-dshm\") pod \"ed0fe067-71e2-4583-a156-96e6539c756b\" (UID: \"ed0fe067-71e2-4583-a156-96e6539c756b\") "
Jul 07 07:31:41.609931 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:41.609808    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed0fe067-71e2-4583-a156-96e6539c756b-tls-certs\") pod \"ed0fe067-71e2-4583-a156-96e6539c756b\" (UID: \"ed0fe067-71e2-4583-a156-96e6539c756b\") "
Jul 07 07:31:41.609931 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:41.609828    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6nmq8\" (UniqueName: \"kubernetes.io/projected/ed0fe067-71e2-4583-a156-96e6539c756b-kube-api-access-6nmq8\") pod \"ed0fe067-71e2-4583-a156-96e6539c756b\" (UID: \"ed0fe067-71e2-4583-a156-96e6539c756b\") "
Jul 07 07:31:41.610172 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:41.610043    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed0fe067-71e2-4583-a156-96e6539c756b-model-cache" (OuterVolumeSpecName: "model-cache") pod "ed0fe067-71e2-4583-a156-96e6539c756b" (UID: "ed0fe067-71e2-4583-a156-96e6539c756b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 07:31:41.610172 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:41.610053    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed0fe067-71e2-4583-a156-96e6539c756b-home" (OuterVolumeSpecName: "home") pod "ed0fe067-71e2-4583-a156-96e6539c756b" (UID: "ed0fe067-71e2-4583-a156-96e6539c756b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 07:31:41.610172 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:41.610065    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed0fe067-71e2-4583-a156-96e6539c756b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ed0fe067-71e2-4583-a156-96e6539c756b" (UID: "ed0fe067-71e2-4583-a156-96e6539c756b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 07:31:41.612143 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:41.612114    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed0fe067-71e2-4583-a156-96e6539c756b-dshm" (OuterVolumeSpecName: "dshm") pod "ed0fe067-71e2-4583-a156-96e6539c756b" (UID: "ed0fe067-71e2-4583-a156-96e6539c756b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 07:31:41.612247 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:41.612137    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed0fe067-71e2-4583-a156-96e6539c756b-kube-api-access-6nmq8" (OuterVolumeSpecName: "kube-api-access-6nmq8") pod "ed0fe067-71e2-4583-a156-96e6539c756b" (UID: "ed0fe067-71e2-4583-a156-96e6539c756b"). InnerVolumeSpecName "kube-api-access-6nmq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 07:31:41.612247 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:41.612163    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed0fe067-71e2-4583-a156-96e6539c756b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ed0fe067-71e2-4583-a156-96e6539c756b" (UID: "ed0fe067-71e2-4583-a156-96e6539c756b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 07:31:41.710895 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:41.710854    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed0fe067-71e2-4583-a156-96e6539c756b-home\") on node \"ip-10-0-138-97.ec2.internal\" DevicePath \"\""
Jul 07 07:31:41.710895 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:41.710889    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed0fe067-71e2-4583-a156-96e6539c756b-dshm\") on node \"ip-10-0-138-97.ec2.internal\" DevicePath \"\""
Jul 07 07:31:41.710895 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:41.710897    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed0fe067-71e2-4583-a156-96e6539c756b-tls-certs\") on node \"ip-10-0-138-97.ec2.internal\" DevicePath \"\""
Jul 07 07:31:41.711148 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:41.710908    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nmq8\" (UniqueName: \"kubernetes.io/projected/ed0fe067-71e2-4583-a156-96e6539c756b-kube-api-access-6nmq8\") on node \"ip-10-0-138-97.ec2.internal\" DevicePath \"\""
Jul 07 07:31:41.711148 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:41.710919    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed0fe067-71e2-4583-a156-96e6539c756b-model-cache\") on node \"ip-10-0-138-97.ec2.internal\" DevicePath \"\""
Jul 07 07:31:41.711148 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:41.710927    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed0fe067-71e2-4583-a156-96e6539c756b-tmp-dir\") on node \"ip-10-0-138-97.ec2.internal\" DevicePath \"\""
Jul 07 07:31:42.408153 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:42.408122    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e309b3/test-model-e309b3-kserve-6474954cb5-ff6wz"
Jul 07 07:31:42.408639 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:42.408172    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e309b3/test-model-e309b3-kserve-6474954cb5-ff6wz" event={"ID":"ed0fe067-71e2-4583-a156-96e6539c756b","Type":"ContainerDied","Data":"bd462d25cd8b38cb4215a1ff1304a1ec1a76ff3ce51fe42b288890267df1128e"}
Jul 07 07:31:42.408639 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:42.408223    2579 scope.go:117] "RemoveContainer" containerID="3bc69adb986c611b059fe124750b57b235452c8147237e1235ee4cebaa89df6a"
Jul 07 07:31:42.438236 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:42.438210    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-e309b3/test-model-e309b3-kserve-6474954cb5-ff6wz"]
Jul 07 07:31:42.457216 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:42.457191    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-e309b3/test-model-e309b3-kserve-6474954cb5-ff6wz"]
Jul 07 07:31:42.964494 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:42.964462    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5fe198f6-279f-4436-b07b-d4cd0e149bcb" path="/var/lib/kubelet/pods/5fe198f6-279f-4436-b07b-d4cd0e149bcb/volumes"
Jul 07 07:31:42.964850 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:31:42.964837    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed0fe067-71e2-4583-a156-96e6539c756b" path="/var/lib/kubelet/pods/ed0fe067-71e2-4583-a156-96e6539c756b/volumes"
Jul 07 07:33:20.608760 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:33:20.608728    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-083d4e/auth-test-model-083d4e-kserve-745957889f-d8594"]
Jul 07 07:33:20.609268 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:33:20.609070    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fe198f6-279f-4436-b07b-d4cd0e149bcb" containerName="main"
Jul 07 07:33:20.609268 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:33:20.609082    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fe198f6-279f-4436-b07b-d4cd0e149bcb" containerName="main"
Jul 07 07:33:20.609268 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:33:20.609096    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed0fe067-71e2-4583-a156-96e6539c756b" containerName="main"
Jul 07 07:33:20.609268 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:33:20.609102    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed0fe067-71e2-4583-a156-96e6539c756b" containerName="main"
Jul 07 07:33:20.609268 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:33:20.609160    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fe198f6-279f-4436-b07b-d4cd0e149bcb" containerName="main"
Jul 07 07:33:20.609268 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:33:20.609168    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed0fe067-71e2-4583-a156-96e6539c756b" containerName="main"
Jul 07 07:33:20.612057 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:33:20.612040    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-083d4e/auth-test-model-083d4e-kserve-745957889f-d8594"
Jul 07 07:33:20.614891 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:33:20.614865    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-083d4e\"/\"kube-root-ca.crt\""
Jul 07 07:33:20.615073 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:33:20.615051    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-083d4e\"/\"openshift-service-ca.crt\""
Jul 07 07:33:20.616318 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:33:20.616304    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-083d4e\"/\"default-dockercfg-7l97m\""
Jul 07 07:33:20.616368 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:33:20.616304    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-083d4e\"/\"auth-test-model-083d4e-kserve-self-signed-certs\""
Jul 07 07:33:20.624396 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:33:20.624376    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-083d4e/auth-test-model-083d4e-kserve-745957889f-d8594"]
Jul 07 07:33:20.764012 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:33:20.763945    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/174f3bf9-bcd3-470b-aa66-1990c379cda7-dshm\") pod \"auth-test-model-083d4e-kserve-745957889f-d8594\" (UID: \"174f3bf9-bcd3-470b-aa66-1990c379cda7\") " pod="ai-tenant-e2e-shared-a-083d4e/auth-test-model-083d4e-kserve-745957889f-d8594"
Jul 07 07:33:20.764188 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:33:20.764030    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/174f3bf9-bcd3-470b-aa66-1990c379cda7-model-cache\") pod \"auth-test-model-083d4e-kserve-745957889f-d8594\" (UID: \"174f3bf9-bcd3-470b-aa66-1990c379cda7\") " pod="ai-tenant-e2e-shared-a-083d4e/auth-test-model-083d4e-kserve-745957889f-d8594"
Jul 07 07:33:20.764188 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:33:20.764058    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/174f3bf9-bcd3-470b-aa66-1990c379cda7-tls-certs\") pod \"auth-test-model-083d4e-kserve-745957889f-d8594\" (UID: \"174f3bf9-bcd3-470b-aa66-1990c379cda7\") " pod="ai-tenant-e2e-shared-a-083d4e/auth-test-model-083d4e-kserve-745957889f-d8594"
Jul 07 07:33:20.764188 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:33:20.764139    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/174f3bf9-bcd3-470b-aa66-1990c379cda7-home\") pod \"auth-test-model-083d4e-kserve-745957889f-d8594\" (UID: \"174f3bf9-bcd3-470b-aa66-1990c379cda7\") " pod="ai-tenant-e2e-shared-a-083d4e/auth-test-model-083d4e-kserve-745957889f-d8594"
Jul 07 07:33:20.764188 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:33:20.764173    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/174f3bf9-bcd3-470b-aa66-1990c379cda7-tmp-dir\") pod \"auth-test-model-083d4e-kserve-745957889f-d8594\" (UID: \"174f3bf9-bcd3-470b-aa66-1990c379cda7\") " pod="ai-tenant-e2e-shared-a-083d4e/auth-test-model-083d4e-kserve-745957889f-d8594"
Jul 07 07:33:20.764349 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:33:20.764193    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sp6n\" (UniqueName: \"kubernetes.io/projected/174f3bf9-bcd3-470b-aa66-1990c379cda7-kube-api-access-7sp6n\") pod \"auth-test-model-083d4e-kserve-745957889f-d8594\" (UID: \"174f3bf9-bcd3-470b-aa66-1990c379cda7\") " pod="ai-tenant-e2e-shared-a-083d4e/auth-test-model-083d4e-kserve-745957889f-d8594"
Jul 07 07:33:20.865428 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:33:20.865341    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/174f3bf9-bcd3-470b-aa66-1990c379cda7-dshm\") pod \"auth-test-model-083d4e-kserve-745957889f-d8594\" (UID: \"174f3bf9-bcd3-470b-aa66-1990c379cda7\") " pod="ai-tenant-e2e-shared-a-083d4e/auth-test-model-083d4e-kserve-745957889f-d8594"
Jul 07 07:33:20.865428 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:33:20.865383    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/174f3bf9-bcd3-470b-aa66-1990c379cda7-model-cache\") pod \"auth-test-model-083d4e-kserve-745957889f-d8594\" (UID: \"174f3bf9-bcd3-470b-aa66-1990c379cda7\") " pod="ai-tenant-e2e-shared-a-083d4e/auth-test-model-083d4e-kserve-745957889f-d8594"
Jul 07 07:33:20.865428 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:33:20.865402    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/174f3bf9-bcd3-470b-aa66-1990c379cda7-tls-certs\") pod \"auth-test-model-083d4e-kserve-745957889f-d8594\" (UID: \"174f3bf9-bcd3-470b-aa66-1990c379cda7\") " pod="ai-tenant-e2e-shared-a-083d4e/auth-test-model-083d4e-kserve-745957889f-d8594"
Jul 07 07:33:20.865715 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:33:20.865531    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/174f3bf9-bcd3-470b-aa66-1990c379cda7-home\") pod \"auth-test-model-083d4e-kserve-745957889f-d8594\" (UID: \"174f3bf9-bcd3-470b-aa66-1990c379cda7\") " pod="ai-tenant-e2e-shared-a-083d4e/auth-test-model-083d4e-kserve-745957889f-d8594"
Jul 07 07:33:20.865715 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:33:20.865571    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/174f3bf9-bcd3-470b-aa66-1990c379cda7-tmp-dir\") pod \"auth-test-model-083d4e-kserve-745957889f-d8594\" (UID: \"174f3bf9-bcd3-470b-aa66-1990c379cda7\") " pod="ai-tenant-e2e-shared-a-083d4e/auth-test-model-083d4e-kserve-745957889f-d8594"
Jul 07 07:33:20.865715 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:33:20.865598    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sp6n\" (UniqueName: \"kubernetes.io/projected/174f3bf9-bcd3-470b-aa66-1990c379cda7-kube-api-access-7sp6n\") pod \"auth-test-model-083d4e-kserve-745957889f-d8594\" (UID: \"174f3bf9-bcd3-470b-aa66-1990c379cda7\") " pod="ai-tenant-e2e-shared-a-083d4e/auth-test-model-083d4e-kserve-745957889f-d8594"
Jul 07 07:33:20.865880 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:33:20.865851    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/174f3bf9-bcd3-470b-aa66-1990c379cda7-model-cache\") pod \"auth-test-model-083d4e-kserve-745957889f-d8594\" (UID: \"174f3bf9-bcd3-470b-aa66-1990c379cda7\") " pod="ai-tenant-e2e-shared-a-083d4e/auth-test-model-083d4e-kserve-745957889f-d8594"
Jul 07 07:33:20.865931 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:33:20.865906    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/174f3bf9-bcd3-470b-aa66-1990c379cda7-tmp-dir\") pod \"auth-test-model-083d4e-kserve-745957889f-d8594\" (UID: \"174f3bf9-bcd3-470b-aa66-1990c379cda7\") " pod="ai-tenant-e2e-shared-a-083d4e/auth-test-model-083d4e-kserve-745957889f-d8594"
Jul 07 07:33:20.866016 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:33:20.865933    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/174f3bf9-bcd3-470b-aa66-1990c379cda7-home\") pod \"auth-test-model-083d4e-kserve-745957889f-d8594\" (UID: \"174f3bf9-bcd3-470b-aa66-1990c379cda7\") " pod="ai-tenant-e2e-shared-a-083d4e/auth-test-model-083d4e-kserve-745957889f-d8594"
Jul 07 07:33:20.867764 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:33:20.867743    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/174f3bf9-bcd3-470b-aa66-1990c379cda7-dshm\") pod \"auth-test-model-083d4e-kserve-745957889f-d8594\" (UID: \"174f3bf9-bcd3-470b-aa66-1990c379cda7\") " pod="ai-tenant-e2e-shared-a-083d4e/auth-test-model-083d4e-kserve-745957889f-d8594"
Jul 07 07:33:20.868031 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:33:20.868012    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/174f3bf9-bcd3-470b-aa66-1990c379cda7-tls-certs\") pod \"auth-test-model-083d4e-kserve-745957889f-d8594\" (UID: \"174f3bf9-bcd3-470b-aa66-1990c379cda7\") " pod="ai-tenant-e2e-shared-a-083d4e/auth-test-model-083d4e-kserve-745957889f-d8594"
Jul 07 07:33:20.881338 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:33:20.881315    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sp6n\" (UniqueName: \"kubernetes.io/projected/174f3bf9-bcd3-470b-aa66-1990c379cda7-kube-api-access-7sp6n\") pod \"auth-test-model-083d4e-kserve-745957889f-d8594\" (UID: \"174f3bf9-bcd3-470b-aa66-1990c379cda7\") " pod="ai-tenant-e2e-shared-a-083d4e/auth-test-model-083d4e-kserve-745957889f-d8594"
Jul 07 07:33:20.921328 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:33:20.921295    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-083d4e/auth-test-model-083d4e-kserve-745957889f-d8594"
Jul 07 07:33:21.054080 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:33:21.054055    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-083d4e/auth-test-model-083d4e-kserve-745957889f-d8594"]
Jul 07 07:33:21.056308 ip-10-0-138-97 kubenswrapper[2579]: W0707 07:33:21.056282    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod174f3bf9_bcd3_470b_aa66_1990c379cda7.slice/crio-9c31a4daff11e2f42990caf6d42d9f8081aebece9e8fb8b2514448aa7a820305 WatchSource:0}: Error finding container 9c31a4daff11e2f42990caf6d42d9f8081aebece9e8fb8b2514448aa7a820305: Status 404 returned error can't find the container with id 9c31a4daff11e2f42990caf6d42d9f8081aebece9e8fb8b2514448aa7a820305
Jul 07 07:33:21.853118 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:33:21.853083    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-083d4e/auth-test-model-083d4e-kserve-745957889f-d8594" event={"ID":"174f3bf9-bcd3-470b-aa66-1990c379cda7","Type":"ContainerStarted","Data":"ed6dd3ace3981ef4c643c9ccfac3361d67d6245067160cbfdac76ad0a35722ad"}
Jul 07 07:33:21.853118 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:33:21.853123    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-083d4e/auth-test-model-083d4e-kserve-745957889f-d8594" event={"ID":"174f3bf9-bcd3-470b-aa66-1990c379cda7","Type":"ContainerStarted","Data":"9c31a4daff11e2f42990caf6d42d9f8081aebece9e8fb8b2514448aa7a820305"}
Jul 07 07:33:21.874091 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:33:21.874044    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-083d4e/auth-test-model-083d4e-kserve-745957889f-d8594" podStartSLOduration=1.874028811 podStartE2EDuration="1.874028811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 07:33:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 07:33:21.872002071 +0000 UTC m=+2081.415763552" watchObservedRunningTime="2026-07-07 07:33:21.874028811 +0000 UTC m=+2081.417790295"
Jul 07 07:33:30.922220 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:33:30.922184    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-083d4e/auth-test-model-083d4e-kserve-745957889f-d8594"
Jul 07 07:33:30.922220 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:33:30.922228    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-083d4e/auth-test-model-083d4e-kserve-745957889f-d8594"
Jul 07 07:33:30.934677 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:33:30.934657    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-083d4e/auth-test-model-083d4e-kserve-745957889f-d8594"
Jul 07 07:33:31.907410 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:33:31.907381    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-083d4e/auth-test-model-083d4e-kserve-745957889f-d8594"
Jul 07 07:33:43.298910 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:33:43.298872    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-1082ad/auth-test-model-1082ad-kserve-5c68db54dc-v5wwj"]
Jul 07 07:33:43.303037 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:33:43.303012    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1082ad/auth-test-model-1082ad-kserve-5c68db54dc-v5wwj"
Jul 07 07:33:43.307454 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:33:43.307402    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-1082ad\"/\"kube-root-ca.crt\""
Jul 07 07:33:43.307595 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:33:43.307455    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-1082ad\"/\"auth-test-model-1082ad-kserve-self-signed-certs\""
Jul 07 07:33:43.307595 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:33:43.307404    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-1082ad\"/\"openshift-service-ca.crt\""
Jul 07 07:33:43.307595 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:33:43.307408    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-1082ad\"/\"default-dockercfg-5xj7h\""
Jul 07 07:33:43.314365 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:33:43.314342    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-1082ad/auth-test-model-1082ad-kserve-5c68db54dc-v5wwj"]
Jul 07 07:33:43.465696 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:33:43.465664    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1803965-fbb3-401f-b2c3-a09205e9564d-tmp-dir\") pod \"auth-test-model-1082ad-kserve-5c68db54dc-v5wwj\" (UID: \"e1803965-fbb3-401f-b2c3-a09205e9564d\") " pod="ai-tenant-e2e-shared-b-1082ad/auth-test-model-1082ad-kserve-5c68db54dc-v5wwj"
Jul 07 07:33:43.465858 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:33:43.465712    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1803965-fbb3-401f-b2c3-a09205e9564d-model-cache\") pod \"auth-test-model-1082ad-kserve-5c68db54dc-v5wwj\" (UID: \"e1803965-fbb3-401f-b2c3-a09205e9564d\") " pod="ai-tenant-e2e-shared-b-1082ad/auth-test-model-1082ad-kserve-5c68db54dc-v5wwj"
Jul 07 07:33:43.465858 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:33:43.465733    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1803965-fbb3-401f-b2c3-a09205e9564d-tls-certs\") pod \"auth-test-model-1082ad-kserve-5c68db54dc-v5wwj\" (UID: \"e1803965-fbb3-401f-b2c3-a09205e9564d\") " pod="ai-tenant-e2e-shared-b-1082ad/auth-test-model-1082ad-kserve-5c68db54dc-v5wwj"
Jul 07 07:33:43.465858 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:33:43.465825    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1803965-fbb3-401f-b2c3-a09205e9564d-home\") pod \"auth-test-model-1082ad-kserve-5c68db54dc-v5wwj\" (UID: \"e1803965-fbb3-401f-b2c3-a09205e9564d\") " pod="ai-tenant-e2e-shared-b-1082ad/auth-test-model-1082ad-kserve-5c68db54dc-v5wwj"
Jul 07 07:33:43.465858 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:33:43.465854    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd9d2\" (UniqueName: \"kubernetes.io/projected/e1803965-fbb3-401f-b2c3-a09205e9564d-kube-api-access-gd9d2\") pod \"auth-test-model-1082ad-kserve-5c68db54dc-v5wwj\" (UID: \"e1803965-fbb3-401f-b2c3-a09205e9564d\") " pod="ai-tenant-e2e-shared-b-1082ad/auth-test-model-1082ad-kserve-5c68db54dc-v5wwj"
Jul 07 07:33:43.466015 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:33:43.465891    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1803965-fbb3-401f-b2c3-a09205e9564d-dshm\") pod \"auth-test-model-1082ad-kserve-5c68db54dc-v5wwj\" (UID: \"e1803965-fbb3-401f-b2c3-a09205e9564d\") " pod="ai-tenant-e2e-shared-b-1082ad/auth-test-model-1082ad-kserve-5c68db54dc-v5wwj"
Jul 07 07:33:43.567166 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:33:43.567083    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1803965-fbb3-401f-b2c3-a09205e9564d-dshm\") pod \"auth-test-model-1082ad-kserve-5c68db54dc-v5wwj\" (UID: \"e1803965-fbb3-401f-b2c3-a09205e9564d\") " pod="ai-tenant-e2e-shared-b-1082ad/auth-test-model-1082ad-kserve-5c68db54dc-v5wwj"
Jul 07 07:33:43.567166 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:33:43.567123    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1803965-fbb3-401f-b2c3-a09205e9564d-tmp-dir\") pod \"auth-test-model-1082ad-kserve-5c68db54dc-v5wwj\" (UID: \"e1803965-fbb3-401f-b2c3-a09205e9564d\") " pod="ai-tenant-e2e-shared-b-1082ad/auth-test-model-1082ad-kserve-5c68db54dc-v5wwj"
Jul 07 07:33:43.567166 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:33:43.567155    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1803965-fbb3-401f-b2c3-a09205e9564d-model-cache\") pod \"auth-test-model-1082ad-kserve-5c68db54dc-v5wwj\" (UID: \"e1803965-fbb3-401f-b2c3-a09205e9564d\") " pod="ai-tenant-e2e-shared-b-1082ad/auth-test-model-1082ad-kserve-5c68db54dc-v5wwj"
Jul 07 07:33:43.567520 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:33:43.567270    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1803965-fbb3-401f-b2c3-a09205e9564d-tls-certs\") pod \"auth-test-model-1082ad-kserve-5c68db54dc-v5wwj\" (UID: \"e1803965-fbb3-401f-b2c3-a09205e9564d\") " pod="ai-tenant-e2e-shared-b-1082ad/auth-test-model-1082ad-kserve-5c68db54dc-v5wwj"
Jul 07 07:33:43.567520 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:33:43.567366    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1803965-fbb3-401f-b2c3-a09205e9564d-home\") pod \"auth-test-model-1082ad-kserve-5c68db54dc-v5wwj\" (UID: \"e1803965-fbb3-401f-b2c3-a09205e9564d\") " pod="ai-tenant-e2e-shared-b-1082ad/auth-test-model-1082ad-kserve-5c68db54dc-v5wwj"
Jul 07 07:33:43.567520 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:33:43.567402    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd9d2\" (UniqueName: \"kubernetes.io/projected/e1803965-fbb3-401f-b2c3-a09205e9564d-kube-api-access-gd9d2\") pod \"auth-test-model-1082ad-kserve-5c68db54dc-v5wwj\" (UID: \"e1803965-fbb3-401f-b2c3-a09205e9564d\") " pod="ai-tenant-e2e-shared-b-1082ad/auth-test-model-1082ad-kserve-5c68db54dc-v5wwj"
Jul 07 07:33:43.567520 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:33:43.567488    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1803965-fbb3-401f-b2c3-a09205e9564d-model-cache\") pod \"auth-test-model-1082ad-kserve-5c68db54dc-v5wwj\" (UID: \"e1803965-fbb3-401f-b2c3-a09205e9564d\") " pod="ai-tenant-e2e-shared-b-1082ad/auth-test-model-1082ad-kserve-5c68db54dc-v5wwj"
Jul 07 07:33:43.567721 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:33:43.567591    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1803965-fbb3-401f-b2c3-a09205e9564d-tmp-dir\") pod \"auth-test-model-1082ad-kserve-5c68db54dc-v5wwj\" (UID: \"e1803965-fbb3-401f-b2c3-a09205e9564d\") " pod="ai-tenant-e2e-shared-b-1082ad/auth-test-model-1082ad-kserve-5c68db54dc-v5wwj"
Jul 07 07:33:43.567760 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:33:43.567713    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1803965-fbb3-401f-b2c3-a09205e9564d-home\") pod \"auth-test-model-1082ad-kserve-5c68db54dc-v5wwj\" (UID: \"e1803965-fbb3-401f-b2c3-a09205e9564d\") " pod="ai-tenant-e2e-shared-b-1082ad/auth-test-model-1082ad-kserve-5c68db54dc-v5wwj"
Jul 07 07:33:43.569453 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:33:43.569417    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1803965-fbb3-401f-b2c3-a09205e9564d-dshm\") pod \"auth-test-model-1082ad-kserve-5c68db54dc-v5wwj\" (UID: \"e1803965-fbb3-401f-b2c3-a09205e9564d\") " pod="ai-tenant-e2e-shared-b-1082ad/auth-test-model-1082ad-kserve-5c68db54dc-v5wwj"
Jul 07 07:33:43.569677 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:33:43.569662    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1803965-fbb3-401f-b2c3-a09205e9564d-tls-certs\") pod \"auth-test-model-1082ad-kserve-5c68db54dc-v5wwj\" (UID: \"e1803965-fbb3-401f-b2c3-a09205e9564d\") " pod="ai-tenant-e2e-shared-b-1082ad/auth-test-model-1082ad-kserve-5c68db54dc-v5wwj"
Jul 07 07:33:43.577589 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:33:43.577568    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd9d2\" (UniqueName: \"kubernetes.io/projected/e1803965-fbb3-401f-b2c3-a09205e9564d-kube-api-access-gd9d2\") pod \"auth-test-model-1082ad-kserve-5c68db54dc-v5wwj\" (UID: \"e1803965-fbb3-401f-b2c3-a09205e9564d\") " pod="ai-tenant-e2e-shared-b-1082ad/auth-test-model-1082ad-kserve-5c68db54dc-v5wwj"
Jul 07 07:33:43.613451 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:33:43.613424    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1082ad/auth-test-model-1082ad-kserve-5c68db54dc-v5wwj"
Jul 07 07:33:43.748095 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:33:43.748068    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-1082ad/auth-test-model-1082ad-kserve-5c68db54dc-v5wwj"]
Jul 07 07:33:43.750437 ip-10-0-138-97 kubenswrapper[2579]: W0707 07:33:43.750408    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode1803965_fbb3_401f_b2c3_a09205e9564d.slice/crio-84bc025c48e043690e30e178dfbbf8b2aa897d56d9958cd1a6fecc28395538c3 WatchSource:0}: Error finding container 84bc025c48e043690e30e178dfbbf8b2aa897d56d9958cd1a6fecc28395538c3: Status 404 returned error can't find the container with id 84bc025c48e043690e30e178dfbbf8b2aa897d56d9958cd1a6fecc28395538c3
Jul 07 07:33:43.950913 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:33:43.950822    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1082ad/auth-test-model-1082ad-kserve-5c68db54dc-v5wwj" event={"ID":"e1803965-fbb3-401f-b2c3-a09205e9564d","Type":"ContainerStarted","Data":"a35e2f0b2e743e7956ec18a810d8a076a64c34a2fed2398cf45c744008b89c50"}
Jul 07 07:33:43.950913 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:33:43.950876    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1082ad/auth-test-model-1082ad-kserve-5c68db54dc-v5wwj" event={"ID":"e1803965-fbb3-401f-b2c3-a09205e9564d","Type":"ContainerStarted","Data":"84bc025c48e043690e30e178dfbbf8b2aa897d56d9958cd1a6fecc28395538c3"}
Jul 07 07:33:43.971607 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:33:43.971559    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-1082ad/auth-test-model-1082ad-kserve-5c68db54dc-v5wwj" podStartSLOduration=0.971544174 podStartE2EDuration="971.544174ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 07:33:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 07:33:43.969989374 +0000 UTC m=+2103.513750854" watchObservedRunningTime="2026-07-07 07:33:43.971544174 +0000 UTC m=+2103.515305714"
Jul 07 07:33:53.614376 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:33:53.614325    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-1082ad/auth-test-model-1082ad-kserve-5c68db54dc-v5wwj"
Jul 07 07:33:53.614376 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:33:53.614386    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-1082ad/auth-test-model-1082ad-kserve-5c68db54dc-v5wwj"
Jul 07 07:33:53.627090 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:33:53.627055    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-1082ad/auth-test-model-1082ad-kserve-5c68db54dc-v5wwj"
Jul 07 07:33:54.006663 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:33:54.006630    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-1082ad/auth-test-model-1082ad-kserve-5c68db54dc-v5wwj"
Jul 07 07:35:35.925495 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:35.925454    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-083d4e/auth-test-model-083d4e-kserve-745957889f-d8594"]
Jul 07 07:35:36.877096 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:36.877062    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-1082ad/auth-test-model-1082ad-kserve-5c68db54dc-v5wwj"]
Jul 07 07:35:50.945516 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:50.945455    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-083d4e/auth-test-model-083d4e-kserve-745957889f-d8594" podUID="174f3bf9-bcd3-470b-aa66-1990c379cda7" containerName="main" containerID="cri-o://ed6dd3ace3981ef4c643c9ccfac3361d67d6245067160cbfdac76ad0a35722ad" gracePeriod=45
Jul 07 07:35:51.184736 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:51.184708    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-083d4e/auth-test-model-083d4e-kserve-745957889f-d8594"
Jul 07 07:35:51.322640 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:51.322597    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/174f3bf9-bcd3-470b-aa66-1990c379cda7-model-cache\") pod \"174f3bf9-bcd3-470b-aa66-1990c379cda7\" (UID: \"174f3bf9-bcd3-470b-aa66-1990c379cda7\") "
Jul 07 07:35:51.322640 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:51.322649    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/174f3bf9-bcd3-470b-aa66-1990c379cda7-tls-certs\") pod \"174f3bf9-bcd3-470b-aa66-1990c379cda7\" (UID: \"174f3bf9-bcd3-470b-aa66-1990c379cda7\") "
Jul 07 07:35:51.322878 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:51.322673    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/174f3bf9-bcd3-470b-aa66-1990c379cda7-dshm\") pod \"174f3bf9-bcd3-470b-aa66-1990c379cda7\" (UID: \"174f3bf9-bcd3-470b-aa66-1990c379cda7\") "
Jul 07 07:35:51.322878 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:51.322715    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7sp6n\" (UniqueName: \"kubernetes.io/projected/174f3bf9-bcd3-470b-aa66-1990c379cda7-kube-api-access-7sp6n\") pod \"174f3bf9-bcd3-470b-aa66-1990c379cda7\" (UID: \"174f3bf9-bcd3-470b-aa66-1990c379cda7\") "
Jul 07 07:35:51.322878 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:51.322746    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/174f3bf9-bcd3-470b-aa66-1990c379cda7-home\") pod \"174f3bf9-bcd3-470b-aa66-1990c379cda7\" (UID: \"174f3bf9-bcd3-470b-aa66-1990c379cda7\") "
Jul 07 07:35:51.322878 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:51.322773    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/174f3bf9-bcd3-470b-aa66-1990c379cda7-tmp-dir\") pod \"174f3bf9-bcd3-470b-aa66-1990c379cda7\" (UID: \"174f3bf9-bcd3-470b-aa66-1990c379cda7\") "
Jul 07 07:35:51.323117 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:51.322982    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/174f3bf9-bcd3-470b-aa66-1990c379cda7-model-cache" (OuterVolumeSpecName: "model-cache") pod "174f3bf9-bcd3-470b-aa66-1990c379cda7" (UID: "174f3bf9-bcd3-470b-aa66-1990c379cda7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 07:35:51.323117 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:51.323082    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/174f3bf9-bcd3-470b-aa66-1990c379cda7-home" (OuterVolumeSpecName: "home") pod "174f3bf9-bcd3-470b-aa66-1990c379cda7" (UID: "174f3bf9-bcd3-470b-aa66-1990c379cda7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 07:35:51.323222 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:51.323175    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/174f3bf9-bcd3-470b-aa66-1990c379cda7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "174f3bf9-bcd3-470b-aa66-1990c379cda7" (UID: "174f3bf9-bcd3-470b-aa66-1990c379cda7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 07:35:51.324986 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:51.324942    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/174f3bf9-bcd3-470b-aa66-1990c379cda7-dshm" (OuterVolumeSpecName: "dshm") pod "174f3bf9-bcd3-470b-aa66-1990c379cda7" (UID: "174f3bf9-bcd3-470b-aa66-1990c379cda7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 07:35:51.325081 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:51.325023    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/174f3bf9-bcd3-470b-aa66-1990c379cda7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "174f3bf9-bcd3-470b-aa66-1990c379cda7" (UID: "174f3bf9-bcd3-470b-aa66-1990c379cda7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 07:35:51.325081 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:51.325061    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/174f3bf9-bcd3-470b-aa66-1990c379cda7-kube-api-access-7sp6n" (OuterVolumeSpecName: "kube-api-access-7sp6n") pod "174f3bf9-bcd3-470b-aa66-1990c379cda7" (UID: "174f3bf9-bcd3-470b-aa66-1990c379cda7"). InnerVolumeSpecName "kube-api-access-7sp6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 07:35:51.423761 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:51.423714    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/174f3bf9-bcd3-470b-aa66-1990c379cda7-home\") on node \"ip-10-0-138-97.ec2.internal\" DevicePath \"\""
Jul 07 07:35:51.423761 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:51.423752    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/174f3bf9-bcd3-470b-aa66-1990c379cda7-tmp-dir\") on node \"ip-10-0-138-97.ec2.internal\" DevicePath \"\""
Jul 07 07:35:51.423761 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:51.423761    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/174f3bf9-bcd3-470b-aa66-1990c379cda7-model-cache\") on node \"ip-10-0-138-97.ec2.internal\" DevicePath \"\""
Jul 07 07:35:51.423761 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:51.423771    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/174f3bf9-bcd3-470b-aa66-1990c379cda7-tls-certs\") on node \"ip-10-0-138-97.ec2.internal\" DevicePath \"\""
Jul 07 07:35:51.424070 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:51.423779    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/174f3bf9-bcd3-470b-aa66-1990c379cda7-dshm\") on node \"ip-10-0-138-97.ec2.internal\" DevicePath \"\""
Jul 07 07:35:51.424070 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:51.423788    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sp6n\" (UniqueName: \"kubernetes.io/projected/174f3bf9-bcd3-470b-aa66-1990c379cda7-kube-api-access-7sp6n\") on node \"ip-10-0-138-97.ec2.internal\" DevicePath \"\""
Jul 07 07:35:51.514211 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:51.514170    2579 generic.go:356] "Generic (PLEG): container finished" podID="174f3bf9-bcd3-470b-aa66-1990c379cda7" containerID="ed6dd3ace3981ef4c643c9ccfac3361d67d6245067160cbfdac76ad0a35722ad" exitCode=0
Jul 07 07:35:51.514211 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:51.514212    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-083d4e/auth-test-model-083d4e-kserve-745957889f-d8594"
Jul 07 07:35:51.514477 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:51.514243    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-083d4e/auth-test-model-083d4e-kserve-745957889f-d8594" event={"ID":"174f3bf9-bcd3-470b-aa66-1990c379cda7","Type":"ContainerDied","Data":"ed6dd3ace3981ef4c643c9ccfac3361d67d6245067160cbfdac76ad0a35722ad"}
Jul 07 07:35:51.514477 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:51.514291    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-083d4e/auth-test-model-083d4e-kserve-745957889f-d8594" event={"ID":"174f3bf9-bcd3-470b-aa66-1990c379cda7","Type":"ContainerDied","Data":"9c31a4daff11e2f42990caf6d42d9f8081aebece9e8fb8b2514448aa7a820305"}
Jul 07 07:35:51.514477 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:51.514313    2579 scope.go:117] "RemoveContainer" containerID="ed6dd3ace3981ef4c643c9ccfac3361d67d6245067160cbfdac76ad0a35722ad"
Jul 07 07:35:51.523874 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:51.523848    2579 scope.go:117] "RemoveContainer" containerID="ed6dd3ace3981ef4c643c9ccfac3361d67d6245067160cbfdac76ad0a35722ad"
Jul 07 07:35:51.524211 ip-10-0-138-97 kubenswrapper[2579]: E0707 07:35:51.524191    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed6dd3ace3981ef4c643c9ccfac3361d67d6245067160cbfdac76ad0a35722ad\": container with ID starting with ed6dd3ace3981ef4c643c9ccfac3361d67d6245067160cbfdac76ad0a35722ad not found: ID does not exist" containerID="ed6dd3ace3981ef4c643c9ccfac3361d67d6245067160cbfdac76ad0a35722ad"
Jul 07 07:35:51.524282 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:51.524222    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed6dd3ace3981ef4c643c9ccfac3361d67d6245067160cbfdac76ad0a35722ad"} err="failed to get container status \"ed6dd3ace3981ef4c643c9ccfac3361d67d6245067160cbfdac76ad0a35722ad\": rpc error: code = NotFound desc = could not find container \"ed6dd3ace3981ef4c643c9ccfac3361d67d6245067160cbfdac76ad0a35722ad\": container with ID starting with ed6dd3ace3981ef4c643c9ccfac3361d67d6245067160cbfdac76ad0a35722ad not found: ID does not exist"
Jul 07 07:35:51.544006 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:51.543947    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-083d4e/auth-test-model-083d4e-kserve-745957889f-d8594"]
Jul 07 07:35:51.550036 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:51.550003    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-083d4e/auth-test-model-083d4e-kserve-745957889f-d8594"]
Jul 07 07:35:51.897273 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:51.897190    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-1082ad/auth-test-model-1082ad-kserve-5c68db54dc-v5wwj" podUID="e1803965-fbb3-401f-b2c3-a09205e9564d" containerName="main" containerID="cri-o://a35e2f0b2e743e7956ec18a810d8a076a64c34a2fed2398cf45c744008b89c50" gracePeriod=45
Jul 07 07:35:52.141117 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:52.141090    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1082ad/auth-test-model-1082ad-kserve-5c68db54dc-v5wwj"
Jul 07 07:35:52.230260 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:52.230225    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1803965-fbb3-401f-b2c3-a09205e9564d-tls-certs\") pod \"e1803965-fbb3-401f-b2c3-a09205e9564d\" (UID: \"e1803965-fbb3-401f-b2c3-a09205e9564d\") "
Jul 07 07:35:52.230447 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:52.230286    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1803965-fbb3-401f-b2c3-a09205e9564d-dshm\") pod \"e1803965-fbb3-401f-b2c3-a09205e9564d\" (UID: \"e1803965-fbb3-401f-b2c3-a09205e9564d\") "
Jul 07 07:35:52.230447 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:52.230315    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1803965-fbb3-401f-b2c3-a09205e9564d-tmp-dir\") pod \"e1803965-fbb3-401f-b2c3-a09205e9564d\" (UID: \"e1803965-fbb3-401f-b2c3-a09205e9564d\") "
Jul 07 07:35:52.230447 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:52.230340    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1803965-fbb3-401f-b2c3-a09205e9564d-home\") pod \"e1803965-fbb3-401f-b2c3-a09205e9564d\" (UID: \"e1803965-fbb3-401f-b2c3-a09205e9564d\") "
Jul 07 07:35:52.230447 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:52.230365    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gd9d2\" (UniqueName: \"kubernetes.io/projected/e1803965-fbb3-401f-b2c3-a09205e9564d-kube-api-access-gd9d2\") pod \"e1803965-fbb3-401f-b2c3-a09205e9564d\" (UID: \"e1803965-fbb3-401f-b2c3-a09205e9564d\") "
Jul 07 07:35:52.230447 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:52.230402    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1803965-fbb3-401f-b2c3-a09205e9564d-model-cache\") pod \"e1803965-fbb3-401f-b2c3-a09205e9564d\" (UID: \"e1803965-fbb3-401f-b2c3-a09205e9564d\") "
Jul 07 07:35:52.230688 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:52.230630    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1803965-fbb3-401f-b2c3-a09205e9564d-home" (OuterVolumeSpecName: "home") pod "e1803965-fbb3-401f-b2c3-a09205e9564d" (UID: "e1803965-fbb3-401f-b2c3-a09205e9564d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 07:35:52.230688 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:52.230640    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1803965-fbb3-401f-b2c3-a09205e9564d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e1803965-fbb3-401f-b2c3-a09205e9564d" (UID: "e1803965-fbb3-401f-b2c3-a09205e9564d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 07:35:52.230812 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:52.230793    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1803965-fbb3-401f-b2c3-a09205e9564d-tmp-dir\") on node \"ip-10-0-138-97.ec2.internal\" DevicePath \"\""
Jul 07 07:35:52.230870 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:52.230824    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1803965-fbb3-401f-b2c3-a09205e9564d-home\") on node \"ip-10-0-138-97.ec2.internal\" DevicePath \"\""
Jul 07 07:35:52.230870 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:52.230798    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1803965-fbb3-401f-b2c3-a09205e9564d-model-cache" (OuterVolumeSpecName: "model-cache") pod "e1803965-fbb3-401f-b2c3-a09205e9564d" (UID: "e1803965-fbb3-401f-b2c3-a09205e9564d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 07:35:52.232658 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:52.232633    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1803965-fbb3-401f-b2c3-a09205e9564d-kube-api-access-gd9d2" (OuterVolumeSpecName: "kube-api-access-gd9d2") pod "e1803965-fbb3-401f-b2c3-a09205e9564d" (UID: "e1803965-fbb3-401f-b2c3-a09205e9564d"). InnerVolumeSpecName "kube-api-access-gd9d2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 07:35:52.232756 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:52.232673    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1803965-fbb3-401f-b2c3-a09205e9564d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e1803965-fbb3-401f-b2c3-a09205e9564d" (UID: "e1803965-fbb3-401f-b2c3-a09205e9564d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 07:35:52.232756 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:52.232673    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1803965-fbb3-401f-b2c3-a09205e9564d-dshm" (OuterVolumeSpecName: "dshm") pod "e1803965-fbb3-401f-b2c3-a09205e9564d" (UID: "e1803965-fbb3-401f-b2c3-a09205e9564d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 07:35:52.331452 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:52.331404    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1803965-fbb3-401f-b2c3-a09205e9564d-tls-certs\") on node \"ip-10-0-138-97.ec2.internal\" DevicePath \"\""
Jul 07 07:35:52.331452 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:52.331440    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1803965-fbb3-401f-b2c3-a09205e9564d-dshm\") on node \"ip-10-0-138-97.ec2.internal\" DevicePath \"\""
Jul 07 07:35:52.331452 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:52.331449    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gd9d2\" (UniqueName: \"kubernetes.io/projected/e1803965-fbb3-401f-b2c3-a09205e9564d-kube-api-access-gd9d2\") on node \"ip-10-0-138-97.ec2.internal\" DevicePath \"\""
Jul 07 07:35:52.331452 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:52.331460    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1803965-fbb3-401f-b2c3-a09205e9564d-model-cache\") on node \"ip-10-0-138-97.ec2.internal\" DevicePath \"\""
Jul 07 07:35:52.520276 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:52.520183    2579 generic.go:356] "Generic (PLEG): container finished" podID="e1803965-fbb3-401f-b2c3-a09205e9564d" containerID="a35e2f0b2e743e7956ec18a810d8a076a64c34a2fed2398cf45c744008b89c50" exitCode=0
Jul 07 07:35:52.520276 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:52.520226    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1082ad/auth-test-model-1082ad-kserve-5c68db54dc-v5wwj"
Jul 07 07:35:52.520502 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:52.520269    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1082ad/auth-test-model-1082ad-kserve-5c68db54dc-v5wwj" event={"ID":"e1803965-fbb3-401f-b2c3-a09205e9564d","Type":"ContainerDied","Data":"a35e2f0b2e743e7956ec18a810d8a076a64c34a2fed2398cf45c744008b89c50"}
Jul 07 07:35:52.520502 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:52.520317    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1082ad/auth-test-model-1082ad-kserve-5c68db54dc-v5wwj" event={"ID":"e1803965-fbb3-401f-b2c3-a09205e9564d","Type":"ContainerDied","Data":"84bc025c48e043690e30e178dfbbf8b2aa897d56d9958cd1a6fecc28395538c3"}
Jul 07 07:35:52.520502 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:52.520341    2579 scope.go:117] "RemoveContainer" containerID="a35e2f0b2e743e7956ec18a810d8a076a64c34a2fed2398cf45c744008b89c50"
Jul 07 07:35:52.531624 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:52.531604    2579 scope.go:117] "RemoveContainer" containerID="a35e2f0b2e743e7956ec18a810d8a076a64c34a2fed2398cf45c744008b89c50"
Jul 07 07:35:52.531888 ip-10-0-138-97 kubenswrapper[2579]: E0707 07:35:52.531868    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a35e2f0b2e743e7956ec18a810d8a076a64c34a2fed2398cf45c744008b89c50\": container with ID starting with a35e2f0b2e743e7956ec18a810d8a076a64c34a2fed2398cf45c744008b89c50 not found: ID does not exist" containerID="a35e2f0b2e743e7956ec18a810d8a076a64c34a2fed2398cf45c744008b89c50"
Jul 07 07:35:52.531932 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:52.531895    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a35e2f0b2e743e7956ec18a810d8a076a64c34a2fed2398cf45c744008b89c50"} err="failed to get container status \"a35e2f0b2e743e7956ec18a810d8a076a64c34a2fed2398cf45c744008b89c50\": rpc error: code = NotFound desc = could not find container \"a35e2f0b2e743e7956ec18a810d8a076a64c34a2fed2398cf45c744008b89c50\": container with ID starting with a35e2f0b2e743e7956ec18a810d8a076a64c34a2fed2398cf45c744008b89c50 not found: ID does not exist"
Jul 07 07:35:52.546646 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:52.546618    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-1082ad/auth-test-model-1082ad-kserve-5c68db54dc-v5wwj"]
Jul 07 07:35:52.550834 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:52.550810    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-1082ad/auth-test-model-1082ad-kserve-5c68db54dc-v5wwj"]
Jul 07 07:35:52.963846 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:52.963764    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="174f3bf9-bcd3-470b-aa66-1990c379cda7" path="/var/lib/kubelet/pods/174f3bf9-bcd3-470b-aa66-1990c379cda7/volumes"
Jul 07 07:35:52.964193 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:52.964180    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1803965-fbb3-401f-b2c3-a09205e9564d" path="/var/lib/kubelet/pods/e1803965-fbb3-401f-b2c3-a09205e9564d/volumes"
Jul 07 07:35:53.312972 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:53.312916    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-083d4e/sub-test-model-083d4e-kserve-8bf6b6cd7-7lkkr"]
Jul 07 07:35:53.313440 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:53.313424    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1803965-fbb3-401f-b2c3-a09205e9564d" containerName="main"
Jul 07 07:35:53.313487 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:53.313444    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1803965-fbb3-401f-b2c3-a09205e9564d" containerName="main"
Jul 07 07:35:53.313487 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:53.313477    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="174f3bf9-bcd3-470b-aa66-1990c379cda7" containerName="main"
Jul 07 07:35:53.313487 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:53.313485    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="174f3bf9-bcd3-470b-aa66-1990c379cda7" containerName="main"
Jul 07 07:35:53.313578 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:53.313565    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1803965-fbb3-401f-b2c3-a09205e9564d" containerName="main"
Jul 07 07:35:53.313611 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:53.313578    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="174f3bf9-bcd3-470b-aa66-1990c379cda7" containerName="main"
Jul 07 07:35:53.318131 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:53.318107    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-083d4e/sub-test-model-083d4e-kserve-8bf6b6cd7-7lkkr"
Jul 07 07:35:53.322196 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:53.321914    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-083d4e\"/\"sub-test-model-083d4e-kserve-self-signed-certs\""
Jul 07 07:35:53.322196 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:53.321918    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-083d4e\"/\"kube-root-ca.crt\""
Jul 07 07:35:53.322196 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:53.322041    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-083d4e\"/\"default-dockercfg-7l97m\""
Jul 07 07:35:53.322454 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:53.322194    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-083d4e\"/\"openshift-service-ca.crt\""
Jul 07 07:35:53.324740 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:53.324716    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-083d4e/sub-test-model-083d4e-kserve-8bf6b6cd7-7lkkr"]
Jul 07 07:35:53.440444 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:53.440407    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bedfc63-7ca6-4c27-af13-b48b20182b47-dshm\") pod \"sub-test-model-083d4e-kserve-8bf6b6cd7-7lkkr\" (UID: \"1bedfc63-7ca6-4c27-af13-b48b20182b47\") " pod="ai-tenant-e2e-shared-a-083d4e/sub-test-model-083d4e-kserve-8bf6b6cd7-7lkkr"
Jul 07 07:35:53.440629 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:53.440456    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bedfc63-7ca6-4c27-af13-b48b20182b47-tls-certs\") pod \"sub-test-model-083d4e-kserve-8bf6b6cd7-7lkkr\" (UID: \"1bedfc63-7ca6-4c27-af13-b48b20182b47\") " pod="ai-tenant-e2e-shared-a-083d4e/sub-test-model-083d4e-kserve-8bf6b6cd7-7lkkr"
Jul 07 07:35:53.440629 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:53.440485    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhwnw\" (UniqueName: \"kubernetes.io/projected/1bedfc63-7ca6-4c27-af13-b48b20182b47-kube-api-access-vhwnw\") pod \"sub-test-model-083d4e-kserve-8bf6b6cd7-7lkkr\" (UID: \"1bedfc63-7ca6-4c27-af13-b48b20182b47\") " pod="ai-tenant-e2e-shared-a-083d4e/sub-test-model-083d4e-kserve-8bf6b6cd7-7lkkr"
Jul 07 07:35:53.440629 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:53.440575    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bedfc63-7ca6-4c27-af13-b48b20182b47-tmp-dir\") pod \"sub-test-model-083d4e-kserve-8bf6b6cd7-7lkkr\" (UID: \"1bedfc63-7ca6-4c27-af13-b48b20182b47\") " pod="ai-tenant-e2e-shared-a-083d4e/sub-test-model-083d4e-kserve-8bf6b6cd7-7lkkr"
Jul 07 07:35:53.440757 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:53.440638    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bedfc63-7ca6-4c27-af13-b48b20182b47-home\") pod \"sub-test-model-083d4e-kserve-8bf6b6cd7-7lkkr\" (UID: \"1bedfc63-7ca6-4c27-af13-b48b20182b47\") " pod="ai-tenant-e2e-shared-a-083d4e/sub-test-model-083d4e-kserve-8bf6b6cd7-7lkkr"
Jul 07 07:35:53.440757 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:53.440688    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bedfc63-7ca6-4c27-af13-b48b20182b47-model-cache\") pod \"sub-test-model-083d4e-kserve-8bf6b6cd7-7lkkr\" (UID: \"1bedfc63-7ca6-4c27-af13-b48b20182b47\") " pod="ai-tenant-e2e-shared-a-083d4e/sub-test-model-083d4e-kserve-8bf6b6cd7-7lkkr"
Jul 07 07:35:53.541803 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:53.541766    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bedfc63-7ca6-4c27-af13-b48b20182b47-tmp-dir\") pod \"sub-test-model-083d4e-kserve-8bf6b6cd7-7lkkr\" (UID: \"1bedfc63-7ca6-4c27-af13-b48b20182b47\") " pod="ai-tenant-e2e-shared-a-083d4e/sub-test-model-083d4e-kserve-8bf6b6cd7-7lkkr"
Jul 07 07:35:53.542025 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:53.541827    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bedfc63-7ca6-4c27-af13-b48b20182b47-home\") pod \"sub-test-model-083d4e-kserve-8bf6b6cd7-7lkkr\" (UID: \"1bedfc63-7ca6-4c27-af13-b48b20182b47\") " pod="ai-tenant-e2e-shared-a-083d4e/sub-test-model-083d4e-kserve-8bf6b6cd7-7lkkr"
Jul 07 07:35:53.542025 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:53.541867    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bedfc63-7ca6-4c27-af13-b48b20182b47-model-cache\") pod \"sub-test-model-083d4e-kserve-8bf6b6cd7-7lkkr\" (UID: \"1bedfc63-7ca6-4c27-af13-b48b20182b47\") " pod="ai-tenant-e2e-shared-a-083d4e/sub-test-model-083d4e-kserve-8bf6b6cd7-7lkkr"
Jul 07 07:35:53.542025 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:53.541917    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bedfc63-7ca6-4c27-af13-b48b20182b47-dshm\") pod \"sub-test-model-083d4e-kserve-8bf6b6cd7-7lkkr\" (UID: \"1bedfc63-7ca6-4c27-af13-b48b20182b47\") " pod="ai-tenant-e2e-shared-a-083d4e/sub-test-model-083d4e-kserve-8bf6b6cd7-7lkkr"
Jul 07 07:35:53.542025 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:53.541981    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bedfc63-7ca6-4c27-af13-b48b20182b47-tls-certs\") pod \"sub-test-model-083d4e-kserve-8bf6b6cd7-7lkkr\" (UID: \"1bedfc63-7ca6-4c27-af13-b48b20182b47\") " pod="ai-tenant-e2e-shared-a-083d4e/sub-test-model-083d4e-kserve-8bf6b6cd7-7lkkr"
Jul 07 07:35:53.542258 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:53.542125    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhwnw\" (UniqueName: \"kubernetes.io/projected/1bedfc63-7ca6-4c27-af13-b48b20182b47-kube-api-access-vhwnw\") pod \"sub-test-model-083d4e-kserve-8bf6b6cd7-7lkkr\" (UID: \"1bedfc63-7ca6-4c27-af13-b48b20182b47\") " pod="ai-tenant-e2e-shared-a-083d4e/sub-test-model-083d4e-kserve-8bf6b6cd7-7lkkr"
Jul 07 07:35:53.542258 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:53.542245    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bedfc63-7ca6-4c27-af13-b48b20182b47-tmp-dir\") pod \"sub-test-model-083d4e-kserve-8bf6b6cd7-7lkkr\" (UID: \"1bedfc63-7ca6-4c27-af13-b48b20182b47\") " pod="ai-tenant-e2e-shared-a-083d4e/sub-test-model-083d4e-kserve-8bf6b6cd7-7lkkr"
Jul 07 07:35:53.542366 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:53.542273    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bedfc63-7ca6-4c27-af13-b48b20182b47-home\") pod \"sub-test-model-083d4e-kserve-8bf6b6cd7-7lkkr\" (UID: \"1bedfc63-7ca6-4c27-af13-b48b20182b47\") " pod="ai-tenant-e2e-shared-a-083d4e/sub-test-model-083d4e-kserve-8bf6b6cd7-7lkkr"
Jul 07 07:35:53.542366 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:53.542314    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bedfc63-7ca6-4c27-af13-b48b20182b47-model-cache\") pod \"sub-test-model-083d4e-kserve-8bf6b6cd7-7lkkr\" (UID: \"1bedfc63-7ca6-4c27-af13-b48b20182b47\") " pod="ai-tenant-e2e-shared-a-083d4e/sub-test-model-083d4e-kserve-8bf6b6cd7-7lkkr"
Jul 07 07:35:53.544368 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:53.544335    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bedfc63-7ca6-4c27-af13-b48b20182b47-dshm\") pod \"sub-test-model-083d4e-kserve-8bf6b6cd7-7lkkr\" (UID: \"1bedfc63-7ca6-4c27-af13-b48b20182b47\") " pod="ai-tenant-e2e-shared-a-083d4e/sub-test-model-083d4e-kserve-8bf6b6cd7-7lkkr"
Jul 07 07:35:53.544518 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:53.544498    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bedfc63-7ca6-4c27-af13-b48b20182b47-tls-certs\") pod \"sub-test-model-083d4e-kserve-8bf6b6cd7-7lkkr\" (UID: \"1bedfc63-7ca6-4c27-af13-b48b20182b47\") " pod="ai-tenant-e2e-shared-a-083d4e/sub-test-model-083d4e-kserve-8bf6b6cd7-7lkkr"
Jul 07 07:35:53.553988 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:53.553950    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhwnw\" (UniqueName: \"kubernetes.io/projected/1bedfc63-7ca6-4c27-af13-b48b20182b47-kube-api-access-vhwnw\") pod \"sub-test-model-083d4e-kserve-8bf6b6cd7-7lkkr\" (UID: \"1bedfc63-7ca6-4c27-af13-b48b20182b47\") " pod="ai-tenant-e2e-shared-a-083d4e/sub-test-model-083d4e-kserve-8bf6b6cd7-7lkkr"
Jul 07 07:35:53.628090 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:53.628008    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-083d4e/sub-test-model-083d4e-kserve-8bf6b6cd7-7lkkr"
Jul 07 07:35:53.760613 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:53.760588    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-083d4e/sub-test-model-083d4e-kserve-8bf6b6cd7-7lkkr"]
Jul 07 07:35:53.762887 ip-10-0-138-97 kubenswrapper[2579]: W0707 07:35:53.762861    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1bedfc63_7ca6_4c27_af13_b48b20182b47.slice/crio-f4dc7fe15dc3f30cc3cfdc2ca7e1647bbcbe5917ec72f92a857192241c2aa0db WatchSource:0}: Error finding container f4dc7fe15dc3f30cc3cfdc2ca7e1647bbcbe5917ec72f92a857192241c2aa0db: Status 404 returned error can't find the container with id f4dc7fe15dc3f30cc3cfdc2ca7e1647bbcbe5917ec72f92a857192241c2aa0db
Jul 07 07:35:54.534058 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:54.534017    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-083d4e/sub-test-model-083d4e-kserve-8bf6b6cd7-7lkkr" event={"ID":"1bedfc63-7ca6-4c27-af13-b48b20182b47","Type":"ContainerStarted","Data":"a5d23a88eb99ed53d78009235eedc9a8cb1aa7717f5bcebfc1e37b38925cd677"}
Jul 07 07:35:54.534058 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:54.534065    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-083d4e/sub-test-model-083d4e-kserve-8bf6b6cd7-7lkkr" event={"ID":"1bedfc63-7ca6-4c27-af13-b48b20182b47","Type":"ContainerStarted","Data":"f4dc7fe15dc3f30cc3cfdc2ca7e1647bbcbe5917ec72f92a857192241c2aa0db"}
Jul 07 07:35:54.558328 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:35:54.558276    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-083d4e/sub-test-model-083d4e-kserve-8bf6b6cd7-7lkkr" podStartSLOduration=1.558249472 podStartE2EDuration="1.558249472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 07:35:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 07:35:54.556131461 +0000 UTC m=+2234.099892964" watchObservedRunningTime="2026-07-07 07:35:54.558249472 +0000 UTC m=+2234.102010954"
Jul 07 07:36:03.628228 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:36:03.628135    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-083d4e/sub-test-model-083d4e-kserve-8bf6b6cd7-7lkkr"
Jul 07 07:36:03.628228 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:36:03.628191    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-083d4e/sub-test-model-083d4e-kserve-8bf6b6cd7-7lkkr"
Jul 07 07:36:03.640915 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:36:03.640889    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-083d4e/sub-test-model-083d4e-kserve-8bf6b6cd7-7lkkr"
Jul 07 07:36:04.591120 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:36:04.591092    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-083d4e/sub-test-model-083d4e-kserve-8bf6b6cd7-7lkkr"
Jul 07 07:36:16.194568 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:36:16.194532    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-1082ad/sub-test-model-1082ad-kserve-fcdf9c9d7-b48rd"]
Jul 07 07:36:16.198047 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:36:16.198026    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1082ad/sub-test-model-1082ad-kserve-fcdf9c9d7-b48rd"
Jul 07 07:36:16.202096 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:36:16.202065    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-1082ad\"/\"openshift-service-ca.crt\""
Jul 07 07:36:16.202249 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:36:16.202066    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-1082ad\"/\"default-dockercfg-5xj7h\""
Jul 07 07:36:16.202249 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:36:16.202065    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-1082ad\"/\"sub-test-model-1082ad-kserve-self-signed-certs\""
Jul 07 07:36:16.202249 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:36:16.202179    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-1082ad\"/\"kube-root-ca.crt\""
Jul 07 07:36:16.207902 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:36:16.207878    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-1082ad/sub-test-model-1082ad-kserve-fcdf9c9d7-b48rd"]
Jul 07 07:36:16.234283 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:36:16.234240    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e0b4654-8438-4637-9337-dc4518e88f23-tmp-dir\") pod \"sub-test-model-1082ad-kserve-fcdf9c9d7-b48rd\" (UID: \"0e0b4654-8438-4637-9337-dc4518e88f23\") " pod="ai-tenant-e2e-shared-b-1082ad/sub-test-model-1082ad-kserve-fcdf9c9d7-b48rd"
Jul 07 07:36:16.234480 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:36:16.234303    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e0b4654-8438-4637-9337-dc4518e88f23-model-cache\") pod \"sub-test-model-1082ad-kserve-fcdf9c9d7-b48rd\" (UID: \"0e0b4654-8438-4637-9337-dc4518e88f23\") " pod="ai-tenant-e2e-shared-b-1082ad/sub-test-model-1082ad-kserve-fcdf9c9d7-b48rd"
Jul 07 07:36:16.234480 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:36:16.234385    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e0b4654-8438-4637-9337-dc4518e88f23-dshm\") pod \"sub-test-model-1082ad-kserve-fcdf9c9d7-b48rd\" (UID: \"0e0b4654-8438-4637-9337-dc4518e88f23\") " pod="ai-tenant-e2e-shared-b-1082ad/sub-test-model-1082ad-kserve-fcdf9c9d7-b48rd"
Jul 07 07:36:16.234480 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:36:16.234454    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e0b4654-8438-4637-9337-dc4518e88f23-home\") pod \"sub-test-model-1082ad-kserve-fcdf9c9d7-b48rd\" (UID: \"0e0b4654-8438-4637-9337-dc4518e88f23\") " pod="ai-tenant-e2e-shared-b-1082ad/sub-test-model-1082ad-kserve-fcdf9c9d7-b48rd"
Jul 07 07:36:16.234659 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:36:16.234513    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e0b4654-8438-4637-9337-dc4518e88f23-tls-certs\") pod \"sub-test-model-1082ad-kserve-fcdf9c9d7-b48rd\" (UID: \"0e0b4654-8438-4637-9337-dc4518e88f23\") " pod="ai-tenant-e2e-shared-b-1082ad/sub-test-model-1082ad-kserve-fcdf9c9d7-b48rd"
Jul 07 07:36:16.234659 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:36:16.234557    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bd29\" (UniqueName: \"kubernetes.io/projected/0e0b4654-8438-4637-9337-dc4518e88f23-kube-api-access-9bd29\") pod \"sub-test-model-1082ad-kserve-fcdf9c9d7-b48rd\" (UID: \"0e0b4654-8438-4637-9337-dc4518e88f23\") " pod="ai-tenant-e2e-shared-b-1082ad/sub-test-model-1082ad-kserve-fcdf9c9d7-b48rd"
Jul 07 07:36:16.335033 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:36:16.334997    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e0b4654-8438-4637-9337-dc4518e88f23-tmp-dir\") pod \"sub-test-model-1082ad-kserve-fcdf9c9d7-b48rd\" (UID: \"0e0b4654-8438-4637-9337-dc4518e88f23\") " pod="ai-tenant-e2e-shared-b-1082ad/sub-test-model-1082ad-kserve-fcdf9c9d7-b48rd"
Jul 07 07:36:16.335220 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:36:16.335044    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e0b4654-8438-4637-9337-dc4518e88f23-model-cache\") pod \"sub-test-model-1082ad-kserve-fcdf9c9d7-b48rd\" (UID: \"0e0b4654-8438-4637-9337-dc4518e88f23\") " pod="ai-tenant-e2e-shared-b-1082ad/sub-test-model-1082ad-kserve-fcdf9c9d7-b48rd"
Jul 07 07:36:16.335220 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:36:16.335064    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e0b4654-8438-4637-9337-dc4518e88f23-dshm\") pod \"sub-test-model-1082ad-kserve-fcdf9c9d7-b48rd\" (UID: \"0e0b4654-8438-4637-9337-dc4518e88f23\") " pod="ai-tenant-e2e-shared-b-1082ad/sub-test-model-1082ad-kserve-fcdf9c9d7-b48rd"
Jul 07 07:36:16.335220 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:36:16.335100    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e0b4654-8438-4637-9337-dc4518e88f23-home\") pod \"sub-test-model-1082ad-kserve-fcdf9c9d7-b48rd\" (UID: \"0e0b4654-8438-4637-9337-dc4518e88f23\") " pod="ai-tenant-e2e-shared-b-1082ad/sub-test-model-1082ad-kserve-fcdf9c9d7-b48rd"
Jul 07 07:36:16.335220 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:36:16.335135    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e0b4654-8438-4637-9337-dc4518e88f23-tls-certs\") pod \"sub-test-model-1082ad-kserve-fcdf9c9d7-b48rd\" (UID: \"0e0b4654-8438-4637-9337-dc4518e88f23\") " pod="ai-tenant-e2e-shared-b-1082ad/sub-test-model-1082ad-kserve-fcdf9c9d7-b48rd"
Jul 07 07:36:16.335220 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:36:16.335162    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bd29\" (UniqueName: \"kubernetes.io/projected/0e0b4654-8438-4637-9337-dc4518e88f23-kube-api-access-9bd29\") pod \"sub-test-model-1082ad-kserve-fcdf9c9d7-b48rd\" (UID: \"0e0b4654-8438-4637-9337-dc4518e88f23\") " pod="ai-tenant-e2e-shared-b-1082ad/sub-test-model-1082ad-kserve-fcdf9c9d7-b48rd"
Jul 07 07:36:16.335483 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:36:16.335422    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e0b4654-8438-4637-9337-dc4518e88f23-tmp-dir\") pod \"sub-test-model-1082ad-kserve-fcdf9c9d7-b48rd\" (UID: \"0e0b4654-8438-4637-9337-dc4518e88f23\") " pod="ai-tenant-e2e-shared-b-1082ad/sub-test-model-1082ad-kserve-fcdf9c9d7-b48rd"
Jul 07 07:36:16.335537 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:36:16.335475    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e0b4654-8438-4637-9337-dc4518e88f23-model-cache\") pod \"sub-test-model-1082ad-kserve-fcdf9c9d7-b48rd\" (UID: \"0e0b4654-8438-4637-9337-dc4518e88f23\") " pod="ai-tenant-e2e-shared-b-1082ad/sub-test-model-1082ad-kserve-fcdf9c9d7-b48rd"
Jul 07 07:36:16.335581 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:36:16.335554    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e0b4654-8438-4637-9337-dc4518e88f23-home\") pod \"sub-test-model-1082ad-kserve-fcdf9c9d7-b48rd\" (UID: \"0e0b4654-8438-4637-9337-dc4518e88f23\") " pod="ai-tenant-e2e-shared-b-1082ad/sub-test-model-1082ad-kserve-fcdf9c9d7-b48rd"
Jul 07 07:36:16.337412 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:36:16.337389    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e0b4654-8438-4637-9337-dc4518e88f23-dshm\") pod \"sub-test-model-1082ad-kserve-fcdf9c9d7-b48rd\" (UID: \"0e0b4654-8438-4637-9337-dc4518e88f23\") " pod="ai-tenant-e2e-shared-b-1082ad/sub-test-model-1082ad-kserve-fcdf9c9d7-b48rd"
Jul 07 07:36:16.337603 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:36:16.337586    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e0b4654-8438-4637-9337-dc4518e88f23-tls-certs\") pod \"sub-test-model-1082ad-kserve-fcdf9c9d7-b48rd\" (UID: \"0e0b4654-8438-4637-9337-dc4518e88f23\") " pod="ai-tenant-e2e-shared-b-1082ad/sub-test-model-1082ad-kserve-fcdf9c9d7-b48rd"
Jul 07 07:36:16.345258 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:36:16.345232    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bd29\" (UniqueName: \"kubernetes.io/projected/0e0b4654-8438-4637-9337-dc4518e88f23-kube-api-access-9bd29\") pod \"sub-test-model-1082ad-kserve-fcdf9c9d7-b48rd\" (UID: \"0e0b4654-8438-4637-9337-dc4518e88f23\") " pod="ai-tenant-e2e-shared-b-1082ad/sub-test-model-1082ad-kserve-fcdf9c9d7-b48rd"
Jul 07 07:36:16.509638 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:36:16.509589    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1082ad/sub-test-model-1082ad-kserve-fcdf9c9d7-b48rd"
Jul 07 07:36:16.647902 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:36:16.647878    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-1082ad/sub-test-model-1082ad-kserve-fcdf9c9d7-b48rd"]
Jul 07 07:36:16.650219 ip-10-0-138-97 kubenswrapper[2579]: W0707 07:36:16.650188    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0e0b4654_8438_4637_9337_dc4518e88f23.slice/crio-eac8c34829dd154bf116ba2b2ab77697e35f8c5b5f467030934be22e66bc5a62 WatchSource:0}: Error finding container eac8c34829dd154bf116ba2b2ab77697e35f8c5b5f467030934be22e66bc5a62: Status 404 returned error can't find the container with id eac8c34829dd154bf116ba2b2ab77697e35f8c5b5f467030934be22e66bc5a62
Jul 07 07:36:17.640888 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:36:17.640851    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1082ad/sub-test-model-1082ad-kserve-fcdf9c9d7-b48rd" event={"ID":"0e0b4654-8438-4637-9337-dc4518e88f23","Type":"ContainerStarted","Data":"dd5839d2b3a6ee09839e0234a163ca74b6c1360ce5aa986a8b9f3809814ecfb8"}
Jul 07 07:36:17.640888 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:36:17.640891    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1082ad/sub-test-model-1082ad-kserve-fcdf9c9d7-b48rd" event={"ID":"0e0b4654-8438-4637-9337-dc4518e88f23","Type":"ContainerStarted","Data":"eac8c34829dd154bf116ba2b2ab77697e35f8c5b5f467030934be22e66bc5a62"}
Jul 07 07:36:17.665382 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:36:17.665321    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-1082ad/sub-test-model-1082ad-kserve-fcdf9c9d7-b48rd" podStartSLOduration=1.66530366 podStartE2EDuration="1.66530366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 07:36:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 07:36:17.66368934 +0000 UTC m=+2257.207450835" watchObservedRunningTime="2026-07-07 07:36:17.66530366 +0000 UTC m=+2257.209065140"
Jul 07 07:36:26.509906 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:36:26.509869    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-1082ad/sub-test-model-1082ad-kserve-fcdf9c9d7-b48rd"
Jul 07 07:36:26.510344 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:36:26.510149    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-1082ad/sub-test-model-1082ad-kserve-fcdf9c9d7-b48rd"
Jul 07 07:36:26.522452 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:36:26.522423    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-1082ad/sub-test-model-1082ad-kserve-fcdf9c9d7-b48rd"
Jul 07 07:36:26.691158 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:36:26.691122    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-1082ad/sub-test-model-1082ad-kserve-fcdf9c9d7-b48rd"
Jul 07 07:36:57.507950 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:36:57.507917    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-083d4e/sub-test-model-083d4e-kserve-8bf6b6cd7-7lkkr"]
Jul 07 07:36:58.490422 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:36:58.490392    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-1082ad/sub-test-model-1082ad-kserve-fcdf9c9d7-b48rd"]
Jul 07 07:37:05.905025 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:05.904988    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-083d4e/rate-test-model-083d4e-kserve-7d9586c6f9-ph66m"]
Jul 07 07:37:05.908570 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:05.908551    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-083d4e/rate-test-model-083d4e-kserve-7d9586c6f9-ph66m"
Jul 07 07:37:05.912270 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:05.912244    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-083d4e\"/\"rate-test-model-083d4e-kserve-self-signed-certs\""
Jul 07 07:37:05.912405 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:05.912290    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-083d4e/rate-test-model-083d4e-kserve-7d9586c6f9-ph66m"]
Jul 07 07:37:05.969602 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:05.969561    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee-dshm\") pod \"rate-test-model-083d4e-kserve-7d9586c6f9-ph66m\" (UID: \"5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee\") " pod="ai-tenant-e2e-shared-a-083d4e/rate-test-model-083d4e-kserve-7d9586c6f9-ph66m"
Jul 07 07:37:05.969794 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:05.969610    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee-tls-certs\") pod \"rate-test-model-083d4e-kserve-7d9586c6f9-ph66m\" (UID: \"5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee\") " pod="ai-tenant-e2e-shared-a-083d4e/rate-test-model-083d4e-kserve-7d9586c6f9-ph66m"
Jul 07 07:37:05.969794 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:05.969659    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee-home\") pod \"rate-test-model-083d4e-kserve-7d9586c6f9-ph66m\" (UID: \"5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee\") " pod="ai-tenant-e2e-shared-a-083d4e/rate-test-model-083d4e-kserve-7d9586c6f9-ph66m"
Jul 07 07:37:05.969794 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:05.969730    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee-tmp-dir\") pod \"rate-test-model-083d4e-kserve-7d9586c6f9-ph66m\" (UID: \"5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee\") " pod="ai-tenant-e2e-shared-a-083d4e/rate-test-model-083d4e-kserve-7d9586c6f9-ph66m"
Jul 07 07:37:05.969794 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:05.969781    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee-model-cache\") pod \"rate-test-model-083d4e-kserve-7d9586c6f9-ph66m\" (UID: \"5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee\") " pod="ai-tenant-e2e-shared-a-083d4e/rate-test-model-083d4e-kserve-7d9586c6f9-ph66m"
Jul 07 07:37:05.969988 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:05.969807    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fc9br\" (UniqueName: \"kubernetes.io/projected/5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee-kube-api-access-fc9br\") pod \"rate-test-model-083d4e-kserve-7d9586c6f9-ph66m\" (UID: \"5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee\") " pod="ai-tenant-e2e-shared-a-083d4e/rate-test-model-083d4e-kserve-7d9586c6f9-ph66m"
Jul 07 07:37:06.070629 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:06.070594    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee-dshm\") pod \"rate-test-model-083d4e-kserve-7d9586c6f9-ph66m\" (UID: \"5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee\") " pod="ai-tenant-e2e-shared-a-083d4e/rate-test-model-083d4e-kserve-7d9586c6f9-ph66m"
Jul 07 07:37:06.070823 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:06.070634    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee-tls-certs\") pod \"rate-test-model-083d4e-kserve-7d9586c6f9-ph66m\" (UID: \"5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee\") " pod="ai-tenant-e2e-shared-a-083d4e/rate-test-model-083d4e-kserve-7d9586c6f9-ph66m"
Jul 07 07:37:06.070823 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:06.070665    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee-home\") pod \"rate-test-model-083d4e-kserve-7d9586c6f9-ph66m\" (UID: \"5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee\") " pod="ai-tenant-e2e-shared-a-083d4e/rate-test-model-083d4e-kserve-7d9586c6f9-ph66m"
Jul 07 07:37:06.070823 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:06.070724    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee-tmp-dir\") pod \"rate-test-model-083d4e-kserve-7d9586c6f9-ph66m\" (UID: \"5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee\") " pod="ai-tenant-e2e-shared-a-083d4e/rate-test-model-083d4e-kserve-7d9586c6f9-ph66m"
Jul 07 07:37:06.070823 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:06.070764    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee-model-cache\") pod \"rate-test-model-083d4e-kserve-7d9586c6f9-ph66m\" (UID: \"5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee\") " pod="ai-tenant-e2e-shared-a-083d4e/rate-test-model-083d4e-kserve-7d9586c6f9-ph66m"
Jul 07 07:37:06.071057 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:06.070893    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc9br\" (UniqueName: \"kubernetes.io/projected/5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee-kube-api-access-fc9br\") pod \"rate-test-model-083d4e-kserve-7d9586c6f9-ph66m\" (UID: \"5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee\") " pod="ai-tenant-e2e-shared-a-083d4e/rate-test-model-083d4e-kserve-7d9586c6f9-ph66m"
Jul 07 07:37:06.071143 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:06.071119    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee-home\") pod \"rate-test-model-083d4e-kserve-7d9586c6f9-ph66m\" (UID: \"5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee\") " pod="ai-tenant-e2e-shared-a-083d4e/rate-test-model-083d4e-kserve-7d9586c6f9-ph66m"
Jul 07 07:37:06.071207 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:06.071167    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee-tmp-dir\") pod \"rate-test-model-083d4e-kserve-7d9586c6f9-ph66m\" (UID: \"5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee\") " pod="ai-tenant-e2e-shared-a-083d4e/rate-test-model-083d4e-kserve-7d9586c6f9-ph66m"
Jul 07 07:37:06.071207 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:06.071199    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee-model-cache\") pod \"rate-test-model-083d4e-kserve-7d9586c6f9-ph66m\" (UID: \"5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee\") " pod="ai-tenant-e2e-shared-a-083d4e/rate-test-model-083d4e-kserve-7d9586c6f9-ph66m"
Jul 07 07:37:06.072968 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:06.072941    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee-dshm\") pod \"rate-test-model-083d4e-kserve-7d9586c6f9-ph66m\" (UID: \"5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee\") " pod="ai-tenant-e2e-shared-a-083d4e/rate-test-model-083d4e-kserve-7d9586c6f9-ph66m"
Jul 07 07:37:06.073220 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:06.073203    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee-tls-certs\") pod \"rate-test-model-083d4e-kserve-7d9586c6f9-ph66m\" (UID: \"5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee\") " pod="ai-tenant-e2e-shared-a-083d4e/rate-test-model-083d4e-kserve-7d9586c6f9-ph66m"
Jul 07 07:37:06.080892 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:06.080872    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc9br\" (UniqueName: \"kubernetes.io/projected/5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee-kube-api-access-fc9br\") pod \"rate-test-model-083d4e-kserve-7d9586c6f9-ph66m\" (UID: \"5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee\") " pod="ai-tenant-e2e-shared-a-083d4e/rate-test-model-083d4e-kserve-7d9586c6f9-ph66m"
Jul 07 07:37:06.218268 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:06.218163    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-083d4e/rate-test-model-083d4e-kserve-7d9586c6f9-ph66m"
Jul 07 07:37:06.354678 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:06.354654    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-083d4e/rate-test-model-083d4e-kserve-7d9586c6f9-ph66m"]
Jul 07 07:37:06.356571 ip-10-0-138-97 kubenswrapper[2579]: W0707 07:37:06.356543    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5ef93b4e_ab1b_4f75_ab4d_246a0cb1eaee.slice/crio-6b920b6f8a3a9d906e0be78d6527328efe0ce42531769f7a14ea3c5336bf7e49 WatchSource:0}: Error finding container 6b920b6f8a3a9d906e0be78d6527328efe0ce42531769f7a14ea3c5336bf7e49: Status 404 returned error can't find the container with id 6b920b6f8a3a9d906e0be78d6527328efe0ce42531769f7a14ea3c5336bf7e49
Jul 07 07:37:06.862002 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:06.861938    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-083d4e/rate-test-model-083d4e-kserve-7d9586c6f9-ph66m" event={"ID":"5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee","Type":"ContainerStarted","Data":"2792d1f88e1cf12ee390cdf5ab58c81f103101e0faa091498d59e474c23a509f"}
Jul 07 07:37:06.862002 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:06.862007    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-083d4e/rate-test-model-083d4e-kserve-7d9586c6f9-ph66m" event={"ID":"5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee","Type":"ContainerStarted","Data":"6b920b6f8a3a9d906e0be78d6527328efe0ce42531769f7a14ea3c5336bf7e49"}
Jul 07 07:37:06.883792 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:06.883740    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-083d4e/rate-test-model-083d4e-kserve-7d9586c6f9-ph66m" podStartSLOduration=1.883726217 podStartE2EDuration="1.883726217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 07:37:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 07:37:06.881708246 +0000 UTC m=+2306.425469729" watchObservedRunningTime="2026-07-07 07:37:06.883726217 +0000 UTC m=+2306.427487739"
Jul 07 07:37:12.528090 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:12.528026    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-083d4e/sub-test-model-083d4e-kserve-8bf6b6cd7-7lkkr" podUID="1bedfc63-7ca6-4c27-af13-b48b20182b47" containerName="main" containerID="cri-o://a5d23a88eb99ed53d78009235eedc9a8cb1aa7717f5bcebfc1e37b38925cd677" gracePeriod=45
Jul 07 07:37:12.788253 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:12.788187    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-083d4e/sub-test-model-083d4e-kserve-8bf6b6cd7-7lkkr"
Jul 07 07:37:12.831951 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:12.831922    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bedfc63-7ca6-4c27-af13-b48b20182b47-tmp-dir\") pod \"1bedfc63-7ca6-4c27-af13-b48b20182b47\" (UID: \"1bedfc63-7ca6-4c27-af13-b48b20182b47\") "
Jul 07 07:37:12.832143 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:12.831986    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bedfc63-7ca6-4c27-af13-b48b20182b47-model-cache\") pod \"1bedfc63-7ca6-4c27-af13-b48b20182b47\" (UID: \"1bedfc63-7ca6-4c27-af13-b48b20182b47\") "
Jul 07 07:37:12.832143 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:12.832058    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bedfc63-7ca6-4c27-af13-b48b20182b47-tls-certs\") pod \"1bedfc63-7ca6-4c27-af13-b48b20182b47\" (UID: \"1bedfc63-7ca6-4c27-af13-b48b20182b47\") "
Jul 07 07:37:12.832143 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:12.832089    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bedfc63-7ca6-4c27-af13-b48b20182b47-home\") pod \"1bedfc63-7ca6-4c27-af13-b48b20182b47\" (UID: \"1bedfc63-7ca6-4c27-af13-b48b20182b47\") "
Jul 07 07:37:12.832143 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:12.832133    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bedfc63-7ca6-4c27-af13-b48b20182b47-dshm\") pod \"1bedfc63-7ca6-4c27-af13-b48b20182b47\" (UID: \"1bedfc63-7ca6-4c27-af13-b48b20182b47\") "
Jul 07 07:37:12.832341 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:12.832169    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vhwnw\" (UniqueName: \"kubernetes.io/projected/1bedfc63-7ca6-4c27-af13-b48b20182b47-kube-api-access-vhwnw\") pod \"1bedfc63-7ca6-4c27-af13-b48b20182b47\" (UID: \"1bedfc63-7ca6-4c27-af13-b48b20182b47\") "
Jul 07 07:37:12.832341 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:12.832191    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bedfc63-7ca6-4c27-af13-b48b20182b47-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1bedfc63-7ca6-4c27-af13-b48b20182b47" (UID: "1bedfc63-7ca6-4c27-af13-b48b20182b47"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 07:37:12.832341 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:12.832270    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bedfc63-7ca6-4c27-af13-b48b20182b47-model-cache" (OuterVolumeSpecName: "model-cache") pod "1bedfc63-7ca6-4c27-af13-b48b20182b47" (UID: "1bedfc63-7ca6-4c27-af13-b48b20182b47"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 07:37:12.832497 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:12.832353    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bedfc63-7ca6-4c27-af13-b48b20182b47-home" (OuterVolumeSpecName: "home") pod "1bedfc63-7ca6-4c27-af13-b48b20182b47" (UID: "1bedfc63-7ca6-4c27-af13-b48b20182b47"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 07:37:12.832555 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:12.832498    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bedfc63-7ca6-4c27-af13-b48b20182b47-home\") on node \"ip-10-0-138-97.ec2.internal\" DevicePath \"\""
Jul 07 07:37:12.832555 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:12.832518    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bedfc63-7ca6-4c27-af13-b48b20182b47-tmp-dir\") on node \"ip-10-0-138-97.ec2.internal\" DevicePath \"\""
Jul 07 07:37:12.832555 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:12.832528    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bedfc63-7ca6-4c27-af13-b48b20182b47-model-cache\") on node \"ip-10-0-138-97.ec2.internal\" DevicePath \"\""
Jul 07 07:37:12.834276 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:12.834244    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bedfc63-7ca6-4c27-af13-b48b20182b47-kube-api-access-vhwnw" (OuterVolumeSpecName: "kube-api-access-vhwnw") pod "1bedfc63-7ca6-4c27-af13-b48b20182b47" (UID: "1bedfc63-7ca6-4c27-af13-b48b20182b47"). InnerVolumeSpecName "kube-api-access-vhwnw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 07:37:12.834624 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:12.834605    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bedfc63-7ca6-4c27-af13-b48b20182b47-dshm" (OuterVolumeSpecName: "dshm") pod "1bedfc63-7ca6-4c27-af13-b48b20182b47" (UID: "1bedfc63-7ca6-4c27-af13-b48b20182b47"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 07:37:12.834679 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:12.834620    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bedfc63-7ca6-4c27-af13-b48b20182b47-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1bedfc63-7ca6-4c27-af13-b48b20182b47" (UID: "1bedfc63-7ca6-4c27-af13-b48b20182b47"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 07:37:12.889855 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:12.889817    2579 generic.go:356] "Generic (PLEG): container finished" podID="1bedfc63-7ca6-4c27-af13-b48b20182b47" containerID="a5d23a88eb99ed53d78009235eedc9a8cb1aa7717f5bcebfc1e37b38925cd677" exitCode=0
Jul 07 07:37:12.889855 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:12.889857    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-083d4e/sub-test-model-083d4e-kserve-8bf6b6cd7-7lkkr"
Jul 07 07:37:12.890129 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:12.889907    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-083d4e/sub-test-model-083d4e-kserve-8bf6b6cd7-7lkkr" event={"ID":"1bedfc63-7ca6-4c27-af13-b48b20182b47","Type":"ContainerDied","Data":"a5d23a88eb99ed53d78009235eedc9a8cb1aa7717f5bcebfc1e37b38925cd677"}
Jul 07 07:37:12.890129 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:12.889946    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-083d4e/sub-test-model-083d4e-kserve-8bf6b6cd7-7lkkr" event={"ID":"1bedfc63-7ca6-4c27-af13-b48b20182b47","Type":"ContainerDied","Data":"f4dc7fe15dc3f30cc3cfdc2ca7e1647bbcbe5917ec72f92a857192241c2aa0db"}
Jul 07 07:37:12.890129 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:12.889988    2579 scope.go:117] "RemoveContainer" containerID="a5d23a88eb99ed53d78009235eedc9a8cb1aa7717f5bcebfc1e37b38925cd677"
Jul 07 07:37:12.899085 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:12.899066    2579 scope.go:117] "RemoveContainer" containerID="a5d23a88eb99ed53d78009235eedc9a8cb1aa7717f5bcebfc1e37b38925cd677"
Jul 07 07:37:12.899347 ip-10-0-138-97 kubenswrapper[2579]: E0707 07:37:12.899329    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5d23a88eb99ed53d78009235eedc9a8cb1aa7717f5bcebfc1e37b38925cd677\": container with ID starting with a5d23a88eb99ed53d78009235eedc9a8cb1aa7717f5bcebfc1e37b38925cd677 not found: ID does not exist" containerID="a5d23a88eb99ed53d78009235eedc9a8cb1aa7717f5bcebfc1e37b38925cd677"
Jul 07 07:37:12.899395 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:12.899358    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5d23a88eb99ed53d78009235eedc9a8cb1aa7717f5bcebfc1e37b38925cd677"} err="failed to get container status \"a5d23a88eb99ed53d78009235eedc9a8cb1aa7717f5bcebfc1e37b38925cd677\": rpc error: code = NotFound desc = could not find container \"a5d23a88eb99ed53d78009235eedc9a8cb1aa7717f5bcebfc1e37b38925cd677\": container with ID starting with a5d23a88eb99ed53d78009235eedc9a8cb1aa7717f5bcebfc1e37b38925cd677 not found: ID does not exist"
Jul 07 07:37:12.915406 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:12.915376    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-083d4e/sub-test-model-083d4e-kserve-8bf6b6cd7-7lkkr"]
Jul 07 07:37:12.920965 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:12.920929    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-083d4e/sub-test-model-083d4e-kserve-8bf6b6cd7-7lkkr"]
Jul 07 07:37:12.933703 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:12.933682    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhwnw\" (UniqueName: \"kubernetes.io/projected/1bedfc63-7ca6-4c27-af13-b48b20182b47-kube-api-access-vhwnw\") on node \"ip-10-0-138-97.ec2.internal\" DevicePath \"\""
Jul 07 07:37:12.933786 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:12.933705    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bedfc63-7ca6-4c27-af13-b48b20182b47-tls-certs\") on node \"ip-10-0-138-97.ec2.internal\" DevicePath \"\""
Jul 07 07:37:12.933786 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:12.933718    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bedfc63-7ca6-4c27-af13-b48b20182b47-dshm\") on node \"ip-10-0-138-97.ec2.internal\" DevicePath \"\""
Jul 07 07:37:12.964115 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:12.964081    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1bedfc63-7ca6-4c27-af13-b48b20182b47" path="/var/lib/kubelet/pods/1bedfc63-7ca6-4c27-af13-b48b20182b47/volumes"
Jul 07 07:37:13.510320 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:13.510277    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-1082ad/sub-test-model-1082ad-kserve-fcdf9c9d7-b48rd" podUID="0e0b4654-8438-4637-9337-dc4518e88f23" containerName="main" containerID="cri-o://dd5839d2b3a6ee09839e0234a163ca74b6c1360ce5aa986a8b9f3809814ecfb8" gracePeriod=45
Jul 07 07:37:13.769754 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:13.769693    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1082ad/sub-test-model-1082ad-kserve-fcdf9c9d7-b48rd"
Jul 07 07:37:13.842550 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:13.842521    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9bd29\" (UniqueName: \"kubernetes.io/projected/0e0b4654-8438-4637-9337-dc4518e88f23-kube-api-access-9bd29\") pod \"0e0b4654-8438-4637-9337-dc4518e88f23\" (UID: \"0e0b4654-8438-4637-9337-dc4518e88f23\") "
Jul 07 07:37:13.842723 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:13.842593    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e0b4654-8438-4637-9337-dc4518e88f23-dshm\") pod \"0e0b4654-8438-4637-9337-dc4518e88f23\" (UID: \"0e0b4654-8438-4637-9337-dc4518e88f23\") "
Jul 07 07:37:13.842723 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:13.842624    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e0b4654-8438-4637-9337-dc4518e88f23-model-cache\") pod \"0e0b4654-8438-4637-9337-dc4518e88f23\" (UID: \"0e0b4654-8438-4637-9337-dc4518e88f23\") "
Jul 07 07:37:13.842723 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:13.842643    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e0b4654-8438-4637-9337-dc4518e88f23-tls-certs\") pod \"0e0b4654-8438-4637-9337-dc4518e88f23\" (UID: \"0e0b4654-8438-4637-9337-dc4518e88f23\") "
Jul 07 07:37:13.842723 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:13.842664    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e0b4654-8438-4637-9337-dc4518e88f23-home\") pod \"0e0b4654-8438-4637-9337-dc4518e88f23\" (UID: \"0e0b4654-8438-4637-9337-dc4518e88f23\") "
Jul 07 07:37:13.842723 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:13.842687    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e0b4654-8438-4637-9337-dc4518e88f23-tmp-dir\") pod \"0e0b4654-8438-4637-9337-dc4518e88f23\" (UID: \"0e0b4654-8438-4637-9337-dc4518e88f23\") "
Jul 07 07:37:13.843015 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:13.842915    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e0b4654-8438-4637-9337-dc4518e88f23-model-cache" (OuterVolumeSpecName: "model-cache") pod "0e0b4654-8438-4637-9337-dc4518e88f23" (UID: "0e0b4654-8438-4637-9337-dc4518e88f23"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 07:37:13.843072 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:13.843024    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e0b4654-8438-4637-9337-dc4518e88f23-home" (OuterVolumeSpecName: "home") pod "0e0b4654-8438-4637-9337-dc4518e88f23" (UID: "0e0b4654-8438-4637-9337-dc4518e88f23"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 07:37:13.843111 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:13.843072    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e0b4654-8438-4637-9337-dc4518e88f23-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0e0b4654-8438-4637-9337-dc4518e88f23" (UID: "0e0b4654-8438-4637-9337-dc4518e88f23"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 07:37:13.844809 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:13.844781    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e0b4654-8438-4637-9337-dc4518e88f23-dshm" (OuterVolumeSpecName: "dshm") pod "0e0b4654-8438-4637-9337-dc4518e88f23" (UID: "0e0b4654-8438-4637-9337-dc4518e88f23"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 07:37:13.844809 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:13.844781    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e0b4654-8438-4637-9337-dc4518e88f23-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0e0b4654-8438-4637-9337-dc4518e88f23" (UID: "0e0b4654-8438-4637-9337-dc4518e88f23"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 07:37:13.844943 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:13.844823    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e0b4654-8438-4637-9337-dc4518e88f23-kube-api-access-9bd29" (OuterVolumeSpecName: "kube-api-access-9bd29") pod "0e0b4654-8438-4637-9337-dc4518e88f23" (UID: "0e0b4654-8438-4637-9337-dc4518e88f23"). InnerVolumeSpecName "kube-api-access-9bd29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 07:37:13.896032 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:13.895994    2579 generic.go:356] "Generic (PLEG): container finished" podID="0e0b4654-8438-4637-9337-dc4518e88f23" containerID="dd5839d2b3a6ee09839e0234a163ca74b6c1360ce5aa986a8b9f3809814ecfb8" exitCode=0
Jul 07 07:37:13.896216 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:13.896037    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1082ad/sub-test-model-1082ad-kserve-fcdf9c9d7-b48rd"
Jul 07 07:37:13.896216 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:13.896078    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1082ad/sub-test-model-1082ad-kserve-fcdf9c9d7-b48rd" event={"ID":"0e0b4654-8438-4637-9337-dc4518e88f23","Type":"ContainerDied","Data":"dd5839d2b3a6ee09839e0234a163ca74b6c1360ce5aa986a8b9f3809814ecfb8"}
Jul 07 07:37:13.896216 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:13.896118    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1082ad/sub-test-model-1082ad-kserve-fcdf9c9d7-b48rd" event={"ID":"0e0b4654-8438-4637-9337-dc4518e88f23","Type":"ContainerDied","Data":"eac8c34829dd154bf116ba2b2ab77697e35f8c5b5f467030934be22e66bc5a62"}
Jul 07 07:37:13.896216 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:13.896134    2579 scope.go:117] "RemoveContainer" containerID="dd5839d2b3a6ee09839e0234a163ca74b6c1360ce5aa986a8b9f3809814ecfb8"
Jul 07 07:37:13.904818 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:13.904798    2579 scope.go:117] "RemoveContainer" containerID="dd5839d2b3a6ee09839e0234a163ca74b6c1360ce5aa986a8b9f3809814ecfb8"
Jul 07 07:37:13.905099 ip-10-0-138-97 kubenswrapper[2579]: E0707 07:37:13.905082    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd5839d2b3a6ee09839e0234a163ca74b6c1360ce5aa986a8b9f3809814ecfb8\": container with ID starting with dd5839d2b3a6ee09839e0234a163ca74b6c1360ce5aa986a8b9f3809814ecfb8 not found: ID does not exist" containerID="dd5839d2b3a6ee09839e0234a163ca74b6c1360ce5aa986a8b9f3809814ecfb8"
Jul 07 07:37:13.905152 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:13.905109    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd5839d2b3a6ee09839e0234a163ca74b6c1360ce5aa986a8b9f3809814ecfb8"} err="failed to get container status \"dd5839d2b3a6ee09839e0234a163ca74b6c1360ce5aa986a8b9f3809814ecfb8\": rpc error: code = NotFound desc = could not find container \"dd5839d2b3a6ee09839e0234a163ca74b6c1360ce5aa986a8b9f3809814ecfb8\": container with ID starting with dd5839d2b3a6ee09839e0234a163ca74b6c1360ce5aa986a8b9f3809814ecfb8 not found: ID does not exist"
Jul 07 07:37:13.919082 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:13.919059    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-1082ad/sub-test-model-1082ad-kserve-fcdf9c9d7-b48rd"]
Jul 07 07:37:13.923640 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:13.923617    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-1082ad/sub-test-model-1082ad-kserve-fcdf9c9d7-b48rd"]
Jul 07 07:37:13.944178 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:13.944151    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e0b4654-8438-4637-9337-dc4518e88f23-model-cache\") on node \"ip-10-0-138-97.ec2.internal\" DevicePath \"\""
Jul 07 07:37:13.944178 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:13.944174    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e0b4654-8438-4637-9337-dc4518e88f23-tls-certs\") on node \"ip-10-0-138-97.ec2.internal\" DevicePath \"\""
Jul 07 07:37:13.944178 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:13.944182    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e0b4654-8438-4637-9337-dc4518e88f23-home\") on node \"ip-10-0-138-97.ec2.internal\" DevicePath \"\""
Jul 07 07:37:13.944349 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:13.944189    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e0b4654-8438-4637-9337-dc4518e88f23-tmp-dir\") on node \"ip-10-0-138-97.ec2.internal\" DevicePath \"\""
Jul 07 07:37:13.944349 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:13.944198    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bd29\" (UniqueName: \"kubernetes.io/projected/0e0b4654-8438-4637-9337-dc4518e88f23-kube-api-access-9bd29\") on node \"ip-10-0-138-97.ec2.internal\" DevicePath \"\""
Jul 07 07:37:13.944349 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:13.944207    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e0b4654-8438-4637-9337-dc4518e88f23-dshm\") on node \"ip-10-0-138-97.ec2.internal\" DevicePath \"\""
Jul 07 07:37:14.964374 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:14.964338    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e0b4654-8438-4637-9337-dc4518e88f23" path="/var/lib/kubelet/pods/0e0b4654-8438-4637-9337-dc4518e88f23/volumes"
Jul 07 07:37:16.218498 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:16.218466    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-083d4e/rate-test-model-083d4e-kserve-7d9586c6f9-ph66m"
Jul 07 07:37:16.218880 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:16.218533    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-083d4e/rate-test-model-083d4e-kserve-7d9586c6f9-ph66m"
Jul 07 07:37:16.231105 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:16.231076    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-083d4e/rate-test-model-083d4e-kserve-7d9586c6f9-ph66m"
Jul 07 07:37:16.920908 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:16.920880    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-083d4e/rate-test-model-083d4e-kserve-7d9586c6f9-ph66m"
Jul 07 07:37:35.217187 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:35.217104    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-1082ad/rate-test-model-1082ad-kserve-ff6dc744c-8g5nz"]
Jul 07 07:37:35.217650 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:35.217579    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e0b4654-8438-4637-9337-dc4518e88f23" containerName="main"
Jul 07 07:37:35.217650 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:35.217599    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e0b4654-8438-4637-9337-dc4518e88f23" containerName="main"
Jul 07 07:37:35.217650 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:35.217623    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bedfc63-7ca6-4c27-af13-b48b20182b47" containerName="main"
Jul 07 07:37:35.217650 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:35.217631    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bedfc63-7ca6-4c27-af13-b48b20182b47" containerName="main"
Jul 07 07:37:35.217852 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:35.217709    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bedfc63-7ca6-4c27-af13-b48b20182b47" containerName="main"
Jul 07 07:37:35.217852 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:35.217728    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e0b4654-8438-4637-9337-dc4518e88f23" containerName="main"
Jul 07 07:37:35.220872 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:35.220849    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1082ad/rate-test-model-1082ad-kserve-ff6dc744c-8g5nz"
Jul 07 07:37:35.225047 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:35.225021    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-1082ad\"/\"openshift-service-ca.crt\""
Jul 07 07:37:35.225159 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:35.225026    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-1082ad\"/\"rate-test-model-1082ad-kserve-self-signed-certs\""
Jul 07 07:37:35.225159 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:35.225058    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-1082ad\"/\"kube-root-ca.crt\""
Jul 07 07:37:35.225159 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:35.225102    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-1082ad\"/\"default-dockercfg-5xj7h\""
Jul 07 07:37:35.230544 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:35.230521    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-1082ad/rate-test-model-1082ad-kserve-ff6dc744c-8g5nz"]
Jul 07 07:37:35.350161 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:35.350124    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d644c04-7621-4248-bad3-0b54fe2579f2-model-cache\") pod \"rate-test-model-1082ad-kserve-ff6dc744c-8g5nz\" (UID: \"8d644c04-7621-4248-bad3-0b54fe2579f2\") " pod="ai-tenant-e2e-shared-b-1082ad/rate-test-model-1082ad-kserve-ff6dc744c-8g5nz"
Jul 07 07:37:35.350356 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:35.350180    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnwfp\" (UniqueName: \"kubernetes.io/projected/8d644c04-7621-4248-bad3-0b54fe2579f2-kube-api-access-dnwfp\") pod \"rate-test-model-1082ad-kserve-ff6dc744c-8g5nz\" (UID: \"8d644c04-7621-4248-bad3-0b54fe2579f2\") " pod="ai-tenant-e2e-shared-b-1082ad/rate-test-model-1082ad-kserve-ff6dc744c-8g5nz"
Jul 07 07:37:35.350356 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:35.350218    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d644c04-7621-4248-bad3-0b54fe2579f2-home\") pod \"rate-test-model-1082ad-kserve-ff6dc744c-8g5nz\" (UID: \"8d644c04-7621-4248-bad3-0b54fe2579f2\") " pod="ai-tenant-e2e-shared-b-1082ad/rate-test-model-1082ad-kserve-ff6dc744c-8g5nz"
Jul 07 07:37:35.350356 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:35.350249    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d644c04-7621-4248-bad3-0b54fe2579f2-tmp-dir\") pod \"rate-test-model-1082ad-kserve-ff6dc744c-8g5nz\" (UID: \"8d644c04-7621-4248-bad3-0b54fe2579f2\") " pod="ai-tenant-e2e-shared-b-1082ad/rate-test-model-1082ad-kserve-ff6dc744c-8g5nz"
Jul 07 07:37:35.350356 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:35.350297    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d644c04-7621-4248-bad3-0b54fe2579f2-tls-certs\") pod \"rate-test-model-1082ad-kserve-ff6dc744c-8g5nz\" (UID: \"8d644c04-7621-4248-bad3-0b54fe2579f2\") " pod="ai-tenant-e2e-shared-b-1082ad/rate-test-model-1082ad-kserve-ff6dc744c-8g5nz"
Jul 07 07:37:35.350356 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:35.350316    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d644c04-7621-4248-bad3-0b54fe2579f2-dshm\") pod \"rate-test-model-1082ad-kserve-ff6dc744c-8g5nz\" (UID: \"8d644c04-7621-4248-bad3-0b54fe2579f2\") " pod="ai-tenant-e2e-shared-b-1082ad/rate-test-model-1082ad-kserve-ff6dc744c-8g5nz"
Jul 07 07:37:35.450893 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:35.450857    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d644c04-7621-4248-bad3-0b54fe2579f2-tmp-dir\") pod \"rate-test-model-1082ad-kserve-ff6dc744c-8g5nz\" (UID: \"8d644c04-7621-4248-bad3-0b54fe2579f2\") " pod="ai-tenant-e2e-shared-b-1082ad/rate-test-model-1082ad-kserve-ff6dc744c-8g5nz"
Jul 07 07:37:35.451122 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:35.450904    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d644c04-7621-4248-bad3-0b54fe2579f2-tls-certs\") pod \"rate-test-model-1082ad-kserve-ff6dc744c-8g5nz\" (UID: \"8d644c04-7621-4248-bad3-0b54fe2579f2\") " pod="ai-tenant-e2e-shared-b-1082ad/rate-test-model-1082ad-kserve-ff6dc744c-8g5nz"
Jul 07 07:37:35.451122 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:35.450924    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d644c04-7621-4248-bad3-0b54fe2579f2-dshm\") pod \"rate-test-model-1082ad-kserve-ff6dc744c-8g5nz\" (UID: \"8d644c04-7621-4248-bad3-0b54fe2579f2\") " pod="ai-tenant-e2e-shared-b-1082ad/rate-test-model-1082ad-kserve-ff6dc744c-8g5nz"
Jul 07 07:37:35.451122 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:35.451001    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d644c04-7621-4248-bad3-0b54fe2579f2-model-cache\") pod \"rate-test-model-1082ad-kserve-ff6dc744c-8g5nz\" (UID: \"8d644c04-7621-4248-bad3-0b54fe2579f2\") " pod="ai-tenant-e2e-shared-b-1082ad/rate-test-model-1082ad-kserve-ff6dc744c-8g5nz"
Jul 07 07:37:35.451122 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:35.451055    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnwfp\" (UniqueName: \"kubernetes.io/projected/8d644c04-7621-4248-bad3-0b54fe2579f2-kube-api-access-dnwfp\") pod \"rate-test-model-1082ad-kserve-ff6dc744c-8g5nz\" (UID: \"8d644c04-7621-4248-bad3-0b54fe2579f2\") " pod="ai-tenant-e2e-shared-b-1082ad/rate-test-model-1082ad-kserve-ff6dc744c-8g5nz"
Jul 07 07:37:35.451122 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:35.451082    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d644c04-7621-4248-bad3-0b54fe2579f2-home\") pod \"rate-test-model-1082ad-kserve-ff6dc744c-8g5nz\" (UID: \"8d644c04-7621-4248-bad3-0b54fe2579f2\") " pod="ai-tenant-e2e-shared-b-1082ad/rate-test-model-1082ad-kserve-ff6dc744c-8g5nz"
Jul 07 07:37:35.451441 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:35.451421    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d644c04-7621-4248-bad3-0b54fe2579f2-tmp-dir\") pod \"rate-test-model-1082ad-kserve-ff6dc744c-8g5nz\" (UID: \"8d644c04-7621-4248-bad3-0b54fe2579f2\") " pod="ai-tenant-e2e-shared-b-1082ad/rate-test-model-1082ad-kserve-ff6dc744c-8g5nz"
Jul 07 07:37:35.451505 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:35.451453    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d644c04-7621-4248-bad3-0b54fe2579f2-model-cache\") pod \"rate-test-model-1082ad-kserve-ff6dc744c-8g5nz\" (UID: \"8d644c04-7621-4248-bad3-0b54fe2579f2\") " pod="ai-tenant-e2e-shared-b-1082ad/rate-test-model-1082ad-kserve-ff6dc744c-8g5nz"
Jul 07 07:37:35.451505 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:35.451474    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d644c04-7621-4248-bad3-0b54fe2579f2-home\") pod \"rate-test-model-1082ad-kserve-ff6dc744c-8g5nz\" (UID: \"8d644c04-7621-4248-bad3-0b54fe2579f2\") " pod="ai-tenant-e2e-shared-b-1082ad/rate-test-model-1082ad-kserve-ff6dc744c-8g5nz"
Jul 07 07:37:35.453227 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:35.453199    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d644c04-7621-4248-bad3-0b54fe2579f2-dshm\") pod \"rate-test-model-1082ad-kserve-ff6dc744c-8g5nz\" (UID: \"8d644c04-7621-4248-bad3-0b54fe2579f2\") " pod="ai-tenant-e2e-shared-b-1082ad/rate-test-model-1082ad-kserve-ff6dc744c-8g5nz"
Jul 07 07:37:35.453541 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:35.453523    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d644c04-7621-4248-bad3-0b54fe2579f2-tls-certs\") pod \"rate-test-model-1082ad-kserve-ff6dc744c-8g5nz\" (UID: \"8d644c04-7621-4248-bad3-0b54fe2579f2\") " pod="ai-tenant-e2e-shared-b-1082ad/rate-test-model-1082ad-kserve-ff6dc744c-8g5nz"
Jul 07 07:37:35.462648 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:35.462627    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnwfp\" (UniqueName: \"kubernetes.io/projected/8d644c04-7621-4248-bad3-0b54fe2579f2-kube-api-access-dnwfp\") pod \"rate-test-model-1082ad-kserve-ff6dc744c-8g5nz\" (UID: \"8d644c04-7621-4248-bad3-0b54fe2579f2\") " pod="ai-tenant-e2e-shared-b-1082ad/rate-test-model-1082ad-kserve-ff6dc744c-8g5nz"
Jul 07 07:37:35.531673 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:35.531639    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1082ad/rate-test-model-1082ad-kserve-ff6dc744c-8g5nz"
Jul 07 07:37:35.665365 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:35.665323    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-1082ad/rate-test-model-1082ad-kserve-ff6dc744c-8g5nz"]
Jul 07 07:37:35.667674 ip-10-0-138-97 kubenswrapper[2579]: W0707 07:37:35.667648    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8d644c04_7621_4248_bad3_0b54fe2579f2.slice/crio-f15069abac371d308baae96c8c280e2c67e13271b1e9ae6353aa2aab106f656d WatchSource:0}: Error finding container f15069abac371d308baae96c8c280e2c67e13271b1e9ae6353aa2aab106f656d: Status 404 returned error can't find the container with id f15069abac371d308baae96c8c280e2c67e13271b1e9ae6353aa2aab106f656d
Jul 07 07:37:35.994117 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:35.994081    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1082ad/rate-test-model-1082ad-kserve-ff6dc744c-8g5nz" event={"ID":"8d644c04-7621-4248-bad3-0b54fe2579f2","Type":"ContainerStarted","Data":"379781c90b634fb41251c3576d8019a92fbfd3a800eef28b4c8fa5e020413c2d"}
Jul 07 07:37:35.994117 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:35.994122    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1082ad/rate-test-model-1082ad-kserve-ff6dc744c-8g5nz" event={"ID":"8d644c04-7621-4248-bad3-0b54fe2579f2","Type":"ContainerStarted","Data":"f15069abac371d308baae96c8c280e2c67e13271b1e9ae6353aa2aab106f656d"}
Jul 07 07:37:45.532769 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:45.532722    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-1082ad/rate-test-model-1082ad-kserve-ff6dc744c-8g5nz"
Jul 07 07:37:45.533200 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:45.532784    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-1082ad/rate-test-model-1082ad-kserve-ff6dc744c-8g5nz"
Jul 07 07:37:45.545169 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:45.545143    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-1082ad/rate-test-model-1082ad-kserve-ff6dc744c-8g5nz"
Jul 07 07:37:45.566451 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:45.566399    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-1082ad/rate-test-model-1082ad-kserve-ff6dc744c-8g5nz" podStartSLOduration=10.566384449 podStartE2EDuration="10.566384449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 07:37:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 07:37:36.017631695 +0000 UTC m=+2335.561393175" watchObservedRunningTime="2026-07-07 07:37:45.566384449 +0000 UTC m=+2345.110145980"
Jul 07 07:37:46.049485 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:37:46.049450    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-1082ad/rate-test-model-1082ad-kserve-ff6dc744c-8g5nz"
Jul 07 07:38:22.756597 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:38:22.756559    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-083d4e/rate-test-model-083d4e-kserve-7d9586c6f9-ph66m"]
Jul 07 07:38:23.661233 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:38:23.661196    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-1082ad/rate-test-model-1082ad-kserve-ff6dc744c-8g5nz"]
Jul 07 07:38:37.776165 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:38:37.776122    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-083d4e/rate-test-model-083d4e-kserve-7d9586c6f9-ph66m" podUID="5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee" containerName="main" containerID="cri-o://2792d1f88e1cf12ee390cdf5ab58c81f103101e0faa091498d59e474c23a509f" gracePeriod=45
Jul 07 07:38:38.016569 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:38:38.016541    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-083d4e/rate-test-model-083d4e-kserve-7d9586c6f9-ph66m"
Jul 07 07:38:38.117390 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:38:38.117304    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fc9br\" (UniqueName: \"kubernetes.io/projected/5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee-kube-api-access-fc9br\") pod \"5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee\" (UID: \"5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee\") "
Jul 07 07:38:38.117390 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:38:38.117365    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee-home\") pod \"5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee\" (UID: \"5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee\") "
Jul 07 07:38:38.117750 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:38:38.117427    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee-tmp-dir\") pod \"5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee\" (UID: \"5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee\") "
Jul 07 07:38:38.117750 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:38:38.117454    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee-dshm\") pod \"5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee\" (UID: \"5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee\") "
Jul 07 07:38:38.117750 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:38:38.117472    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee-model-cache\") pod \"5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee\" (UID: \"5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee\") "
Jul 07 07:38:38.117750 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:38:38.117513    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee-tls-certs\") pod \"5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee\" (UID: \"5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee\") "
Jul 07 07:38:38.117750 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:38:38.117638    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee-home" (OuterVolumeSpecName: "home") pod "5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee" (UID: "5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 07:38:38.118006 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:38:38.117754    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee" (UID: "5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 07:38:38.118006 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:38:38.117769    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee-home\") on node \"ip-10-0-138-97.ec2.internal\" DevicePath \"\""
Jul 07 07:38:38.118006 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:38:38.117771    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee-model-cache" (OuterVolumeSpecName: "model-cache") pod "5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee" (UID: "5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 07:38:38.119528 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:38:38.119496    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee-kube-api-access-fc9br" (OuterVolumeSpecName: "kube-api-access-fc9br") pod "5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee" (UID: "5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee"). InnerVolumeSpecName "kube-api-access-fc9br". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 07:38:38.119662 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:38:38.119576    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee-dshm" (OuterVolumeSpecName: "dshm") pod "5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee" (UID: "5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 07:38:38.119662 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:38:38.119587    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee" (UID: "5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 07:38:38.219011 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:38:38.218934    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee-tmp-dir\") on node \"ip-10-0-138-97.ec2.internal\" DevicePath \"\""
Jul 07 07:38:38.219011 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:38:38.219005    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee-dshm\") on node \"ip-10-0-138-97.ec2.internal\" DevicePath \"\""
Jul 07 07:38:38.219011 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:38:38.219016    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee-model-cache\") on node \"ip-10-0-138-97.ec2.internal\" DevicePath \"\""
Jul 07 07:38:38.219011 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:38:38.219025    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee-tls-certs\") on node \"ip-10-0-138-97.ec2.internal\" DevicePath \"\""
Jul 07 07:38:38.219011 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:38:38.219033    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fc9br\" (UniqueName: \"kubernetes.io/projected/5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee-kube-api-access-fc9br\") on node \"ip-10-0-138-97.ec2.internal\" DevicePath \"\""
Jul 07 07:38:38.272442 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:38:38.272409    2579 generic.go:356] "Generic (PLEG): container finished" podID="5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee" containerID="2792d1f88e1cf12ee390cdf5ab58c81f103101e0faa091498d59e474c23a509f" exitCode=0
Jul 07 07:38:38.272606 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:38:38.272455    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-083d4e/rate-test-model-083d4e-kserve-7d9586c6f9-ph66m"
Jul 07 07:38:38.272606 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:38:38.272492    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-083d4e/rate-test-model-083d4e-kserve-7d9586c6f9-ph66m" event={"ID":"5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee","Type":"ContainerDied","Data":"2792d1f88e1cf12ee390cdf5ab58c81f103101e0faa091498d59e474c23a509f"}
Jul 07 07:38:38.272606 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:38:38.272530    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-083d4e/rate-test-model-083d4e-kserve-7d9586c6f9-ph66m" event={"ID":"5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee","Type":"ContainerDied","Data":"6b920b6f8a3a9d906e0be78d6527328efe0ce42531769f7a14ea3c5336bf7e49"}
Jul 07 07:38:38.272606 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:38:38.272547    2579 scope.go:117] "RemoveContainer" containerID="2792d1f88e1cf12ee390cdf5ab58c81f103101e0faa091498d59e474c23a509f"
Jul 07 07:38:38.281456 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:38:38.281439    2579 scope.go:117] "RemoveContainer" containerID="2792d1f88e1cf12ee390cdf5ab58c81f103101e0faa091498d59e474c23a509f"
Jul 07 07:38:38.281706 ip-10-0-138-97 kubenswrapper[2579]: E0707 07:38:38.281687    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2792d1f88e1cf12ee390cdf5ab58c81f103101e0faa091498d59e474c23a509f\": container with ID starting with 2792d1f88e1cf12ee390cdf5ab58c81f103101e0faa091498d59e474c23a509f not found: ID does not exist" containerID="2792d1f88e1cf12ee390cdf5ab58c81f103101e0faa091498d59e474c23a509f"
Jul 07 07:38:38.281748 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:38:38.281714    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2792d1f88e1cf12ee390cdf5ab58c81f103101e0faa091498d59e474c23a509f"} err="failed to get container status \"2792d1f88e1cf12ee390cdf5ab58c81f103101e0faa091498d59e474c23a509f\": rpc error: code = NotFound desc = could not find container \"2792d1f88e1cf12ee390cdf5ab58c81f103101e0faa091498d59e474c23a509f\": container with ID starting with 2792d1f88e1cf12ee390cdf5ab58c81f103101e0faa091498d59e474c23a509f not found: ID does not exist"
Jul 07 07:38:38.300404 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:38:38.300372    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-083d4e/rate-test-model-083d4e-kserve-7d9586c6f9-ph66m"]
Jul 07 07:38:38.310374 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:38:38.310352    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-083d4e/rate-test-model-083d4e-kserve-7d9586c6f9-ph66m"]
Jul 07 07:38:38.680866 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:38:38.680817    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-1082ad/rate-test-model-1082ad-kserve-ff6dc744c-8g5nz" podUID="8d644c04-7621-4248-bad3-0b54fe2579f2" containerName="main" containerID="cri-o://379781c90b634fb41251c3576d8019a92fbfd3a800eef28b4c8fa5e020413c2d" gracePeriod=45
Jul 07 07:38:38.936466 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:38:38.936407    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1082ad/rate-test-model-1082ad-kserve-ff6dc744c-8g5nz"
Jul 07 07:38:38.965415 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:38:38.965379    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee" path="/var/lib/kubelet/pods/5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee/volumes"
Jul 07 07:38:39.026505 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:38:39.026470    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dnwfp\" (UniqueName: \"kubernetes.io/projected/8d644c04-7621-4248-bad3-0b54fe2579f2-kube-api-access-dnwfp\") pod \"8d644c04-7621-4248-bad3-0b54fe2579f2\" (UID: \"8d644c04-7621-4248-bad3-0b54fe2579f2\") "
Jul 07 07:38:39.026505 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:38:39.026511    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d644c04-7621-4248-bad3-0b54fe2579f2-model-cache\") pod \"8d644c04-7621-4248-bad3-0b54fe2579f2\" (UID: \"8d644c04-7621-4248-bad3-0b54fe2579f2\") "
Jul 07 07:38:39.026842 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:38:39.026527    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d644c04-7621-4248-bad3-0b54fe2579f2-dshm\") pod \"8d644c04-7621-4248-bad3-0b54fe2579f2\" (UID: \"8d644c04-7621-4248-bad3-0b54fe2579f2\") "
Jul 07 07:38:39.026842 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:38:39.026586    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d644c04-7621-4248-bad3-0b54fe2579f2-tls-certs\") pod \"8d644c04-7621-4248-bad3-0b54fe2579f2\" (UID: \"8d644c04-7621-4248-bad3-0b54fe2579f2\") "
Jul 07 07:38:39.026842 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:38:39.026605    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d644c04-7621-4248-bad3-0b54fe2579f2-home\") pod \"8d644c04-7621-4248-bad3-0b54fe2579f2\" (UID: \"8d644c04-7621-4248-bad3-0b54fe2579f2\") "
Jul 07 07:38:39.026842 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:38:39.026654    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d644c04-7621-4248-bad3-0b54fe2579f2-tmp-dir\") pod \"8d644c04-7621-4248-bad3-0b54fe2579f2\" (UID: \"8d644c04-7621-4248-bad3-0b54fe2579f2\") "
Jul 07 07:38:39.026842 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:38:39.026814    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d644c04-7621-4248-bad3-0b54fe2579f2-model-cache" (OuterVolumeSpecName: "model-cache") pod "8d644c04-7621-4248-bad3-0b54fe2579f2" (UID: "8d644c04-7621-4248-bad3-0b54fe2579f2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 07:38:39.027147 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:38:39.027066    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d644c04-7621-4248-bad3-0b54fe2579f2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8d644c04-7621-4248-bad3-0b54fe2579f2" (UID: "8d644c04-7621-4248-bad3-0b54fe2579f2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 07:38:39.027203 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:38:39.027161    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d644c04-7621-4248-bad3-0b54fe2579f2-home" (OuterVolumeSpecName: "home") pod "8d644c04-7621-4248-bad3-0b54fe2579f2" (UID: "8d644c04-7621-4248-bad3-0b54fe2579f2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 07:38:39.028121 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:38:39.027839    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d644c04-7621-4248-bad3-0b54fe2579f2-model-cache\") on node \"ip-10-0-138-97.ec2.internal\" DevicePath \"\""
Jul 07 07:38:39.028121 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:38:39.027865    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d644c04-7621-4248-bad3-0b54fe2579f2-home\") on node \"ip-10-0-138-97.ec2.internal\" DevicePath \"\""
Jul 07 07:38:39.028121 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:38:39.027881    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d644c04-7621-4248-bad3-0b54fe2579f2-tmp-dir\") on node \"ip-10-0-138-97.ec2.internal\" DevicePath \"\""
Jul 07 07:38:39.029075 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:38:39.029052    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d644c04-7621-4248-bad3-0b54fe2579f2-dshm" (OuterVolumeSpecName: "dshm") pod "8d644c04-7621-4248-bad3-0b54fe2579f2" (UID: "8d644c04-7621-4248-bad3-0b54fe2579f2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 07:38:39.029289 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:38:39.029263    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d644c04-7621-4248-bad3-0b54fe2579f2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8d644c04-7621-4248-bad3-0b54fe2579f2" (UID: "8d644c04-7621-4248-bad3-0b54fe2579f2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 07:38:39.029480 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:38:39.029460    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d644c04-7621-4248-bad3-0b54fe2579f2-kube-api-access-dnwfp" (OuterVolumeSpecName: "kube-api-access-dnwfp") pod "8d644c04-7621-4248-bad3-0b54fe2579f2" (UID: "8d644c04-7621-4248-bad3-0b54fe2579f2"). InnerVolumeSpecName "kube-api-access-dnwfp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 07:38:39.128716 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:38:39.128679    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnwfp\" (UniqueName: \"kubernetes.io/projected/8d644c04-7621-4248-bad3-0b54fe2579f2-kube-api-access-dnwfp\") on node \"ip-10-0-138-97.ec2.internal\" DevicePath \"\""
Jul 07 07:38:39.128716 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:38:39.128711    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d644c04-7621-4248-bad3-0b54fe2579f2-dshm\") on node \"ip-10-0-138-97.ec2.internal\" DevicePath \"\""
Jul 07 07:38:39.128716 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:38:39.128724    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d644c04-7621-4248-bad3-0b54fe2579f2-tls-certs\") on node \"ip-10-0-138-97.ec2.internal\" DevicePath \"\""
Jul 07 07:38:39.279504 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:38:39.279470    2579 generic.go:356] "Generic (PLEG): container finished" podID="8d644c04-7621-4248-bad3-0b54fe2579f2" containerID="379781c90b634fb41251c3576d8019a92fbfd3a800eef28b4c8fa5e020413c2d" exitCode=0
Jul 07 07:38:39.279713 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:38:39.279506    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1082ad/rate-test-model-1082ad-kserve-ff6dc744c-8g5nz" event={"ID":"8d644c04-7621-4248-bad3-0b54fe2579f2","Type":"ContainerDied","Data":"379781c90b634fb41251c3576d8019a92fbfd3a800eef28b4c8fa5e020413c2d"}
Jul 07 07:38:39.279713 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:38:39.279534    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1082ad/rate-test-model-1082ad-kserve-ff6dc744c-8g5nz"
Jul 07 07:38:39.279713 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:38:39.279550    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1082ad/rate-test-model-1082ad-kserve-ff6dc744c-8g5nz" event={"ID":"8d644c04-7621-4248-bad3-0b54fe2579f2","Type":"ContainerDied","Data":"f15069abac371d308baae96c8c280e2c67e13271b1e9ae6353aa2aab106f656d"}
Jul 07 07:38:39.279713 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:38:39.279573    2579 scope.go:117] "RemoveContainer" containerID="379781c90b634fb41251c3576d8019a92fbfd3a800eef28b4c8fa5e020413c2d"
Jul 07 07:38:39.288368 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:38:39.288353    2579 scope.go:117] "RemoveContainer" containerID="379781c90b634fb41251c3576d8019a92fbfd3a800eef28b4c8fa5e020413c2d"
Jul 07 07:38:39.288601 ip-10-0-138-97 kubenswrapper[2579]: E0707 07:38:39.288582    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"379781c90b634fb41251c3576d8019a92fbfd3a800eef28b4c8fa5e020413c2d\": container with ID starting with 379781c90b634fb41251c3576d8019a92fbfd3a800eef28b4c8fa5e020413c2d not found: ID does not exist" containerID="379781c90b634fb41251c3576d8019a92fbfd3a800eef28b4c8fa5e020413c2d"
Jul 07 07:38:39.288663 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:38:39.288613    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"379781c90b634fb41251c3576d8019a92fbfd3a800eef28b4c8fa5e020413c2d"} err="failed to get container status \"379781c90b634fb41251c3576d8019a92fbfd3a800eef28b4c8fa5e020413c2d\": rpc error: code = NotFound desc = could not find container \"379781c90b634fb41251c3576d8019a92fbfd3a800eef28b4c8fa5e020413c2d\": container with ID starting with 379781c90b634fb41251c3576d8019a92fbfd3a800eef28b4c8fa5e020413c2d not found: ID does not exist"
Jul 07 07:38:39.312609 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:38:39.312581    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-1082ad/rate-test-model-1082ad-kserve-ff6dc744c-8g5nz"]
Jul 07 07:38:39.318386 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:38:39.318357    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-1082ad/rate-test-model-1082ad-kserve-ff6dc744c-8g5nz"]
Jul 07 07:38:40.964248 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:38:40.964214    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d644c04-7621-4248-bad3-0b54fe2579f2" path="/var/lib/kubelet/pods/8d644c04-7621-4248-bad3-0b54fe2579f2/volumes"
Jul 07 07:39:19.392713 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:39:19.392669    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-68575fcd4c-j27b8_6a4ff039-bad1-4d23-8dc8-b8b7844b3034/authorino/0.log"
Jul 07 07:39:25.236288 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:39:25.236253    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29723490-7crxf_87666c82-44bb-4505-a97f-870202bd331e/cleanup/0.log"
Jul 07 07:39:25.945441 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:39:25.945407    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-dbf9cc589-qfksk_50a43470-6929-4b81-a4c8-fa225d250335/copy-manifests/0.log"
Jul 07 07:39:25.968700 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:39:25.968675    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-dbf9cc589-qfksk_50a43470-6929-4b81-a4c8-fa225d250335/manager/0.log"
Jul 07 07:39:27.751208 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:39:27.751175    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-68575fcd4c-j27b8_6a4ff039-bad1-4d23-8dc8-b8b7844b3034/authorino/0.log"
Jul 07 07:39:28.635708 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:39:28.635673    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-dmq79_d8b2175d-87fe-4164-bdec-c36e4923701b/manager/0.log"
Jul 07 07:39:29.232571 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:39:29.232538    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54bd8c98d7-jw7hc_9e42bb72-5874-40c2-b79d-a24298bb8586/kube-auth-proxy/0.log"
Jul 07 07:39:30.350193 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:39:30.350156    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-ct89n_7368ec91-3412-4c08-b895-955fafebd42f/main/0.log"
Jul 07 07:39:30.472914 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:39:30.472882    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-s265r_7e5077c3-eff4-4f30-84dc-14614e08cca2/main/0.log"
Jul 07 07:39:30.844646 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:39:30.844619    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-kjkht_2db964dd-d624-4187-8126-8705719cfe18/main/0.log"
Jul 07 07:39:39.182117 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:39:39.182086    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-gnlks_575f1248-fecc-48d9-9c6a-db334596c8b6/global-pull-secret-syncer/0.log"
Jul 07 07:39:39.430118 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:39:39.430085    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-z246s_338d4939-ce17-426b-a06b-bb8f40ebe2a1/konnectivity-agent/0.log"
Jul 07 07:39:39.646210 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:39:39.646176    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-97.ec2.internal_a566f4272b430543cc74511348672fa6/haproxy/0.log"
Jul 07 07:39:44.318541 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:39:44.318433    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-68575fcd4c-j27b8_6a4ff039-bad1-4d23-8dc8-b8b7844b3034/authorino/0.log"
Jul 07 07:39:44.873096 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:39:44.873058    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-dmq79_d8b2175d-87fe-4164-bdec-c36e4923701b/manager/0.log"
Jul 07 07:39:47.213968 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:39:47.213930    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bc9lh_2f31a127-490d-41ff-a0ab-f24c4aec47f3/node-exporter/0.log"
Jul 07 07:39:47.256883 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:39:47.256859    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bc9lh_2f31a127-490d-41ff-a0ab-f24c4aec47f3/kube-rbac-proxy/0.log"
Jul 07 07:39:47.295225 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:39:47.295201    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bc9lh_2f31a127-490d-41ff-a0ab-f24c4aec47f3/init-textfile/0.log"
Jul 07 07:39:47.703278 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:39:47.703248    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_560793a0-f73a-4192-9898-82202b318714/prometheus/0.log"
Jul 07 07:39:47.736075 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:39:47.736042    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_560793a0-f73a-4192-9898-82202b318714/config-reloader/0.log"
Jul 07 07:39:47.766302 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:39:47.766258    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_560793a0-f73a-4192-9898-82202b318714/thanos-sidecar/0.log"
Jul 07 07:39:47.796756 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:39:47.796732    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_560793a0-f73a-4192-9898-82202b318714/kube-rbac-proxy-web/0.log"
Jul 07 07:39:47.836937 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:39:47.836909    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_560793a0-f73a-4192-9898-82202b318714/kube-rbac-proxy/0.log"
Jul 07 07:39:47.875088 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:39:47.875063    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_560793a0-f73a-4192-9898-82202b318714/kube-rbac-proxy-thanos/0.log"
Jul 07 07:39:47.903763 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:39:47.903736    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_560793a0-f73a-4192-9898-82202b318714/init-config-reloader/0.log"
Jul 07 07:39:47.943341 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:39:47.943310    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-szt4k_e540ce11-a18b-4631-9d0f-0727a72c0a23/prometheus-operator/0.log"
Jul 07 07:39:47.970886 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:39:47.970797    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-szt4k_e540ce11-a18b-4631-9d0f-0727a72c0a23/kube-rbac-proxy/0.log"
Jul 07 07:39:48.503348 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:39:48.503315    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zwr9g/perf-node-gather-daemonset-rpqc9"]
Jul 07 07:39:48.503760 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:39:48.503670    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee" containerName="main"
Jul 07 07:39:48.503760 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:39:48.503681    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee" containerName="main"
Jul 07 07:39:48.503760 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:39:48.503700    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d644c04-7621-4248-bad3-0b54fe2579f2" containerName="main"
Jul 07 07:39:48.503760 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:39:48.503707    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d644c04-7621-4248-bad3-0b54fe2579f2" containerName="main"
Jul 07 07:39:48.503760 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:39:48.503759    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d644c04-7621-4248-bad3-0b54fe2579f2" containerName="main"
Jul 07 07:39:48.503921 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:39:48.503769    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ef93b4e-ab1b-4f75-ab4d-246a0cb1eaee" containerName="main"
Jul 07 07:39:48.506671 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:39:48.506654    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zwr9g/perf-node-gather-daemonset-rpqc9"
Jul 07 07:39:48.509418 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:39:48.509398    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zwr9g\"/\"openshift-service-ca.crt\""
Jul 07 07:39:48.509541 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:39:48.509496    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zwr9g\"/\"kube-root-ca.crt\""
Jul 07 07:39:48.510460 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:39:48.510440    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zwr9g\"/\"default-dockercfg-r2qnp\""
Jul 07 07:39:48.516915 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:39:48.516896    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zwr9g/perf-node-gather-daemonset-rpqc9"]
Jul 07 07:39:48.643866 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:39:48.643832    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d8c7e0bd-5a22-4d7f-9db1-61bd55923d9c-podres\") pod \"perf-node-gather-daemonset-rpqc9\" (UID: \"d8c7e0bd-5a22-4d7f-9db1-61bd55923d9c\") " pod="openshift-must-gather-zwr9g/perf-node-gather-daemonset-rpqc9"
Jul 07 07:39:48.644055 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:39:48.643874    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d8c7e0bd-5a22-4d7f-9db1-61bd55923d9c-lib-modules\") pod \"perf-node-gather-daemonset-rpqc9\" (UID: \"d8c7e0bd-5a22-4d7f-9db1-61bd55923d9c\") " pod="openshift-must-gather-zwr9g/perf-node-gather-daemonset-rpqc9"
Jul 07 07:39:48.644055 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:39:48.643983    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d8c7e0bd-5a22-4d7f-9db1-61bd55923d9c-proc\") pod \"perf-node-gather-daemonset-rpqc9\" (UID: \"d8c7e0bd-5a22-4d7f-9db1-61bd55923d9c\") " pod="openshift-must-gather-zwr9g/perf-node-gather-daemonset-rpqc9"
Jul 07 07:39:48.644055 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:39:48.644026    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk4zd\" (UniqueName: \"kubernetes.io/projected/d8c7e0bd-5a22-4d7f-9db1-61bd55923d9c-kube-api-access-tk4zd\") pod \"perf-node-gather-daemonset-rpqc9\" (UID: \"d8c7e0bd-5a22-4d7f-9db1-61bd55923d9c\") " pod="openshift-must-gather-zwr9g/perf-node-gather-daemonset-rpqc9"
Jul 07 07:39:48.644171 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:39:48.644135    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d8c7e0bd-5a22-4d7f-9db1-61bd55923d9c-sys\") pod \"perf-node-gather-daemonset-rpqc9\" (UID: \"d8c7e0bd-5a22-4d7f-9db1-61bd55923d9c\") " pod="openshift-must-gather-zwr9g/perf-node-gather-daemonset-rpqc9"
Jul 07 07:39:48.744982 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:39:48.744921    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d8c7e0bd-5a22-4d7f-9db1-61bd55923d9c-sys\") pod \"perf-node-gather-daemonset-rpqc9\" (UID: \"d8c7e0bd-5a22-4d7f-9db1-61bd55923d9c\") " pod="openshift-must-gather-zwr9g/perf-node-gather-daemonset-rpqc9"
Jul 07 07:39:48.744982 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:39:48.744989    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d8c7e0bd-5a22-4d7f-9db1-61bd55923d9c-podres\") pod \"perf-node-gather-daemonset-rpqc9\" (UID: \"d8c7e0bd-5a22-4d7f-9db1-61bd55923d9c\") " pod="openshift-must-gather-zwr9g/perf-node-gather-daemonset-rpqc9"
Jul 07 07:39:48.745241 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:39:48.745011    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d8c7e0bd-5a22-4d7f-9db1-61bd55923d9c-lib-modules\") pod \"perf-node-gather-daemonset-rpqc9\" (UID: \"d8c7e0bd-5a22-4d7f-9db1-61bd55923d9c\") " pod="openshift-must-gather-zwr9g/perf-node-gather-daemonset-rpqc9"
Jul 07 07:39:48.745241 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:39:48.745040    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d8c7e0bd-5a22-4d7f-9db1-61bd55923d9c-proc\") pod \"perf-node-gather-daemonset-rpqc9\" (UID: \"d8c7e0bd-5a22-4d7f-9db1-61bd55923d9c\") " pod="openshift-must-gather-zwr9g/perf-node-gather-daemonset-rpqc9"
Jul 07 07:39:48.745241 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:39:48.745054    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d8c7e0bd-5a22-4d7f-9db1-61bd55923d9c-sys\") pod \"perf-node-gather-daemonset-rpqc9\" (UID: \"d8c7e0bd-5a22-4d7f-9db1-61bd55923d9c\") " pod="openshift-must-gather-zwr9g/perf-node-gather-daemonset-rpqc9"
Jul 07 07:39:48.745241 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:39:48.745063    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk4zd\" (UniqueName: \"kubernetes.io/projected/d8c7e0bd-5a22-4d7f-9db1-61bd55923d9c-kube-api-access-tk4zd\") pod \"perf-node-gather-daemonset-rpqc9\" (UID: \"d8c7e0bd-5a22-4d7f-9db1-61bd55923d9c\") " pod="openshift-must-gather-zwr9g/perf-node-gather-daemonset-rpqc9"
Jul 07 07:39:48.745241 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:39:48.745147    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d8c7e0bd-5a22-4d7f-9db1-61bd55923d9c-proc\") pod \"perf-node-gather-daemonset-rpqc9\" (UID: \"d8c7e0bd-5a22-4d7f-9db1-61bd55923d9c\") " pod="openshift-must-gather-zwr9g/perf-node-gather-daemonset-rpqc9"
Jul 07 07:39:48.745241 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:39:48.745183    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d8c7e0bd-5a22-4d7f-9db1-61bd55923d9c-lib-modules\") pod \"perf-node-gather-daemonset-rpqc9\" (UID: \"d8c7e0bd-5a22-4d7f-9db1-61bd55923d9c\") " pod="openshift-must-gather-zwr9g/perf-node-gather-daemonset-rpqc9"
Jul 07 07:39:48.745241 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:39:48.745186    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d8c7e0bd-5a22-4d7f-9db1-61bd55923d9c-podres\") pod \"perf-node-gather-daemonset-rpqc9\" (UID: \"d8c7e0bd-5a22-4d7f-9db1-61bd55923d9c\") " pod="openshift-must-gather-zwr9g/perf-node-gather-daemonset-rpqc9"
Jul 07 07:39:48.757400 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:39:48.757340    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk4zd\" (UniqueName: \"kubernetes.io/projected/d8c7e0bd-5a22-4d7f-9db1-61bd55923d9c-kube-api-access-tk4zd\") pod \"perf-node-gather-daemonset-rpqc9\" (UID: \"d8c7e0bd-5a22-4d7f-9db1-61bd55923d9c\") " pod="openshift-must-gather-zwr9g/perf-node-gather-daemonset-rpqc9"
Jul 07 07:39:48.817279 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:39:48.817246    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zwr9g/perf-node-gather-daemonset-rpqc9"
Jul 07 07:39:48.963726 ip-10-0-138-97 kubenswrapper[2579]: W0707 07:39:48.963695    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd8c7e0bd_5a22_4d7f_9db1_61bd55923d9c.slice/crio-3ea3cfa66379feaa7f0da2032743fa4d7708da05117764edd91902a5955180e9 WatchSource:0}: Error finding container 3ea3cfa66379feaa7f0da2032743fa4d7708da05117764edd91902a5955180e9: Status 404 returned error can't find the container with id 3ea3cfa66379feaa7f0da2032743fa4d7708da05117764edd91902a5955180e9
Jul 07 07:39:48.964577 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:39:48.964554    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zwr9g/perf-node-gather-daemonset-rpqc9"]
Jul 07 07:39:49.599568 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:39:49.599526    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zwr9g/perf-node-gather-daemonset-rpqc9" event={"ID":"d8c7e0bd-5a22-4d7f-9db1-61bd55923d9c","Type":"ContainerStarted","Data":"6b7b3970039c7c9c4d94ac1fbb25d6f832d27bf141ae2f303c6e4df5008cb0b4"}
Jul 07 07:39:49.600045 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:39:49.599589    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zwr9g/perf-node-gather-daemonset-rpqc9" event={"ID":"d8c7e0bd-5a22-4d7f-9db1-61bd55923d9c","Type":"ContainerStarted","Data":"3ea3cfa66379feaa7f0da2032743fa4d7708da05117764edd91902a5955180e9"}
Jul 07 07:39:49.600045 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:39:49.599732    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-zwr9g/perf-node-gather-daemonset-rpqc9"
Jul 07 07:39:49.630707 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:39:49.630658    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-zwr9g/perf-node-gather-daemonset-rpqc9" podStartSLOduration=1.630644104 podStartE2EDuration="1.630644104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 07:39:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 07:39:49.629274092 +0000 UTC m=+2469.173035571" watchObservedRunningTime="2026-07-07 07:39:49.630644104 +0000 UTC m=+2469.174405584"
Jul 07 07:39:50.741401 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:39:50.741365    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6dcfdb8b9-272nf_1894f384-040c-4d22-b4fd-a294b9cca8fb/console/0.log"
Jul 07 07:39:52.872555 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:39:52.872529    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hpwxr_4fb6f6e3-b570-408c-aaaa-a77d5bea0b24/dns/0.log"
Jul 07 07:39:52.904750 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:39:52.904723    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hpwxr_4fb6f6e3-b570-408c-aaaa-a77d5bea0b24/kube-rbac-proxy/0.log"
Jul 07 07:39:53.084991 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:39:53.084969    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-r77h8_86018993-6534-4639-9254-a898f62948fe/dns-node-resolver/0.log"
Jul 07 07:39:53.802203 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:39:53.802175    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-fzzb4_7c1bef18-765e-4290-8c9c-80c985fcb304/node-ca/0.log"
Jul 07 07:39:55.108027 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:39:55.107985    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54bd8c98d7-jw7hc_9e42bb72-5874-40c2-b79d-a24298bb8586/kube-auth-proxy/0.log"
Jul 07 07:39:55.614348 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:39:55.614302    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-zwr9g/perf-node-gather-daemonset-rpqc9"
Jul 07 07:39:56.312684 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:39:56.312658    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jhkfq_2915b9bd-a9ae-4d0f-a362-43b2ce659350/serve-healthcheck-canary/0.log"
Jul 07 07:39:57.035628 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:39:57.035596    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-brh4b_a852d8bb-79ad-4ead-8002-8e51919be749/kube-rbac-proxy/0.log"
Jul 07 07:39:57.067755 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:39:57.067728    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-brh4b_a852d8bb-79ad-4ead-8002-8e51919be749/exporter/0.log"
Jul 07 07:39:57.097224 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:39:57.097193    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-brh4b_a852d8bb-79ad-4ead-8002-8e51919be749/extractor/0.log"
Jul 07 07:39:59.132015 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:39:59.131915    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54bd8c98d7-jw7hc_9e42bb72-5874-40c2-b79d-a24298bb8586/kube-auth-proxy/0.log"
Jul 07 07:40:00.154078 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:40:00.154043    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29723490-7crxf_87666c82-44bb-4505-a97f-870202bd331e/cleanup/0.log"
Jul 07 07:40:00.446168 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:40:00.446135    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-dbf9cc589-qfksk_50a43470-6929-4b81-a4c8-fa225d250335/manager/0.log"
Jul 07 07:40:00.472442 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:40:00.472414    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-dbf9cc589-qfksk_50a43470-6929-4b81-a4c8-fa225d250335/copy-manifests/0.log"
Jul 07 07:40:00.680733 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:40:00.680648    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54bd8c98d7-jw7hc_9e42bb72-5874-40c2-b79d-a24298bb8586/kube-auth-proxy/0.log"
Jul 07 07:40:01.987162 ip-10-0-138-97 kubenswrapper[2579]: I0707 07:40:01.987132    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5c6f759dd-gwbjb_31a943d1-8ef0-4f15-9e3a-ba696db451b7/manager/0.log"