Jul 08 12:01:50.156142 ip-10-0-138-89 systemd[1]: Starting Kubernetes Kubelet...
Jul 08 12:01:50.530205 ip-10-0-138-89 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 08 12:01:50.530205 ip-10-0-138-89 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 08 12:01:50.530205 ip-10-0-138-89 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 08 12:01:50.530205 ip-10-0-138-89 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 08 12:01:50.530205 ip-10-0-138-89 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 08 12:01:50.530934 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.530260    2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 08 12:01:50.537077 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537056    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 08 12:01:50.537077 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537077    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 08 12:01:50.537150 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537080    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 08 12:01:50.537150 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537084    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 08 12:01:50.537150 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537087    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 08 12:01:50.537150 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537089    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 08 12:01:50.537150 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537092    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 08 12:01:50.537150 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537095    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 08 12:01:50.537150 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537097    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 08 12:01:50.537150 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537100    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 08 12:01:50.537150 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537102    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 08 12:01:50.537150 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537105    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 08 12:01:50.537150 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537109    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 08 12:01:50.537150 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537111    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 08 12:01:50.537150 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537114    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 08 12:01:50.537150 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537118    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 08 12:01:50.537150 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537122    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 08 12:01:50.537150 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537125    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 08 12:01:50.537150 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537128    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 08 12:01:50.537150 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537131    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 08 12:01:50.537150 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537135    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 08 12:01:50.537150 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537139    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 08 12:01:50.537622 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537142    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 08 12:01:50.537622 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537144    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 08 12:01:50.537622 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537147    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 08 12:01:50.537622 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537149    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 08 12:01:50.537622 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537152    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 08 12:01:50.537622 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537154    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 08 12:01:50.537622 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537157    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 08 12:01:50.537622 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537159    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 08 12:01:50.537622 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537162    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 08 12:01:50.537622 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537164    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 08 12:01:50.537622 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537167    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 08 12:01:50.537622 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537169    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 08 12:01:50.537622 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537172    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 08 12:01:50.537622 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537174    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 08 12:01:50.537622 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537176    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 08 12:01:50.537622 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537179    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 08 12:01:50.537622 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537181    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 08 12:01:50.537622 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537183    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 08 12:01:50.537622 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537186    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 08 12:01:50.537622 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537188    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 08 12:01:50.538215 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537191    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 08 12:01:50.538215 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537193    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 08 12:01:50.538215 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537195    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 08 12:01:50.538215 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537198    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 08 12:01:50.538215 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537200    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 08 12:01:50.538215 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537202    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 08 12:01:50.538215 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537205    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 08 12:01:50.538215 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537207    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 08 12:01:50.538215 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537209    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 08 12:01:50.538215 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537211    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 08 12:01:50.538215 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537214    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 08 12:01:50.538215 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537216    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 08 12:01:50.538215 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537219    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 08 12:01:50.538215 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537224    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 08 12:01:50.538215 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537226    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 08 12:01:50.538215 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537229    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 08 12:01:50.538215 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537231    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 08 12:01:50.538215 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537234    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 08 12:01:50.538215 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537236    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 08 12:01:50.538215 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537238    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 08 12:01:50.538713 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537240    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 08 12:01:50.538713 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537243    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 08 12:01:50.538713 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537245    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 08 12:01:50.538713 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537248    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 08 12:01:50.538713 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537250    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 08 12:01:50.538713 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537254    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 08 12:01:50.538713 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537257    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 08 12:01:50.538713 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537259    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 08 12:01:50.538713 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537262    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 08 12:01:50.538713 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537264    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 08 12:01:50.538713 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537267    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 08 12:01:50.538713 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537269    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 08 12:01:50.538713 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537271    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 08 12:01:50.538713 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537274    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 08 12:01:50.538713 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537276    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 08 12:01:50.538713 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537278    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 08 12:01:50.538713 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537280    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 08 12:01:50.538713 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537285    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 08 12:01:50.538713 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537288    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 08 12:01:50.539180 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537291    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 08 12:01:50.539180 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537294    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 08 12:01:50.539180 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537297    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 08 12:01:50.539180 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537299    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 08 12:01:50.539180 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537302    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 08 12:01:50.539180 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537304    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 08 12:01:50.539180 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537308    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 08 12:01:50.539180 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537310    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 08 12:01:50.539180 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537313    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 08 12:01:50.539180 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537315    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 08 12:01:50.539180 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537317    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 08 12:01:50.539180 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537320    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 08 12:01:50.539180 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537322    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 08 12:01:50.539180 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537325    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 08 12:01:50.539180 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537327    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 08 12:01:50.539180 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537330    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 08 12:01:50.539180 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537332    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 08 12:01:50.539180 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.537335    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 08 12:01:50.539180 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538664    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 08 12:01:50.539657 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538672    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 08 12:01:50.539657 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538675    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 08 12:01:50.539657 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538679    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 08 12:01:50.539657 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538681    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 08 12:01:50.539657 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538684    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 08 12:01:50.539657 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538687    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 08 12:01:50.539657 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538690    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 08 12:01:50.539657 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538693    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 08 12:01:50.539657 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538695    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 08 12:01:50.539657 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538698    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 08 12:01:50.539657 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538701    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 08 12:01:50.539657 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538705    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 08 12:01:50.539657 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538707    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 08 12:01:50.539657 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538710    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 08 12:01:50.539657 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538712    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 08 12:01:50.539657 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538715    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 08 12:01:50.539657 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538717    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 08 12:01:50.539657 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538720    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 08 12:01:50.539657 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538723    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 08 12:01:50.539657 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538726    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 08 12:01:50.540162 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538729    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 08 12:01:50.540162 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538732    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 08 12:01:50.540162 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538735    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 08 12:01:50.540162 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538737    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 08 12:01:50.540162 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538739    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 08 12:01:50.540162 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538742    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 08 12:01:50.540162 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538744    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 08 12:01:50.540162 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538747    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 08 12:01:50.540162 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538749    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 08 12:01:50.540162 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538752    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 08 12:01:50.540162 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538754    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 08 12:01:50.540162 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538756    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 08 12:01:50.540162 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538759    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 08 12:01:50.540162 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538761    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 08 12:01:50.540162 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538764    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 08 12:01:50.540162 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538782    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 08 12:01:50.540162 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538784    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 08 12:01:50.540162 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538789    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 08 12:01:50.540162 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538793    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 08 12:01:50.540938 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538795    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 08 12:01:50.540938 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538799    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 08 12:01:50.540938 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538802    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 08 12:01:50.540938 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538805    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 08 12:01:50.540938 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538808    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 08 12:01:50.540938 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538810    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 08 12:01:50.540938 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538813    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 08 12:01:50.540938 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538816    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 08 12:01:50.540938 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538819    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 08 12:01:50.540938 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538821    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 08 12:01:50.540938 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538824    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 08 12:01:50.540938 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538826    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 08 12:01:50.540938 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538829    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 08 12:01:50.540938 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538831    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 08 12:01:50.540938 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538834    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 08 12:01:50.540938 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538837    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 08 12:01:50.540938 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538839    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 08 12:01:50.540938 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538841    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 08 12:01:50.540938 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538844    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 08 12:01:50.541660 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538846    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 08 12:01:50.541660 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538849    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 08 12:01:50.541660 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538851    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 08 12:01:50.541660 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538854    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 08 12:01:50.541660 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538856    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 08 12:01:50.541660 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538858    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 08 12:01:50.541660 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538861    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 08 12:01:50.541660 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538863    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 08 12:01:50.541660 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538866    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 08 12:01:50.541660 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538868    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 08 12:01:50.541660 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538870    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 08 12:01:50.541660 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538873    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 08 12:01:50.541660 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538875    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 08 12:01:50.541660 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538878    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 08 12:01:50.541660 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538880    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 08 12:01:50.541660 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538882    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 08 12:01:50.541660 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538885    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 08 12:01:50.541660 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538887    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 08 12:01:50.541660 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538891    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 08 12:01:50.541660 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538893    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 08 12:01:50.542218 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538896    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 08 12:01:50.542218 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538898    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 08 12:01:50.542218 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538901    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 08 12:01:50.542218 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538917    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 08 12:01:50.542218 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538921    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 08 12:01:50.542218 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538924    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 08 12:01:50.542218 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538927    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 08 12:01:50.542218 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538930    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 08 12:01:50.542218 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538934    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 08 12:01:50.542218 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538936    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 08 12:01:50.542218 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538939    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 08 12:01:50.542218 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538942    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 08 12:01:50.542218 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538946    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 08 12:01:50.542218 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538950    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 08 12:01:50.542218 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538953    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 08 12:01:50.542218 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538956    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 08 12:01:50.542218 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538959    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 08 12:01:50.542218 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538962    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 08 12:01:50.542218 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538965    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 08 12:01:50.542663 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.538968    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 08 12:01:50.542663 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539351    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 08 12:01:50.542663 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539356    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 08 12:01:50.542663 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539359    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 08 12:01:50.542663 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539363    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 08 12:01:50.542663 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539367    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 08 12:01:50.542663 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539370    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 08 12:01:50.542663 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539381    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 08 12:01:50.542663 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539384    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 08 12:01:50.542663 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539387    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 08 12:01:50.542663 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539390    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 08 12:01:50.542663 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539392    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 08 12:01:50.542663 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539395    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 08 12:01:50.542663 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539398    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 08 12:01:50.542663 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539401    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 08 12:01:50.542663 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539403    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 08 12:01:50.542663 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539406    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 08 12:01:50.542663 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539408    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 08 12:01:50.542663 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539411    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 08 12:01:50.543127 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539414    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 08 12:01:50.543127 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539416    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 08 12:01:50.543127 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539419    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 08 12:01:50.543127 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539422    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 08 12:01:50.543127 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539424    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 08 12:01:50.543127 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539427    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 08 12:01:50.543127 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539430    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 08 12:01:50.543127 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539432    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 08 12:01:50.543127 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539435    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 08 12:01:50.543127 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539437    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 08 12:01:50.543127 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539439    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 08 12:01:50.543127 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539441    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 08 12:01:50.543127 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539443    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 08 12:01:50.543127 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539446    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 08 12:01:50.543127 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539448    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 08 12:01:50.543127 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539450    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 08 12:01:50.543127 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539453    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 08 12:01:50.543127 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539455    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 08 12:01:50.543127 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539457    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 08 12:01:50.543127 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539459    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 08 12:01:50.543591 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539461    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 08 12:01:50.543591 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539464    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 08 12:01:50.543591 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539466    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 08 12:01:50.543591 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539469    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 08 12:01:50.543591 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539471    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 08 12:01:50.543591 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539474    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 08 12:01:50.543591 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539477    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 08 12:01:50.543591 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539479    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 08 12:01:50.543591 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539481    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 08 12:01:50.543591 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539484    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 08 12:01:50.543591 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539486    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 08 12:01:50.543591 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539488    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 08 12:01:50.543591 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539490    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 08 12:01:50.543591 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539493    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 08 12:01:50.543591 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539495    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 08 12:01:50.543591 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539500    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 08 12:01:50.543591 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539502    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 08 12:01:50.543591 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539505    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 08 12:01:50.543591 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539507    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 08 12:01:50.543591 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539510    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 08 12:01:50.544088 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539512    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 08 12:01:50.544088 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539515    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 08 12:01:50.544088 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539517    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 08 12:01:50.544088 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539519    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 08 12:01:50.544088 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539521    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 08 12:01:50.544088 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539525    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 08 12:01:50.544088 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539528    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 08 12:01:50.544088 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539532    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 08 12:01:50.544088 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539534    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 08 12:01:50.544088 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539536    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 08 12:01:50.544088 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539540    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 08 12:01:50.544088 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539543    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 08 12:01:50.544088 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539545    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 08 12:01:50.544088 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539548    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 08 12:01:50.544088 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539550    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 08 12:01:50.544088 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539553    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 08 12:01:50.544088 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539557    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 08 12:01:50.544088 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539559    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 08 12:01:50.544088 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539562    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 08 12:01:50.544516 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539565    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 08 12:01:50.544516 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539567    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 08 12:01:50.544516 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539570    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 08 12:01:50.544516 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539572    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 08 12:01:50.544516 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539574    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 08 12:01:50.544516 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539577    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 08 12:01:50.544516 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539579    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 08 12:01:50.544516 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539581    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 08 12:01:50.544516 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539584    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 08 12:01:50.544516 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539587    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 08 12:01:50.544516 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539589    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 08 12:01:50.544516 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539591    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 08 12:01:50.544516 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539594    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 08 12:01:50.544516 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539596    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 08 12:01:50.544516 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539598    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 08 12:01:50.544516 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539600    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 08 12:01:50.544516 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539603    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 08 12:01:50.544516 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539605    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 08 12:01:50.544516 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539607    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 08 12:01:50.544516 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539612    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 08 12:01:50.545006 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539616    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 08 12:01:50.545006 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.539618    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 08 12:01:50.545006 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539692    2579 flags.go:64] FLAG: --address="0.0.0.0"
Jul 08 12:01:50.545006 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539700    2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 08 12:01:50.545006 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539706    2579 flags.go:64] FLAG: --anonymous-auth="true"
Jul 08 12:01:50.545006 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539710    2579 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 08 12:01:50.545006 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539714    2579 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 08 12:01:50.545006 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539717    2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 08 12:01:50.545006 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539722    2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 08 12:01:50.545006 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539726    2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 08 12:01:50.545006 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539729    2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 08 12:01:50.545006 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539731    2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 08 12:01:50.545006 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539734    2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 08 12:01:50.545006 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539738    2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 08 12:01:50.545006 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539740    2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 08 12:01:50.545006 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539743    2579 flags.go:64] FLAG: --cgroup-root=""
Jul 08 12:01:50.545006 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539746    2579 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 08 12:01:50.545006 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539749    2579 flags.go:64] FLAG: --client-ca-file=""
Jul 08 12:01:50.545006 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539752    2579 flags.go:64] FLAG: --cloud-provider="external"
Jul 08 12:01:50.545006 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539754    2579 flags.go:64] FLAG: --cluster-dns="[]"
Jul 08 12:01:50.545006 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539758    2579 flags.go:64] FLAG: --cluster-domain=""
Jul 08 12:01:50.545006 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539761    2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 08 12:01:50.545006 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539764    2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 08 12:01:50.545006 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539784    2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 08 12:01:50.545566 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539788    2579 flags.go:64] FLAG: --container-log-max-files="5"
Jul 08 12:01:50.545566 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539791    2579 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 08 12:01:50.545566 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539794    2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 08 12:01:50.545566 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539797    2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 08 12:01:50.545566 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539809    2579 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 08 12:01:50.545566 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539813    2579 flags.go:64] FLAG: --contention-profiling="false"
Jul 08 12:01:50.545566 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539816    2579 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 08 12:01:50.545566 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539821    2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 08 12:01:50.545566 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539827    2579 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 08 12:01:50.545566 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539830    2579 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 08 12:01:50.545566 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539834    2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 08 12:01:50.545566 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539837    2579 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 08 12:01:50.545566 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539839    2579 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 08 12:01:50.545566 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539842    2579 flags.go:64] FLAG: --enable-load-reader="false"
Jul 08 12:01:50.545566 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539845    2579 flags.go:64] FLAG: --enable-server="true"
Jul 08 12:01:50.545566 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539848    2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 08 12:01:50.545566 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539852    2579 flags.go:64] FLAG: --event-burst="100"
Jul 08 12:01:50.545566 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539854    2579 flags.go:64] FLAG: --event-qps="50"
Jul 08 12:01:50.545566 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539857    2579 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 08 12:01:50.545566 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539860    2579 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 08 12:01:50.545566 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539863    2579 flags.go:64] FLAG: --eviction-hard=""
Jul 08 12:01:50.545566 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539866    2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 08 12:01:50.545566 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539869    2579 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 08 12:01:50.545566 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539872    2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 08 12:01:50.545566 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539875    2579 flags.go:64] FLAG: --eviction-soft=""
Jul 08 12:01:50.546166 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539877    2579 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 08 12:01:50.546166 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539880    2579 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 08 12:01:50.546166 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539883    2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 08 12:01:50.546166 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539885    2579 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 08 12:01:50.546166 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539888    2579 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 08 12:01:50.546166 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539891    2579 flags.go:64] FLAG: --fail-swap-on="true"
Jul 08 12:01:50.546166 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539894    2579 flags.go:64] FLAG: --feature-gates=""
Jul 08 12:01:50.546166 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539898    2579 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 08 12:01:50.546166 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539901    2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 08 12:01:50.546166 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539904    2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 08 12:01:50.546166 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539907    2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 08 12:01:50.546166 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539909    2579 flags.go:64] FLAG: --healthz-port="10248"
Jul 08 12:01:50.546166 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539912    2579 flags.go:64] FLAG: --help="false"
Jul 08 12:01:50.546166 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539915    2579 flags.go:64] FLAG: --hostname-override="ip-10-0-138-89.ec2.internal"
Jul 08 12:01:50.546166 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539920    2579 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 08 12:01:50.546166 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539924    2579 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 08 12:01:50.546166 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539927    2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 08 12:01:50.546166 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539930    2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 08 12:01:50.546166 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539934    2579 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 08 12:01:50.546166 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539936    2579 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 08 12:01:50.546166 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539939    2579 flags.go:64] FLAG: --image-service-endpoint=""
Jul 08 12:01:50.546166 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539942    2579 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 08 12:01:50.546166 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539944    2579 flags.go:64] FLAG: --kube-api-burst="100"
Jul 08 12:01:50.546793 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539947    2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 08 12:01:50.546793 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539950    2579 flags.go:64] FLAG: --kube-api-qps="50"
Jul 08 12:01:50.546793 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539953    2579 flags.go:64] FLAG: --kube-reserved=""
Jul 08 12:01:50.546793 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539955    2579 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 08 12:01:50.546793 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539958    2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 08 12:01:50.546793 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539961    2579 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 08 12:01:50.546793 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539963    2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 08 12:01:50.546793 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539966    2579 flags.go:64] FLAG: --lock-file=""
Jul 08 12:01:50.546793 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539969    2579 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 08 12:01:50.546793 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539971    2579 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 08 12:01:50.546793 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539974    2579 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 08 12:01:50.546793 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539979    2579 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 08 12:01:50.546793 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539982    2579 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 08 12:01:50.546793 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539984    2579 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 08 12:01:50.546793 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539987    2579 flags.go:64] FLAG: --logging-format="text"
Jul 08 12:01:50.546793 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539989    2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 08 12:01:50.546793 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539993    2579 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 08 12:01:50.546793 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539996    2579 flags.go:64] FLAG: --manifest-url=""
Jul 08 12:01:50.546793 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.539998    2579 flags.go:64] FLAG: --manifest-url-header=""
Jul 08 12:01:50.546793 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.540002    2579 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 08 12:01:50.546793 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.540005    2579 flags.go:64] FLAG: --max-open-files="1000000"
Jul 08 12:01:50.546793 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.540009    2579 flags.go:64] FLAG: --max-pods="110"
Jul 08 12:01:50.546793 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.540012    2579 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 08 12:01:50.546793 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.540016    2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 08 12:01:50.546793 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.540019    2579 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 08 12:01:50.547400 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.540022    2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 08 12:01:50.547400 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.540025    2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 08 12:01:50.547400 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.540027    2579 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 08 12:01:50.547400 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.540030    2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 08 12:01:50.547400 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.540038    2579 flags.go:64] FLAG: --node-status-max-images="50"
Jul 08 12:01:50.547400 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.540041    2579 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 08 12:01:50.547400 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.540044    2579 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 08 12:01:50.547400 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.540047    2579 flags.go:64] FLAG: --pod-cidr=""
Jul 08 12:01:50.547400 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.540050    2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5"
Jul 08 12:01:50.547400 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.540055    2579 flags.go:64] FLAG: --pod-manifest-path=""
Jul 08 12:01:50.547400 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.540058    2579 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 08 12:01:50.547400 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.540061    2579 flags.go:64] FLAG: --pods-per-core="0"
Jul 08 12:01:50.547400 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.540064    2579 flags.go:64] FLAG: --port="10250"
Jul 08 12:01:50.547400 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.540067    2579 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 08 12:01:50.547400 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.540069    2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0341bccea12095f62"
Jul 08 12:01:50.547400 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.540072    2579 flags.go:64] FLAG: --qos-reserved=""
Jul 08 12:01:50.547400 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.540075    2579 flags.go:64] FLAG: --read-only-port="10255"
Jul 08 12:01:50.547400 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.540078    2579 flags.go:64] FLAG: --register-node="true"
Jul 08 12:01:50.547400 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.540081    2579 flags.go:64] FLAG: --register-with-taints=""
Jul 08 12:01:50.547400 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.540084    2579 flags.go:64] FLAG: --registry-burst="10"
Jul 08 12:01:50.547400 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.540087    2579 flags.go:64] FLAG: --registry-qps="5"
Jul 08 12:01:50.547400 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.540090    2579 flags.go:64] FLAG: --reserved-cpus=""
Jul 08 12:01:50.547400 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.540092    2579 flags.go:64] FLAG: --reserved-memory=""
Jul 08 12:01:50.547400 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.540096    2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 08 12:01:50.547400 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.540099    2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 08 12:01:50.548058 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.540102    2579 flags.go:64] FLAG: --rotate-certificates="false"
Jul 08 12:01:50.548058 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.540104    2579 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 08 12:01:50.548058 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.540107    2579 flags.go:64] FLAG: --runonce="false"
Jul 08 12:01:50.548058 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.540110    2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 08 12:01:50.548058 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.540112    2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 08 12:01:50.548058 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.540115    2579 flags.go:64] FLAG: --seccomp-default="false"
Jul 08 12:01:50.548058 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.540120    2579 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 08 12:01:50.548058 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.540123    2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 08 12:01:50.548058 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.540125    2579 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 08 12:01:50.548058 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.540128    2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 08 12:01:50.548058 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.540131    2579 flags.go:64] FLAG: --storage-driver-password="root"
Jul 08 12:01:50.548058 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.540134    2579 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 08 12:01:50.548058 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.540137    2579 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 08 12:01:50.548058 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.540139    2579 flags.go:64] FLAG: --storage-driver-user="root"
Jul 08 12:01:50.548058 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.540142    2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 08 12:01:50.548058 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.540145    2579 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 08 12:01:50.548058 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.540148    2579 flags.go:64] FLAG: --system-cgroups=""
Jul 08 12:01:50.548058 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.540150    2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 08 12:01:50.548058 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.540155    2579 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 08 12:01:50.548058 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.540158    2579 flags.go:64] FLAG: --tls-cert-file=""
Jul 08 12:01:50.548058 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.540160    2579 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 08 12:01:50.548058 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.540164    2579 flags.go:64] FLAG: --tls-min-version=""
Jul 08 12:01:50.548058 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.540167    2579 flags.go:64] FLAG: --tls-private-key-file=""
Jul 08 12:01:50.548058 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.540170    2579 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 08 12:01:50.548058 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.540172    2579 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 08 12:01:50.548644 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.540175    2579 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 08 12:01:50.548644 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.540178    2579 flags.go:64] FLAG: --v="2"
Jul 08 12:01:50.548644 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.540182    2579 flags.go:64] FLAG: --version="false"
Jul 08 12:01:50.548644 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.540186    2579 flags.go:64] FLAG: --vmodule=""
Jul 08 12:01:50.548644 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.540190    2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 08 12:01:50.548644 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.540193    2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 08 12:01:50.548644 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541383    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 08 12:01:50.548644 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541402    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 08 12:01:50.548644 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541409    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 08 12:01:50.548644 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541414    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 08 12:01:50.548644 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541419    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 08 12:01:50.548644 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541423    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 08 12:01:50.548644 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541427    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 08 12:01:50.548644 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541436    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 08 12:01:50.548644 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541441    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 08 12:01:50.548644 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541445    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 08 12:01:50.548644 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541450    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 08 12:01:50.548644 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541459    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 08 12:01:50.548644 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541464    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 08 12:01:50.548644 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541468    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 08 12:01:50.549137 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541473    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 08 12:01:50.549137 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541477    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 08 12:01:50.549137 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541481    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 08 12:01:50.549137 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541485    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 08 12:01:50.549137 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541489    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 08 12:01:50.549137 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541493    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 08 12:01:50.549137 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541501    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 08 12:01:50.549137 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541508    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 08 12:01:50.549137 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541518    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 08 12:01:50.549137 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541528    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 08 12:01:50.549137 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541533    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 08 12:01:50.549137 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541536    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 08 12:01:50.549137 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541539    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 08 12:01:50.549137 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541542    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 08 12:01:50.549137 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541561    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 08 12:01:50.549137 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541586    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 08 12:01:50.549137 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541593    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 08 12:01:50.549137 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541601    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 08 12:01:50.549137 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541606    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 08 12:01:50.549586 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541611    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 08 12:01:50.549586 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541615    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 08 12:01:50.549586 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541620    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 08 12:01:50.549586 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541624    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 08 12:01:50.549586 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541628    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 08 12:01:50.549586 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541633    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 08 12:01:50.549586 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541637    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 08 12:01:50.549586 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541641    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 08 12:01:50.549586 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541651    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 08 12:01:50.549586 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541656    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 08 12:01:50.549586 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541660    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 08 12:01:50.549586 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541664    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 08 12:01:50.549586 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541672    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 08 12:01:50.549586 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541678    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 08 12:01:50.549586 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541682    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 08 12:01:50.549586 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541686    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 08 12:01:50.549586 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541690    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 08 12:01:50.549586 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541694    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 08 12:01:50.549586 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541697    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 08 12:01:50.549586 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541701    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 08 12:01:50.550116 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541710    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 08 12:01:50.550116 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541714    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 08 12:01:50.550116 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541718    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 08 12:01:50.550116 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541721    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 08 12:01:50.550116 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541726    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 08 12:01:50.550116 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541729    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 08 12:01:50.550116 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541733    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 08 12:01:50.550116 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541738    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 08 12:01:50.550116 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541742    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 08 12:01:50.550116 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541748    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 08 12:01:50.550116 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541752    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 08 12:01:50.550116 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541756    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 08 12:01:50.550116 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541764    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 08 12:01:50.550116 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541784    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 08 12:01:50.550116 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541788    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 08 12:01:50.550116 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541793    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 08 12:01:50.550116 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541797    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 08 12:01:50.550116 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541801    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 08 12:01:50.550116 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541806    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 08 12:01:50.550116 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541809    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 08 12:01:50.550597 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541813    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 08 12:01:50.550597 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541817    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 08 12:01:50.550597 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541821    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 08 12:01:50.550597 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541826    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 08 12:01:50.550597 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541830    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 08 12:01:50.550597 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541847    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 08 12:01:50.550597 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541858    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 08 12:01:50.550597 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541862    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 08 12:01:50.550597 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541870    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 08 12:01:50.550597 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541972    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 08 12:01:50.550597 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541978    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 08 12:01:50.550597 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541983    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 08 12:01:50.550597 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541987    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 08 12:01:50.550597 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541990    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 08 12:01:50.550597 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541994    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 08 12:01:50.550597 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.541998    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 08 12:01:50.550597 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.542002    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 08 12:01:50.550597 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.542005    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 08 12:01:50.550597 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.542009    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 08 12:01:50.550597 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.542013    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 08 12:01:50.551095 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.542018    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 08 12:01:50.551095 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.542021    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 08 12:01:50.551095 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.542025    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 08 12:01:50.551095 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.542028    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 08 12:01:50.551095 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.542033    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 08 12:01:50.551095 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.542036    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 08 12:01:50.551095 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.542045    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 08 12:01:50.551095 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.548511    2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 08 12:01:50.551095 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.548528    2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 08 12:01:50.551095 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.549330    2579 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 08 12:01:50.551095 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.549338    2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 08 12:01:50.551095 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549399    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 08 12:01:50.551095 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549405    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 08 12:01:50.551095 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549408    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 08 12:01:50.551095 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549411    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 08 12:01:50.551489 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549414    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 08 12:01:50.551489 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549417    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 08 12:01:50.551489 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549420    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 08 12:01:50.551489 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549422    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 08 12:01:50.551489 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549425    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 08 12:01:50.551489 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549428    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 08 12:01:50.551489 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549430    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 08 12:01:50.551489 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549433    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 08 12:01:50.551489 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549435    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 08 12:01:50.551489 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549438    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 08 12:01:50.551489 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549440    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 08 12:01:50.551489 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549442    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 08 12:01:50.551489 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549444    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 08 12:01:50.551489 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549447    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 08 12:01:50.551489 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549450    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 08 12:01:50.551489 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549452    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 08 12:01:50.551489 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549455    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 08 12:01:50.551489 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549457    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 08 12:01:50.551489 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549459    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 08 12:01:50.551966 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549463    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 08 12:01:50.551966 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549467    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 08 12:01:50.551966 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549470    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 08 12:01:50.551966 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549472    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 08 12:01:50.551966 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549475    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 08 12:01:50.551966 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549478    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 08 12:01:50.551966 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549483    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 08 12:01:50.551966 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549486    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 08 12:01:50.551966 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549489    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 08 12:01:50.551966 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549492    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 08 12:01:50.551966 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549495    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 08 12:01:50.551966 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549497    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 08 12:01:50.551966 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549499    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 08 12:01:50.551966 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549502    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 08 12:01:50.551966 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549504    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 08 12:01:50.551966 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549507    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 08 12:01:50.551966 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549509    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 08 12:01:50.551966 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549511    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 08 12:01:50.551966 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549514    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 08 12:01:50.552412 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549516    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 08 12:01:50.552412 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549518    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 08 12:01:50.552412 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549521    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 08 12:01:50.552412 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549523    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 08 12:01:50.552412 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549526    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 08 12:01:50.552412 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549528    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 08 12:01:50.552412 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549531    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 08 12:01:50.552412 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549533    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 08 12:01:50.552412 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549536    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 08 12:01:50.552412 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549538    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 08 12:01:50.552412 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549540    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 08 12:01:50.552412 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549543    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 08 12:01:50.552412 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549545    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 08 12:01:50.552412 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549547    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 08 12:01:50.552412 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549550    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 08 12:01:50.552412 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549552    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 08 12:01:50.552412 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549554    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 08 12:01:50.552412 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549557    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 08 12:01:50.552412 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549559    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 08 12:01:50.552412 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549562    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 08 12:01:50.552955 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549564    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 08 12:01:50.552955 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549567    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 08 12:01:50.552955 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549570    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 08 12:01:50.552955 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549572    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 08 12:01:50.552955 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549574    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 08 12:01:50.552955 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549577    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 08 12:01:50.552955 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549579    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 08 12:01:50.552955 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549582    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 08 12:01:50.552955 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549584    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 08 12:01:50.552955 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549587    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 08 12:01:50.552955 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549589    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 08 12:01:50.552955 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549592    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 08 12:01:50.552955 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549594    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 08 12:01:50.552955 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549597    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 08 12:01:50.552955 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549600    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 08 12:01:50.552955 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549611    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 08 12:01:50.552955 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549613    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 08 12:01:50.552955 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549616    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 08 12:01:50.552955 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549618    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 08 12:01:50.552955 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549621    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 08 12:01:50.553423 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549623    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 08 12:01:50.553423 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549626    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 08 12:01:50.553423 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549628    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 08 12:01:50.553423 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549631    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 08 12:01:50.553423 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549633    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 08 12:01:50.553423 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549637    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 08 12:01:50.553423 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549641    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 08 12:01:50.553423 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549643    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 08 12:01:50.553423 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549646    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 08 12:01:50.553423 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549648    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 08 12:01:50.553423 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549650    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 08 12:01:50.553423 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549653    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 08 12:01:50.553423 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549655    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 08 12:01:50.553423 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549658    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 08 12:01:50.553423 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549660    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 08 12:01:50.553423 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549663    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 08 12:01:50.553423 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549665    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 08 12:01:50.553833 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.549670    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 08 12:01:50.553833 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549799    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 08 12:01:50.553833 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549805    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 08 12:01:50.553833 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549808    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 08 12:01:50.553833 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549810    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 08 12:01:50.553833 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549813    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 08 12:01:50.553833 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549816    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 08 12:01:50.553833 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549818    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 08 12:01:50.553833 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549821    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 08 12:01:50.553833 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549823    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 08 12:01:50.553833 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549825    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 08 12:01:50.553833 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549828    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 08 12:01:50.553833 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549831    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 08 12:01:50.553833 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549834    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 08 12:01:50.554178 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549836    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 08 12:01:50.554178 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549839    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 08 12:01:50.554178 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549841    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 08 12:01:50.554178 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549844    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 08 12:01:50.554178 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549846    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 08 12:01:50.554178 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549848    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 08 12:01:50.554178 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549851    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 08 12:01:50.554178 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549853    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 08 12:01:50.554178 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549855    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 08 12:01:50.554178 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549858    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 08 12:01:50.554178 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549860    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 08 12:01:50.554178 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549862    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 08 12:01:50.554178 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549865    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 08 12:01:50.554178 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549867    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 08 12:01:50.554178 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549870    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 08 12:01:50.554178 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549872    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 08 12:01:50.554178 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549874    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 08 12:01:50.554178 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549876    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 08 12:01:50.554178 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549879    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 08 12:01:50.554178 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549881    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 08 12:01:50.554634 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549883    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 08 12:01:50.554634 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549886    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 08 12:01:50.554634 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549888    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 08 12:01:50.554634 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549890    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 08 12:01:50.554634 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549893    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 08 12:01:50.554634 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549895    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 08 12:01:50.554634 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549897    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 08 12:01:50.554634 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549899    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 08 12:01:50.554634 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549902    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 08 12:01:50.554634 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549904    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 08 12:01:50.554634 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549907    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 08 12:01:50.554634 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549910    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 08 12:01:50.554634 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549912    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 08 12:01:50.554634 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549915    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 08 12:01:50.554634 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549918    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 08 12:01:50.554634 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549920    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 08 12:01:50.554634 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549922    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 08 12:01:50.554634 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549925    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 08 12:01:50.554634 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549928    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 08 12:01:50.554634 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549930    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 08 12:01:50.555106 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549933    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 08 12:01:50.555106 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549935    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 08 12:01:50.555106 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549937    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 08 12:01:50.555106 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549939    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 08 12:01:50.555106 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549942    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 08 12:01:50.555106 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549944    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 08 12:01:50.555106 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549946    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 08 12:01:50.555106 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549949    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 08 12:01:50.555106 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549951    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 08 12:01:50.555106 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549953    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 08 12:01:50.555106 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549955    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 08 12:01:50.555106 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549958    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 08 12:01:50.555106 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549960    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 08 12:01:50.555106 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549962    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 08 12:01:50.555106 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549964    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 08 12:01:50.555106 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549967    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 08 12:01:50.555106 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549969    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 08 12:01:50.555106 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549971    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 08 12:01:50.555106 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549973    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 08 12:01:50.555106 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549976    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 08 12:01:50.555576 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549978    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 08 12:01:50.555576 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549980    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 08 12:01:50.555576 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549982    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 08 12:01:50.555576 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549985    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 08 12:01:50.555576 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549987    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 08 12:01:50.555576 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549992    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 08 12:01:50.555576 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549994    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 08 12:01:50.555576 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549996    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 08 12:01:50.555576 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.549999    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 08 12:01:50.555576 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.550002    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 08 12:01:50.555576 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.550005    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 08 12:01:50.555576 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.550008    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 08 12:01:50.555576 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.550010    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 08 12:01:50.555576 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.550012    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 08 12:01:50.555576 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.550015    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 08 12:01:50.555576 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.550017    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 08 12:01:50.555576 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.550019    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 08 12:01:50.555576 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.550022    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 08 12:01:50.555576 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.550025    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 08 12:01:50.556038 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.550028    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 08 12:01:50.556038 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.550032    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 08 12:01:50.556038 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.550036    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 08 12:01:50.556038 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.550040    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 08 12:01:50.556038 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.550043    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 08 12:01:50.556038 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.550046    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 08 12:01:50.556038 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:50.550049    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 08 12:01:50.556038 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.550054    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 08 12:01:50.556038 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.550175    2579 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 08 12:01:50.556038 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.551967    2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 08 12:01:50.556038 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.553174    2579 server.go:1019] "Starting client certificate rotation"
Jul 08 12:01:50.556038 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.553269    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 08 12:01:50.556038 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.553329    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 08 12:01:50.556342 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.555792    2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 08 12:01:50.556939 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.556919    2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 08 12:01:50.567363 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.567343    2579 log.go:25] "Validated CRI v1 runtime API"
Jul 08 12:01:50.571246 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.571224    2579 log.go:25] "Validated CRI v1 image API"
Jul 08 12:01:50.572128 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.572110    2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 08 12:01:50.574481 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.574458    2579 fs.go:135] Filesystem UUIDs: map[76bb2590-34fc-44eb-9c1b-f82aea5a4a4a:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 8cf0dfac-5709-4855-8c28-98341c5b914c:/dev/nvme0n1p4]
Jul 08 12:01:50.574585 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.574479    2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 08 12:01:50.579986 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.579860    2579 manager.go:216] Machine: {Timestamp:2026-07-08 12:01:50.578582226 +0000 UTC m=+0.308957187 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3206808 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec288ecf16e76a94abf9ffb122e0fceb SystemUUID:ec288ecf-16e7-6a94-abf9-ffb122e0fceb BootID:0ecb0818-b241-4233-b829-372ffb139d00 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:34:43:4f:4c:f3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:34:43:4f:4c:f3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:06:85:52:a3:d7:fa Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 08 12:01:50.579986 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.579976    2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 08 12:01:50.580112 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.580060    2579 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 08 12:01:50.583240 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.583201    2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 08 12:01:50.583416 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.583246    2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-89.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 08 12:01:50.584001 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.583991    2579 topology_manager.go:138] "Creating topology manager with none policy"
Jul 08 12:01:50.584035 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.584003    2579 container_manager_linux.go:309] "Creating device plugin manager"
Jul 08 12:01:50.584035 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.584011    2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 08 12:01:50.584708 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.584698    2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 08 12:01:50.585950 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.585940    2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 08 12:01:50.586183 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.586166    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 08 12:01:50.588996 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.588982    2579 state_mem.go:36] "Initialized new in-memory state store"
Jul 08 12:01:50.589109 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.589101    2579 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 08 12:01:50.589145 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.589133    2579 kubelet.go:486] "Attempting to sync node with API server"
Jul 08 12:01:50.589145 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.589143    2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 08 12:01:50.589198 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.589164    2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 08 12:01:50.589228 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.589200    2579 kubelet.go:398] "Adding apiserver pod source"
Jul 08 12:01:50.589228 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.589211    2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 08 12:01:50.589582 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.589571    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 08 12:01:50.589627 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.589589    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 08 12:01:50.591794 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.591759    2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 08 12:01:50.593552 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.593535    2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 08 12:01:50.593645 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.593574    2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 08 12:01:50.594527 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.594514    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 08 12:01:50.594598 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.594533    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 08 12:01:50.594598 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.594542    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 08 12:01:50.594598 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.594569    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 08 12:01:50.594598 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.594583    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 08 12:01:50.594598 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.594591    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 08 12:01:50.594794 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.594602    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 08 12:01:50.594794 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.594610    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 08 12:01:50.594794 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.594620    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 08 12:01:50.594794 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.594627    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 08 12:01:50.594794 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.594649    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 08 12:01:50.594794 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.594657    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 08 12:01:50.596145 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.596133    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 08 12:01:50.596208 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.596148    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 08 12:01:50.599356 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.599339    2579 server.go:1268] "Started kubelet"
Jul 08 12:01:50.599464 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.599429    2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 08 12:01:50.599611 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.599556    2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 08 12:01:50.599730 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.599630    2579 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 08 12:01:50.600306 ip-10-0-138-89 systemd[1]: Started Kubernetes Kubelet.
Jul 08 12:01:50.601111 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.601089    2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-89.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 08 12:01:50.601111 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:01:50.601087    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-89.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 08 12:01:50.601384 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:01:50.601224    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 08 12:01:50.602405 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.602383    2579 server.go:310] "Adding debug handlers to kubelet server"
Jul 08 12:01:50.603358 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.603339    2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 08 12:01:50.607154 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:01:50.607132    2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 08 12:01:50.609398 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.609386    2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 08 12:01:50.609450 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.609414    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 08 12:01:50.609582 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:01:50.609558    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-89.ec2.internal\" not found"
Jul 08 12:01:50.609974 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:01:50.609106    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-89.ec2.internal.18c04ed0ace10f4b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-89.ec2.internal,UID:ip-10-0-138-89.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-89.ec2.internal,},FirstTimestamp:2026-07-08 12:01:50.599311179 +0000 UTC m=+0.329686141,LastTimestamp:2026-07-08 12:01:50.599311179 +0000 UTC m=+0.329686141,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-89.ec2.internal,}"
Jul 08 12:01:50.610231 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.610211    2579 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 08 12:01:50.610231 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.610233    2579 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 08 12:01:50.610360 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.610220    2579 factory.go:55] Registering systemd factory
Jul 08 12:01:50.610360 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.610221    2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 08 12:01:50.610360 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.610293    2579 factory.go:223] Registration of the systemd container factory successfully
Jul 08 12:01:50.610360 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.610352    2579 reconstruct.go:97] "Volume reconstruction finished"
Jul 08 12:01:50.610360 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.610363    2579 reconciler.go:29] "Reconciler: start to sync state"
Jul 08 12:01:50.610550 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.610538    2579 factory.go:153] Registering CRI-O factory
Jul 08 12:01:50.610586 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.610554    2579 factory.go:223] Registration of the crio container factory successfully
Jul 08 12:01:50.610651 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.610605    2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 08 12:01:50.610651 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.610629    2579 factory.go:103] Registering Raw factory
Jul 08 12:01:50.610651 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.610643    2579 manager.go:1196] Started watching for new ooms in manager
Jul 08 12:01:50.611265 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.611251    2579 manager.go:349] Starting recovery of all containers
Jul 08 12:01:50.611611 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.611590    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-njxxw"
Jul 08 12:01:50.613469 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:01:50.613440    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 08 12:01:50.613577 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:01:50.613514    2579 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-89.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 08 12:01:50.620381 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.620243    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-njxxw"
Jul 08 12:01:50.620718 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.620656    2579 manager.go:354] Recovery completed
Jul 08 12:01:50.625041 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.624997    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 12:01:50.626915 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.626899    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-89.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 12:01:50.626978 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.626933    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-89.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 12:01:50.626978 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.626944    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-89.ec2.internal" event="NodeHasSufficientPID"
Jul 08 12:01:50.627657 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.627637    2579 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 08 12:01:50.627657 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.627652    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 12:01:50.627818 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.627657    2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 08 12:01:50.627818 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.627722    2579 state_mem.go:36] "Initialized new in-memory state store"
Jul 08 12:01:50.628430 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.628396    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-89.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 12:01:50.628495 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.628446    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-89.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 12:01:50.628495 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.628461    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-89.ec2.internal" event="NodeHasSufficientPID"
Jul 08 12:01:50.628784 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.628759    2579 policy_none.go:49] "None policy: Start"
Jul 08 12:01:50.628837 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.628790    2579 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 08 12:01:50.628837 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.628803    2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 08 12:01:50.629057 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:01:50.628967    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-89.ec2.internal.18c04ed0ae864967  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-89.ec2.internal,UID:ip-10-0-138-89.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-89.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-89.ec2.internal,},FirstTimestamp:2026-07-08 12:01:50.626916711 +0000 UTC m=+0.357291675,LastTimestamp:2026-07-08 12:01:50.626916711 +0000 UTC m=+0.357291675,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-89.ec2.internal,}"
Jul 08 12:01:50.629694 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.629685    2579 policy_none.go:47] "Start"
Jul 08 12:01:50.670367 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.670340    2579 manager.go:337] "Starting Device Plugin manager"
Jul 08 12:01:50.670482 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:01:50.670387    2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 08 12:01:50.670482 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.670400    2579 server.go:87] "Starting device plugin registration server"
Jul 08 12:01:50.670677 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.670663    2579 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 08 12:01:50.670732 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.670679    2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 08 12:01:50.670809 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.670783    2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 08 12:01:50.670886 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.670869    2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 08 12:01:50.670886 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.670883    2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 08 12:01:50.671459 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:01:50.671434    2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 08 12:01:50.671522 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:01:50.671485    2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-89.ec2.internal\" not found"
Jul 08 12:01:50.708660 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.708629    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 08 12:01:50.711408 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.709990    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 08 12:01:50.711408 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.710024    2579 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 08 12:01:50.711408 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.710050    2579 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 08 12:01:50.711408 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:01:50.710091    2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 08 12:01:50.712237 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.712217    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 08 12:01:50.771367 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.771328    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 12:01:50.773221 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.773202    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-89.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 12:01:50.773334 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.773236    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-89.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 12:01:50.773334 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.773247    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-89.ec2.internal" event="NodeHasSufficientPID"
Jul 08 12:01:50.773334 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.773273    2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-89.ec2.internal"
Jul 08 12:01:50.782303 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.782253    2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-89.ec2.internal"
Jul 08 12:01:50.782303 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:01:50.782279    2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-89.ec2.internal\": node \"ip-10-0-138-89.ec2.internal\" not found"
Jul 08 12:01:50.802343 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:01:50.802317    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-89.ec2.internal\" not found"
Jul 08 12:01:50.810434 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.810398    2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-89.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-89.ec2.internal"]
Jul 08 12:01:50.810508 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.810476    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 12:01:50.811939 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.811923    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-89.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 12:01:50.812004 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.811955    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-89.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 12:01:50.812004 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.811965    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-89.ec2.internal" event="NodeHasSufficientPID"
Jul 08 12:01:50.813245 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.813233    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 12:01:50.813360 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.813346    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-89.ec2.internal"
Jul 08 12:01:50.813414 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.813376    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 12:01:50.815505 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.815489    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-89.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 12:01:50.815505 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.815504    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-89.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 12:01:50.815638 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.815520    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-89.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 12:01:50.815638 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.815524    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-89.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 12:01:50.815638 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.815534    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-89.ec2.internal" event="NodeHasSufficientPID"
Jul 08 12:01:50.815638 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.815534    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-89.ec2.internal" event="NodeHasSufficientPID"
Jul 08 12:01:50.816760 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.816744    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-89.ec2.internal"
Jul 08 12:01:50.816863 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.816787    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 12:01:50.817565 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.817549    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-89.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 12:01:50.817641 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.817582    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-89.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 12:01:50.817641 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.817595    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-89.ec2.internal" event="NodeHasSufficientPID"
Jul 08 12:01:50.844643 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:01:50.844610    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-89.ec2.internal\" not found" node="ip-10-0-138-89.ec2.internal"
Jul 08 12:01:50.849054 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:01:50.849035    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-89.ec2.internal\" not found" node="ip-10-0-138-89.ec2.internal"
Jul 08 12:01:50.903295 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:01:50.903265    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-89.ec2.internal\" not found"
Jul 08 12:01:50.911563 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.911535    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8e5e073dfe04675ff90daa0b47b9f43e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-89.ec2.internal\" (UID: \"8e5e073dfe04675ff90daa0b47b9f43e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-89.ec2.internal"
Jul 08 12:01:50.911636 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.911567    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8e5e073dfe04675ff90daa0b47b9f43e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-89.ec2.internal\" (UID: \"8e5e073dfe04675ff90daa0b47b9f43e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-89.ec2.internal"
Jul 08 12:01:50.911636 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:50.911584    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/01a7cd28e5cb68af10b158e7471bca6f-config\") pod \"kube-apiserver-proxy-ip-10-0-138-89.ec2.internal\" (UID: \"01a7cd28e5cb68af10b158e7471bca6f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-89.ec2.internal"
Jul 08 12:01:51.004269 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:01:51.004232    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-89.ec2.internal\" not found"
Jul 08 12:01:51.012578 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.012554    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8e5e073dfe04675ff90daa0b47b9f43e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-89.ec2.internal\" (UID: \"8e5e073dfe04675ff90daa0b47b9f43e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-89.ec2.internal"
Jul 08 12:01:51.012643 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.012598    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8e5e073dfe04675ff90daa0b47b9f43e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-89.ec2.internal\" (UID: \"8e5e073dfe04675ff90daa0b47b9f43e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-89.ec2.internal"
Jul 08 12:01:51.012643 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.012622    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/01a7cd28e5cb68af10b158e7471bca6f-config\") pod \"kube-apiserver-proxy-ip-10-0-138-89.ec2.internal\" (UID: \"01a7cd28e5cb68af10b158e7471bca6f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-89.ec2.internal"
Jul 08 12:01:51.012702 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.012672    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8e5e073dfe04675ff90daa0b47b9f43e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-89.ec2.internal\" (UID: \"8e5e073dfe04675ff90daa0b47b9f43e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-89.ec2.internal"
Jul 08 12:01:51.012702 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.012675    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8e5e073dfe04675ff90daa0b47b9f43e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-89.ec2.internal\" (UID: \"8e5e073dfe04675ff90daa0b47b9f43e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-89.ec2.internal"
Jul 08 12:01:51.012761 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.012722    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/01a7cd28e5cb68af10b158e7471bca6f-config\") pod \"kube-apiserver-proxy-ip-10-0-138-89.ec2.internal\" (UID: \"01a7cd28e5cb68af10b158e7471bca6f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-89.ec2.internal"
Jul 08 12:01:51.105067 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:01:51.104981    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-89.ec2.internal\" not found"
Jul 08 12:01:51.148606 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.148563    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-89.ec2.internal"
Jul 08 12:01:51.151426 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.151399    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-89.ec2.internal"
Jul 08 12:01:51.205594 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:01:51.205548    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-89.ec2.internal\" not found"
Jul 08 12:01:51.306193 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:01:51.306154    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-89.ec2.internal\" not found"
Jul 08 12:01:51.406838 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:01:51.406730    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-89.ec2.internal\" not found"
Jul 08 12:01:51.459787 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.459755    2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 08 12:01:51.510248 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.510218    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-89.ec2.internal"
Jul 08 12:01:51.522522 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.522490    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 08 12:01:51.524149 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.524131    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-89.ec2.internal"
Jul 08 12:01:51.532678 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.532657    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 08 12:01:51.552859 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.552828    2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 08 12:01:51.553002 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.552983    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 08 12:01:51.553043 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.553004    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 08 12:01:51.590177 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.590145    2579 apiserver.go:52] "Watching apiserver"
Jul 08 12:01:51.607730 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.607706    2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 08 12:01:51.609738 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.609719    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 08 12:01:51.610471 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.610440    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-k4jsh","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-89.ec2.internal","openshift-multus/network-metrics-daemon-m5628","openshift-network-diagnostics/network-check-target-842d7","kube-system/global-pull-secret-syncer-cslr6","openshift-cluster-node-tuning-operator/tuned-2jgc8","openshift-multus/multus-additional-cni-plugins-hrzgb","openshift-multus/multus-vm6ql","openshift-network-operator/iptables-alerter-5wpmj","openshift-ovn-kubernetes/ovnkube-node-xkp4b","kube-system/konnectivity-agent-pp86n","kube-system/kube-apiserver-proxy-ip-10-0-138-89.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwvt9","openshift-dns/node-resolver-26hpb"]
Jul 08 12:01:51.611948 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.611929    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hrzgb"
Jul 08 12:01:51.613064 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.613042    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwvt9"
Jul 08 12:01:51.614046 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.614023    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m5628"
Jul 08 12:01:51.614194 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:01:51.614149    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m5628" podUID="a64d26d8-d849-4f64-b43d-3a9133970f52"
Jul 08 12:01:51.614583 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.614562    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 08 12:01:51.614720 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.614562    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 08 12:01:51.614720 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.614606    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 08 12:01:51.614720 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.614710    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 08 12:01:51.614864 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.614605    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 08 12:01:51.614864 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.614823    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-58tkw\""
Jul 08 12:01:51.615260 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.615241    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 08 12:01:51.615666 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.615645    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 08 12:01:51.615742 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.615701    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 08 12:01:51.615813 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.615751    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-842d7"
Jul 08 12:01:51.615855 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:01:51.615826    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-842d7" podUID="ecbe5ccd-c4c5-4820-9238-72591f011842"
Jul 08 12:01:51.615915 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.615901    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-p98xd\""
Jul 08 12:01:51.619876 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.619855    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cslr6"
Jul 08 12:01:51.619979 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:01:51.619927    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cslr6" podUID="9e4572a9-69d7-418c-b751-07c5d6759d5a"
Jul 08 12:01:51.621261 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.621247    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2jgc8"
Jul 08 12:01:51.622339 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.622324    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5wpmj"
Jul 08 12:01:51.623331 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.623307    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 08 12:01:51.623525 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.623510    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-lkkn5\""
Jul 08 12:01:51.623596 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.623510    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 08 12:01:51.623659 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.623645    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-k4jsh"
Jul 08 12:01:51.623789 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.623759    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 08 12:01:51.624438 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.624419    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 08 12:01:51.625094 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.624818    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 08 12:01:51.625094 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.624844    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-bn55m\""
Jul 08 12:01:51.625094 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.624932    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 08 12:01:51.625094 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.625040    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-26hpb"
Jul 08 12:01:51.625379 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.625276    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-07 11:56:51 +0000 UTC" deadline="2028-03-23 01:08:27.524267734 +0000 UTC"
Jul 08 12:01:51.625379 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.625318    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14965h6m35.898953159s"
Jul 08 12:01:51.626182 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.626168    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 08 12:01:51.626446 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.626425    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xkp4b"
Jul 08 12:01:51.626993 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.626652    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 08 12:01:51.627117 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.627096    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 08 12:01:51.627210 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.627185    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vst85\""
Jul 08 12:01:51.627391 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.627337    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 08 12:01:51.627391 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.627344    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2hgbt\""
Jul 08 12:01:51.627786 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.627753    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 08 12:01:51.627887 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.627851    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pp86n"
Jul 08 12:01:51.628615 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.628597    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-lfkvw\""
Jul 08 12:01:51.628919 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.628898    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vm6ql"
Jul 08 12:01:51.629808 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.629545    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 08 12:01:51.629808 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.629612    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 08 12:01:51.629808 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.629624    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 08 12:01:51.629808 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.629654    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 08 12:01:51.630078 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.629886    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 08 12:01:51.630078 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.629888    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 08 12:01:51.630902 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.630884    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-scm6v\""
Jul 08 12:01:51.631005 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.630887    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 08 12:01:51.631005 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.630928    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 08 12:01:51.631559 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.631407    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-w42pw\""
Jul 08 12:01:51.632531 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.632511    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 08 12:01:51.651927 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.651904    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-9vcp2"
Jul 08 12:01:51.660433 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.660354    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-9vcp2"
Jul 08 12:01:51.711053 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.711029    2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 08 12:01:51.715608 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.715583    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq5n6\" (UniqueName: \"kubernetes.io/projected/c08cf95f-760d-49d9-8b97-b7cb64dc6c95-kube-api-access-cq5n6\") pod \"aws-ebs-csi-driver-node-wwvt9\" (UID: \"c08cf95f-760d-49d9-8b97-b7cb64dc6c95\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwvt9"
Jul 08 12:01:51.715742 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.715616    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z6tq\" (UniqueName: \"kubernetes.io/projected/641b596d-df5a-491a-a40e-e269de0a676e-kube-api-access-5z6tq\") pod \"node-ca-k4jsh\" (UID: \"641b596d-df5a-491a-a40e-e269de0a676e\") " pod="openshift-image-registry/node-ca-k4jsh"
Jul 08 12:01:51.715742 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.715640    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/24f4637c-726e-45ae-991a-c517b065e0cc-etc-systemd\") pod \"tuned-2jgc8\" (UID: \"24f4637c-726e-45ae-991a-c517b065e0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgc8"
Jul 08 12:01:51.715742 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.715661    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/24f4637c-726e-45ae-991a-c517b065e0cc-run\") pod \"tuned-2jgc8\" (UID: \"24f4637c-726e-45ae-991a-c517b065e0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgc8"
Jul 08 12:01:51.715742 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.715683    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/636b6729-2bf5-43f2-8db0-c1052576d023-multus-socket-dir-parent\") pod \"multus-vm6ql\" (UID: \"636b6729-2bf5-43f2-8db0-c1052576d023\") " pod="openshift-multus/multus-vm6ql"
Jul 08 12:01:51.715920 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.715742    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/636b6729-2bf5-43f2-8db0-c1052576d023-multus-daemon-config\") pod \"multus-vm6ql\" (UID: \"636b6729-2bf5-43f2-8db0-c1052576d023\") " pod="openshift-multus/multus-vm6ql"
Jul 08 12:01:51.715920 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.715792    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ffda1903-e8ae-4139-bb4c-d01904b23925-system-cni-dir\") pod \"multus-additional-cni-plugins-hrzgb\" (UID: \"ffda1903-e8ae-4139-bb4c-d01904b23925\") " pod="openshift-multus/multus-additional-cni-plugins-hrzgb"
Jul 08 12:01:51.715920 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.715812    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b6d8afe0-1555-462f-921e-eaa3d7acdb7a-ovnkube-config\") pod \"ovnkube-node-xkp4b\" (UID: \"b6d8afe0-1555-462f-921e-eaa3d7acdb7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkp4b"
Jul 08 12:01:51.715920 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.715826    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c08cf95f-760d-49d9-8b97-b7cb64dc6c95-socket-dir\") pod \"aws-ebs-csi-driver-node-wwvt9\" (UID: \"c08cf95f-760d-49d9-8b97-b7cb64dc6c95\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwvt9"
Jul 08 12:01:51.715920 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.715847    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c08cf95f-760d-49d9-8b97-b7cb64dc6c95-etc-selinux\") pod \"aws-ebs-csi-driver-node-wwvt9\" (UID: \"c08cf95f-760d-49d9-8b97-b7cb64dc6c95\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwvt9"
Jul 08 12:01:51.715920 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.715862    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/24f4637c-726e-45ae-991a-c517b065e0cc-host\") pod \"tuned-2jgc8\" (UID: \"24f4637c-726e-45ae-991a-c517b065e0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgc8"
Jul 08 12:01:51.715920 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.715898    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdj4j\" (UniqueName: \"kubernetes.io/projected/ecbe5ccd-c4c5-4820-9238-72591f011842-kube-api-access-sdj4j\") pod \"network-check-target-842d7\" (UID: \"ecbe5ccd-c4c5-4820-9238-72591f011842\") " pod="openshift-network-diagnostics/network-check-target-842d7"
Jul 08 12:01:51.716136 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.715926    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c08cf95f-760d-49d9-8b97-b7cb64dc6c95-device-dir\") pod \"aws-ebs-csi-driver-node-wwvt9\" (UID: \"c08cf95f-760d-49d9-8b97-b7cb64dc6c95\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwvt9"
Jul 08 12:01:51.716136 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.715951    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/636b6729-2bf5-43f2-8db0-c1052576d023-cni-binary-copy\") pod \"multus-vm6ql\" (UID: \"636b6729-2bf5-43f2-8db0-c1052576d023\") " pod="openshift-multus/multus-vm6ql"
Jul 08 12:01:51.716136 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.715970    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9e4572a9-69d7-418c-b751-07c5d6759d5a-kubelet-config\") pod \"global-pull-secret-syncer-cslr6\" (UID: \"9e4572a9-69d7-418c-b751-07c5d6759d5a\") " pod="kube-system/global-pull-secret-syncer-cslr6"
Jul 08 12:01:51.716136 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.715986    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b6d8afe0-1555-462f-921e-eaa3d7acdb7a-log-socket\") pod \"ovnkube-node-xkp4b\" (UID: \"b6d8afe0-1555-462f-921e-eaa3d7acdb7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkp4b"
Jul 08 12:01:51.716136 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.715999    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/641b596d-df5a-491a-a40e-e269de0a676e-serviceca\") pod \"node-ca-k4jsh\" (UID: \"641b596d-df5a-491a-a40e-e269de0a676e\") " pod="openshift-image-registry/node-ca-k4jsh"
Jul 08 12:01:51.716136 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.716020    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b6d8afe0-1555-462f-921e-eaa3d7acdb7a-systemd-units\") pod \"ovnkube-node-xkp4b\" (UID: \"b6d8afe0-1555-462f-921e-eaa3d7acdb7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkp4b"
Jul 08 12:01:51.716136 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.716037    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b6d8afe0-1555-462f-921e-eaa3d7acdb7a-run-systemd\") pod \"ovnkube-node-xkp4b\" (UID: \"b6d8afe0-1555-462f-921e-eaa3d7acdb7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkp4b"
Jul 08 12:01:51.716136 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.716050    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b6d8afe0-1555-462f-921e-eaa3d7acdb7a-host-cni-netd\") pod \"ovnkube-node-xkp4b\" (UID: \"b6d8afe0-1555-462f-921e-eaa3d7acdb7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkp4b"
Jul 08 12:01:51.716136 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.716063    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/24f4637c-726e-45ae-991a-c517b065e0cc-etc-tuned\") pod \"tuned-2jgc8\" (UID: \"24f4637c-726e-45ae-991a-c517b065e0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgc8"
Jul 08 12:01:51.716136 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.716076    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/24f4637c-726e-45ae-991a-c517b065e0cc-etc-kubernetes\") pod \"tuned-2jgc8\" (UID: \"24f4637c-726e-45ae-991a-c517b065e0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgc8"
Jul 08 12:01:51.716136 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.716089    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a64d26d8-d849-4f64-b43d-3a9133970f52-metrics-certs\") pod \"network-metrics-daemon-m5628\" (UID: \"a64d26d8-d849-4f64-b43d-3a9133970f52\") " pod="openshift-multus/network-metrics-daemon-m5628"
Jul 08 12:01:51.716136 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.716111    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b6d8afe0-1555-462f-921e-eaa3d7acdb7a-host-kubelet\") pod \"ovnkube-node-xkp4b\" (UID: \"b6d8afe0-1555-462f-921e-eaa3d7acdb7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkp4b"
Jul 08 12:01:51.716480 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.716139    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b6d8afe0-1555-462f-921e-eaa3d7acdb7a-host-slash\") pod \"ovnkube-node-xkp4b\" (UID: \"b6d8afe0-1555-462f-921e-eaa3d7acdb7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkp4b"
Jul 08 12:01:51.716480 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.716167    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b6d8afe0-1555-462f-921e-eaa3d7acdb7a-host-run-netns\") pod \"ovnkube-node-xkp4b\" (UID: \"b6d8afe0-1555-462f-921e-eaa3d7acdb7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkp4b"
Jul 08 12:01:51.716480 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.716199    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b6d8afe0-1555-462f-921e-eaa3d7acdb7a-run-openvswitch\") pod \"ovnkube-node-xkp4b\" (UID: \"b6d8afe0-1555-462f-921e-eaa3d7acdb7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkp4b"
Jul 08 12:01:51.716480 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.716228    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqzjw\" (UniqueName: \"kubernetes.io/projected/b6d8afe0-1555-462f-921e-eaa3d7acdb7a-kube-api-access-cqzjw\") pod \"ovnkube-node-xkp4b\" (UID: \"b6d8afe0-1555-462f-921e-eaa3d7acdb7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkp4b"
Jul 08 12:01:51.716480 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.716261    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c08cf95f-760d-49d9-8b97-b7cb64dc6c95-sys-fs\") pod \"aws-ebs-csi-driver-node-wwvt9\" (UID: \"c08cf95f-760d-49d9-8b97-b7cb64dc6c95\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwvt9"
Jul 08 12:01:51.716480 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.716282    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/24f4637c-726e-45ae-991a-c517b065e0cc-etc-sysconfig\") pod \"tuned-2jgc8\" (UID: \"24f4637c-726e-45ae-991a-c517b065e0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgc8"
Jul 08 12:01:51.716480 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.716297    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b6d8afe0-1555-462f-921e-eaa3d7acdb7a-env-overrides\") pod \"ovnkube-node-xkp4b\" (UID: \"b6d8afe0-1555-462f-921e-eaa3d7acdb7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkp4b"
Jul 08 12:01:51.716480 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.716311    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b6d8afe0-1555-462f-921e-eaa3d7acdb7a-ovnkube-script-lib\") pod \"ovnkube-node-xkp4b\" (UID: \"b6d8afe0-1555-462f-921e-eaa3d7acdb7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkp4b"
Jul 08 12:01:51.716480 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.716340    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9e4572a9-69d7-418c-b751-07c5d6759d5a-dbus\") pod \"global-pull-secret-syncer-cslr6\" (UID: \"9e4572a9-69d7-418c-b751-07c5d6759d5a\") " pod="kube-system/global-pull-secret-syncer-cslr6"
Jul 08 12:01:51.716480 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.716361    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9e4572a9-69d7-418c-b751-07c5d6759d5a-original-pull-secret\") pod \"global-pull-secret-syncer-cslr6\" (UID: \"9e4572a9-69d7-418c-b751-07c5d6759d5a\") " pod="kube-system/global-pull-secret-syncer-cslr6"
Jul 08 12:01:51.716480 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.716379    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9f9c\" (UniqueName: \"kubernetes.io/projected/e37bfd95-6a30-453c-be17-846c687e4d9e-kube-api-access-h9f9c\") pod \"iptables-alerter-5wpmj\" (UID: \"e37bfd95-6a30-453c-be17-846c687e4d9e\") " pod="openshift-network-operator/iptables-alerter-5wpmj"
Jul 08 12:01:51.716480 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.716397    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/636b6729-2bf5-43f2-8db0-c1052576d023-host-var-lib-cni-multus\") pod \"multus-vm6ql\" (UID: \"636b6729-2bf5-43f2-8db0-c1052576d023\") " pod="openshift-multus/multus-vm6ql"
Jul 08 12:01:51.716480 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.716414    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/636b6729-2bf5-43f2-8db0-c1052576d023-host-run-multus-certs\") pod \"multus-vm6ql\" (UID: \"636b6729-2bf5-43f2-8db0-c1052576d023\") " pod="openshift-multus/multus-vm6ql"
Jul 08 12:01:51.716480 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.716433    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ffda1903-e8ae-4139-bb4c-d01904b23925-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hrzgb\" (UID: \"ffda1903-e8ae-4139-bb4c-d01904b23925\") " pod="openshift-multus/multus-additional-cni-plugins-hrzgb"
Jul 08 12:01:51.716480 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.716449    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ffda1903-e8ae-4139-bb4c-d01904b23925-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hrzgb\" (UID: \"ffda1903-e8ae-4139-bb4c-d01904b23925\") " pod="openshift-multus/multus-additional-cni-plugins-hrzgb"
Jul 08 12:01:51.716480 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.716481    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b6d8afe0-1555-462f-921e-eaa3d7acdb7a-run-ovn\") pod \"ovnkube-node-xkp4b\" (UID: \"b6d8afe0-1555-462f-921e-eaa3d7acdb7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkp4b"
Jul 08 12:01:51.717134 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.716496    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/24f4637c-726e-45ae-991a-c517b065e0cc-etc-sysctl-d\") pod \"tuned-2jgc8\" (UID: \"24f4637c-726e-45ae-991a-c517b065e0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgc8"
Jul 08 12:01:51.717134 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.716518    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/24f4637c-726e-45ae-991a-c517b065e0cc-lib-modules\") pod \"tuned-2jgc8\" (UID: \"24f4637c-726e-45ae-991a-c517b065e0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgc8"
Jul 08 12:01:51.717134 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.716541    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e37bfd95-6a30-453c-be17-846c687e4d9e-iptables-alerter-script\") pod \"iptables-alerter-5wpmj\" (UID: \"e37bfd95-6a30-453c-be17-846c687e4d9e\") " pod="openshift-network-operator/iptables-alerter-5wpmj"
Jul 08 12:01:51.717134 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.716562    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hckqt\" (UniqueName: \"kubernetes.io/projected/a64d26d8-d849-4f64-b43d-3a9133970f52-kube-api-access-hckqt\") pod \"network-metrics-daemon-m5628\" (UID: \"a64d26d8-d849-4f64-b43d-3a9133970f52\") " pod="openshift-multus/network-metrics-daemon-m5628"
Jul 08 12:01:51.717134 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.716577    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b6d8afe0-1555-462f-921e-eaa3d7acdb7a-var-lib-openvswitch\") pod \"ovnkube-node-xkp4b\" (UID: \"b6d8afe0-1555-462f-921e-eaa3d7acdb7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkp4b"
Jul 08 12:01:51.717134 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.716603    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/636b6729-2bf5-43f2-8db0-c1052576d023-os-release\") pod \"multus-vm6ql\" (UID: \"636b6729-2bf5-43f2-8db0-c1052576d023\") " pod="openshift-multus/multus-vm6ql"
Jul 08 12:01:51.717134 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.716637    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/636b6729-2bf5-43f2-8db0-c1052576d023-host-var-lib-kubelet\") pod \"multus-vm6ql\" (UID: \"636b6729-2bf5-43f2-8db0-c1052576d023\") " pod="openshift-multus/multus-vm6ql"
Jul 08 12:01:51.717134 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.716659    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c08cf95f-760d-49d9-8b97-b7cb64dc6c95-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wwvt9\" (UID: \"c08cf95f-760d-49d9-8b97-b7cb64dc6c95\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwvt9"
Jul 08 12:01:51.717134 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.716681    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/24f4637c-726e-45ae-991a-c517b065e0cc-sys\") pod \"tuned-2jgc8\" (UID: \"24f4637c-726e-45ae-991a-c517b065e0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgc8"
Jul 08 12:01:51.717134 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.716699    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwwxw\" (UniqueName: \"kubernetes.io/projected/c0abdbfc-0312-4fd1-9ca4-098e3fa3e7e1-kube-api-access-fwwxw\") pod \"node-resolver-26hpb\" (UID: \"c0abdbfc-0312-4fd1-9ca4-098e3fa3e7e1\") " pod="openshift-dns/node-resolver-26hpb"
Jul 08 12:01:51.717134 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.716720    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/636b6729-2bf5-43f2-8db0-c1052576d023-multus-conf-dir\") pod \"multus-vm6ql\" (UID: \"636b6729-2bf5-43f2-8db0-c1052576d023\") " pod="openshift-multus/multus-vm6ql"
Jul 08 12:01:51.717134 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.716744    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbt58\" (UniqueName: \"kubernetes.io/projected/ffda1903-e8ae-4139-bb4c-d01904b23925-kube-api-access-kbt58\") pod \"multus-additional-cni-plugins-hrzgb\" (UID: \"ffda1903-e8ae-4139-bb4c-d01904b23925\") " pod="openshift-multus/multus-additional-cni-plugins-hrzgb"
Jul 08 12:01:51.717134 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.716760    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b6d8afe0-1555-462f-921e-eaa3d7acdb7a-node-log\") pod \"ovnkube-node-xkp4b\" (UID: \"b6d8afe0-1555-462f-921e-eaa3d7acdb7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkp4b"
Jul 08 12:01:51.717134 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.716800    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0abdbfc-0312-4fd1-9ca4-098e3fa3e7e1-tmp-dir\") pod \"node-resolver-26hpb\" (UID: \"c0abdbfc-0312-4fd1-9ca4-098e3fa3e7e1\") " pod="openshift-dns/node-resolver-26hpb"
Jul 08 12:01:51.717134 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.716821    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/636b6729-2bf5-43f2-8db0-c1052576d023-host-run-k8s-cni-cncf-io\") pod \"multus-vm6ql\" (UID: \"636b6729-2bf5-43f2-8db0-c1052576d023\") " pod="openshift-multus/multus-vm6ql"
Jul 08 12:01:51.717134 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.716839    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b6d8afe0-1555-462f-921e-eaa3d7acdb7a-ovn-node-metrics-cert\") pod \"ovnkube-node-xkp4b\" (UID: \"b6d8afe0-1555-462f-921e-eaa3d7acdb7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkp4b"
Jul 08 12:01:51.717743 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.716857    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/56220650-f743-4804-9e3e-5a0cffd4fd3a-agent-certs\") pod \"konnectivity-agent-pp86n\" (UID: \"56220650-f743-4804-9e3e-5a0cffd4fd3a\") " pod="kube-system/konnectivity-agent-pp86n"
Jul 08 12:01:51.717743 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.716920    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc6jr\" (UniqueName: \"kubernetes.io/projected/24f4637c-726e-45ae-991a-c517b065e0cc-kube-api-access-nc6jr\") pod \"tuned-2jgc8\" (UID: \"24f4637c-726e-45ae-991a-c517b065e0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgc8"
Jul 08 12:01:51.717743 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.716936    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48n2w\" (UniqueName: \"kubernetes.io/projected/636b6729-2bf5-43f2-8db0-c1052576d023-kube-api-access-48n2w\") pod \"multus-vm6ql\" (UID: \"636b6729-2bf5-43f2-8db0-c1052576d023\") " pod="openshift-multus/multus-vm6ql"
Jul 08 12:01:51.717743 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.716958    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b6d8afe0-1555-462f-921e-eaa3d7acdb7a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xkp4b\" (UID: \"b6d8afe0-1555-462f-921e-eaa3d7acdb7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkp4b"
Jul 08 12:01:51.717743 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.716981    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/56220650-f743-4804-9e3e-5a0cffd4fd3a-konnectivity-ca\") pod \"konnectivity-agent-pp86n\" (UID: \"56220650-f743-4804-9e3e-5a0cffd4fd3a\") " pod="kube-system/konnectivity-agent-pp86n"
Jul 08 12:01:51.717743 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.716996    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/24f4637c-726e-45ae-991a-c517b065e0cc-etc-modprobe-d\") pod \"tuned-2jgc8\" (UID: \"24f4637c-726e-45ae-991a-c517b065e0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgc8"
Jul 08 12:01:51.717743 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.717010    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ffda1903-e8ae-4139-bb4c-d01904b23925-os-release\") pod \"multus-additional-cni-plugins-hrzgb\" (UID: \"ffda1903-e8ae-4139-bb4c-d01904b23925\") " pod="openshift-multus/multus-additional-cni-plugins-hrzgb"
Jul 08 12:01:51.717743 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.717024    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/641b596d-df5a-491a-a40e-e269de0a676e-host\") pod \"node-ca-k4jsh\" (UID: \"641b596d-df5a-491a-a40e-e269de0a676e\") " pod="openshift-image-registry/node-ca-k4jsh"
Jul 08 12:01:51.717743 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.717046    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b6d8afe0-1555-462f-921e-eaa3d7acdb7a-host-cni-bin\") pod \"ovnkube-node-xkp4b\" (UID: \"b6d8afe0-1555-462f-921e-eaa3d7acdb7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkp4b"
Jul 08 12:01:51.717743 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.717079    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/24f4637c-726e-45ae-991a-c517b065e0cc-var-lib-kubelet\") pod \"tuned-2jgc8\" (UID: \"24f4637c-726e-45ae-991a-c517b065e0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgc8"
Jul 08 12:01:51.717743 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.717101    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ffda1903-e8ae-4139-bb4c-d01904b23925-cnibin\") pod \"multus-additional-cni-plugins-hrzgb\" (UID: \"ffda1903-e8ae-4139-bb4c-d01904b23925\") " pod="openshift-multus/multus-additional-cni-plugins-hrzgb"
Jul 08 12:01:51.717743 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.717122    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b6d8afe0-1555-462f-921e-eaa3d7acdb7a-etc-openvswitch\") pod \"ovnkube-node-xkp4b\" (UID: \"b6d8afe0-1555-462f-921e-eaa3d7acdb7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkp4b"
Jul 08 12:01:51.717743 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.717153    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c08cf95f-760d-49d9-8b97-b7cb64dc6c95-registration-dir\") pod \"aws-ebs-csi-driver-node-wwvt9\" (UID: \"c08cf95f-760d-49d9-8b97-b7cb64dc6c95\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwvt9"
Jul 08 12:01:51.717743 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.717157    2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 08 12:01:51.717743 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.717227    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e37bfd95-6a30-453c-be17-846c687e4d9e-host-slash\") pod \"iptables-alerter-5wpmj\" (UID: \"e37bfd95-6a30-453c-be17-846c687e4d9e\") " pod="openshift-network-operator/iptables-alerter-5wpmj"
Jul 08 12:01:51.717743 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.717251    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c0abdbfc-0312-4fd1-9ca4-098e3fa3e7e1-hosts-file\") pod \"node-resolver-26hpb\" (UID: \"c0abdbfc-0312-4fd1-9ca4-098e3fa3e7e1\") " pod="openshift-dns/node-resolver-26hpb"
Jul 08 12:01:51.717743 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.717272    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/636b6729-2bf5-43f2-8db0-c1052576d023-system-cni-dir\") pod \"multus-vm6ql\" (UID: \"636b6729-2bf5-43f2-8db0-c1052576d023\") " pod="openshift-multus/multus-vm6ql"
Jul 08 12:01:51.718337 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.717310    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/636b6729-2bf5-43f2-8db0-c1052576d023-multus-cni-dir\") pod \"multus-vm6ql\" (UID: \"636b6729-2bf5-43f2-8db0-c1052576d023\") " pod="openshift-multus/multus-vm6ql"
Jul 08 12:01:51.718337 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.717329    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/636b6729-2bf5-43f2-8db0-c1052576d023-cnibin\") pod \"multus-vm6ql\" (UID: \"636b6729-2bf5-43f2-8db0-c1052576d023\") " pod="openshift-multus/multus-vm6ql"
Jul 08 12:01:51.718337 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.717350    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/24f4637c-726e-45ae-991a-c517b065e0cc-etc-sysctl-conf\") pod \"tuned-2jgc8\" (UID: \"24f4637c-726e-45ae-991a-c517b065e0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgc8"
Jul 08 12:01:51.718337 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.717374    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b6d8afe0-1555-462f-921e-eaa3d7acdb7a-host-run-ovn-kubernetes\") pod \"ovnkube-node-xkp4b\" (UID: \"b6d8afe0-1555-462f-921e-eaa3d7acdb7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkp4b"
Jul 08 12:01:51.718337 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.717397    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24f4637c-726e-45ae-991a-c517b065e0cc-tmp\") pod \"tuned-2jgc8\" (UID: \"24f4637c-726e-45ae-991a-c517b065e0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgc8"
Jul 08 12:01:51.718337 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.717426    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/636b6729-2bf5-43f2-8db0-c1052576d023-host-run-netns\") pod \"multus-vm6ql\" (UID: \"636b6729-2bf5-43f2-8db0-c1052576d023\") " pod="openshift-multus/multus-vm6ql"
Jul 08 12:01:51.718337 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.717447    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/636b6729-2bf5-43f2-8db0-c1052576d023-hostroot\") pod \"multus-vm6ql\" (UID: \"636b6729-2bf5-43f2-8db0-c1052576d023\") " pod="openshift-multus/multus-vm6ql"
Jul 08 12:01:51.718337 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.717485    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ffda1903-e8ae-4139-bb4c-d01904b23925-cni-binary-copy\") pod \"multus-additional-cni-plugins-hrzgb\" (UID: \"ffda1903-e8ae-4139-bb4c-d01904b23925\") " pod="openshift-multus/multus-additional-cni-plugins-hrzgb"
Jul 08 12:01:51.718337 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.717515    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ffda1903-e8ae-4139-bb4c-d01904b23925-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hrzgb\" (UID: \"ffda1903-e8ae-4139-bb4c-d01904b23925\") " pod="openshift-multus/multus-additional-cni-plugins-hrzgb"
Jul 08 12:01:51.718337 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.717540    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/636b6729-2bf5-43f2-8db0-c1052576d023-host-var-lib-cni-bin\") pod \"multus-vm6ql\" (UID: \"636b6729-2bf5-43f2-8db0-c1052576d023\") " pod="openshift-multus/multus-vm6ql"
Jul 08 12:01:51.718337 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.717594    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/636b6729-2bf5-43f2-8db0-c1052576d023-etc-kubernetes\") pod \"multus-vm6ql\" (UID: \"636b6729-2bf5-43f2-8db0-c1052576d023\") " pod="openshift-multus/multus-vm6ql"
Jul 08 12:01:51.718337 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:51.718251    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01a7cd28e5cb68af10b158e7471bca6f.slice/crio-f490e103b2188be561e65a2499a02db4c19b5f0b9dc66a892b425cca1099986c WatchSource:0}: Error finding container f490e103b2188be561e65a2499a02db4c19b5f0b9dc66a892b425cca1099986c: Status 404 returned error can't find the container with id f490e103b2188be561e65a2499a02db4c19b5f0b9dc66a892b425cca1099986c
Jul 08 12:01:51.718731 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:51.718460    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e5e073dfe04675ff90daa0b47b9f43e.slice/crio-aad6178b3249d6654253e4810ee9be022c0dc31507ef0a855e02f0afc061ce23 WatchSource:0}: Error finding container aad6178b3249d6654253e4810ee9be022c0dc31507ef0a855e02f0afc061ce23: Status 404 returned error can't find the container with id aad6178b3249d6654253e4810ee9be022c0dc31507ef0a855e02f0afc061ce23
Jul 08 12:01:51.723205 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.723190    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 08 12:01:51.817910 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.817883    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b6d8afe0-1555-462f-921e-eaa3d7acdb7a-env-overrides\") pod \"ovnkube-node-xkp4b\" (UID: \"b6d8afe0-1555-462f-921e-eaa3d7acdb7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkp4b"
Jul 08 12:01:51.817910 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.817914    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b6d8afe0-1555-462f-921e-eaa3d7acdb7a-ovnkube-script-lib\") pod \"ovnkube-node-xkp4b\" (UID: \"b6d8afe0-1555-462f-921e-eaa3d7acdb7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkp4b"
Jul 08 12:01:51.818158 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.817930    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9e4572a9-69d7-418c-b751-07c5d6759d5a-dbus\") pod \"global-pull-secret-syncer-cslr6\" (UID: \"9e4572a9-69d7-418c-b751-07c5d6759d5a\") " pod="kube-system/global-pull-secret-syncer-cslr6"
Jul 08 12:01:51.818158 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.817945    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9e4572a9-69d7-418c-b751-07c5d6759d5a-original-pull-secret\") pod \"global-pull-secret-syncer-cslr6\" (UID: \"9e4572a9-69d7-418c-b751-07c5d6759d5a\") " pod="kube-system/global-pull-secret-syncer-cslr6"
Jul 08 12:01:51.818158 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.817960    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9f9c\" (UniqueName: \"kubernetes.io/projected/e37bfd95-6a30-453c-be17-846c687e4d9e-kube-api-access-h9f9c\") pod \"iptables-alerter-5wpmj\" (UID: \"e37bfd95-6a30-453c-be17-846c687e4d9e\") " pod="openshift-network-operator/iptables-alerter-5wpmj"
Jul 08 12:01:51.818158 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.817983    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/636b6729-2bf5-43f2-8db0-c1052576d023-host-var-lib-cni-multus\") pod \"multus-vm6ql\" (UID: \"636b6729-2bf5-43f2-8db0-c1052576d023\") " pod="openshift-multus/multus-vm6ql"
Jul 08 12:01:51.818158 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.818004    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/636b6729-2bf5-43f2-8db0-c1052576d023-host-run-multus-certs\") pod \"multus-vm6ql\" (UID: \"636b6729-2bf5-43f2-8db0-c1052576d023\") " pod="openshift-multus/multus-vm6ql"
Jul 08 12:01:51.818158 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.818042    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9e4572a9-69d7-418c-b751-07c5d6759d5a-dbus\") pod \"global-pull-secret-syncer-cslr6\" (UID: \"9e4572a9-69d7-418c-b751-07c5d6759d5a\") " pod="kube-system/global-pull-secret-syncer-cslr6"
Jul 08 12:01:51.818158 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.818048    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/636b6729-2bf5-43f2-8db0-c1052576d023-host-var-lib-cni-multus\") pod \"multus-vm6ql\" (UID: \"636b6729-2bf5-43f2-8db0-c1052576d023\") " pod="openshift-multus/multus-vm6ql"
Jul 08 12:01:51.818158 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.818066    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ffda1903-e8ae-4139-bb4c-d01904b23925-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hrzgb\" (UID: \"ffda1903-e8ae-4139-bb4c-d01904b23925\") " pod="openshift-multus/multus-additional-cni-plugins-hrzgb"
Jul 08 12:01:51.818158 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:01:51.818081    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 12:01:51.818158 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.818101    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/636b6729-2bf5-43f2-8db0-c1052576d023-host-run-multus-certs\") pod \"multus-vm6ql\" (UID: \"636b6729-2bf5-43f2-8db0-c1052576d023\") " pod="openshift-multus/multus-vm6ql"
Jul 08 12:01:51.818158 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.818110    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ffda1903-e8ae-4139-bb4c-d01904b23925-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hrzgb\" (UID: \"ffda1903-e8ae-4139-bb4c-d01904b23925\") " pod="openshift-multus/multus-additional-cni-plugins-hrzgb"
Jul 08 12:01:51.819365 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:01:51.818180    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e4572a9-69d7-418c-b751-07c5d6759d5a-original-pull-secret podName:9e4572a9-69d7-418c-b751-07c5d6759d5a nodeName:}" failed. No retries permitted until 2026-07-08 12:01:52.318133589 +0000 UTC m=+2.048508555 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9e4572a9-69d7-418c-b751-07c5d6759d5a-original-pull-secret") pod "global-pull-secret-syncer-cslr6" (UID: "9e4572a9-69d7-418c-b751-07c5d6759d5a") : object "kube-system"/"original-pull-secret" not registered
Jul 08 12:01:51.819365 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.818208    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b6d8afe0-1555-462f-921e-eaa3d7acdb7a-run-ovn\") pod \"ovnkube-node-xkp4b\" (UID: \"b6d8afe0-1555-462f-921e-eaa3d7acdb7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkp4b"
Jul 08 12:01:51.819365 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.818233    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/24f4637c-726e-45ae-991a-c517b065e0cc-etc-sysctl-d\") pod \"tuned-2jgc8\" (UID: \"24f4637c-726e-45ae-991a-c517b065e0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgc8"
Jul 08 12:01:51.819365 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.818255    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/24f4637c-726e-45ae-991a-c517b065e0cc-lib-modules\") pod \"tuned-2jgc8\" (UID: \"24f4637c-726e-45ae-991a-c517b065e0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgc8"
Jul 08 12:01:51.819365 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.818265    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b6d8afe0-1555-462f-921e-eaa3d7acdb7a-run-ovn\") pod \"ovnkube-node-xkp4b\" (UID: \"b6d8afe0-1555-462f-921e-eaa3d7acdb7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkp4b"
Jul 08 12:01:51.819365 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.818281    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e37bfd95-6a30-453c-be17-846c687e4d9e-iptables-alerter-script\") pod \"iptables-alerter-5wpmj\" (UID: \"e37bfd95-6a30-453c-be17-846c687e4d9e\") " pod="openshift-network-operator/iptables-alerter-5wpmj"
Jul 08 12:01:51.819365 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.818273    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ffda1903-e8ae-4139-bb4c-d01904b23925-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hrzgb\" (UID: \"ffda1903-e8ae-4139-bb4c-d01904b23925\") " pod="openshift-multus/multus-additional-cni-plugins-hrzgb"
Jul 08 12:01:51.819365 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.818310    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hckqt\" (UniqueName: \"kubernetes.io/projected/a64d26d8-d849-4f64-b43d-3a9133970f52-kube-api-access-hckqt\") pod \"network-metrics-daemon-m5628\" (UID: \"a64d26d8-d849-4f64-b43d-3a9133970f52\") " pod="openshift-multus/network-metrics-daemon-m5628"
Jul 08 12:01:51.819365 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.818334    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b6d8afe0-1555-462f-921e-eaa3d7acdb7a-var-lib-openvswitch\") pod \"ovnkube-node-xkp4b\" (UID: \"b6d8afe0-1555-462f-921e-eaa3d7acdb7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkp4b"
Jul 08 12:01:51.819365 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.818356    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/636b6729-2bf5-43f2-8db0-c1052576d023-os-release\") pod \"multus-vm6ql\" (UID: \"636b6729-2bf5-43f2-8db0-c1052576d023\") " pod="openshift-multus/multus-vm6ql"
Jul 08 12:01:51.819365 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.818369    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/24f4637c-726e-45ae-991a-c517b065e0cc-etc-sysctl-d\") pod \"tuned-2jgc8\" (UID: \"24f4637c-726e-45ae-991a-c517b065e0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgc8"
Jul 08 12:01:51.819365 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.818376    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/636b6729-2bf5-43f2-8db0-c1052576d023-host-var-lib-kubelet\") pod \"multus-vm6ql\" (UID: \"636b6729-2bf5-43f2-8db0-c1052576d023\") " pod="openshift-multus/multus-vm6ql"
Jul 08 12:01:51.819365 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.818383    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/24f4637c-726e-45ae-991a-c517b065e0cc-lib-modules\") pod \"tuned-2jgc8\" (UID: \"24f4637c-726e-45ae-991a-c517b065e0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgc8"
Jul 08 12:01:51.819365 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.818413    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c08cf95f-760d-49d9-8b97-b7cb64dc6c95-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wwvt9\" (UID: \"c08cf95f-760d-49d9-8b97-b7cb64dc6c95\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwvt9"
Jul 08 12:01:51.819365 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.818421    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b6d8afe0-1555-462f-921e-eaa3d7acdb7a-var-lib-openvswitch\") pod \"ovnkube-node-xkp4b\" (UID: \"b6d8afe0-1555-462f-921e-eaa3d7acdb7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkp4b"
Jul 08 12:01:51.819365 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.818438    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/24f4637c-726e-45ae-991a-c517b065e0cc-sys\") pod \"tuned-2jgc8\" (UID: \"24f4637c-726e-45ae-991a-c517b065e0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgc8"
Jul 08 12:01:51.819365 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.818416    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/636b6729-2bf5-43f2-8db0-c1052576d023-host-var-lib-kubelet\") pod \"multus-vm6ql\" (UID: \"636b6729-2bf5-43f2-8db0-c1052576d023\") " pod="openshift-multus/multus-vm6ql"
Jul 08 12:01:51.820230 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.818463    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwwxw\" (UniqueName: \"kubernetes.io/projected/c0abdbfc-0312-4fd1-9ca4-098e3fa3e7e1-kube-api-access-fwwxw\") pod \"node-resolver-26hpb\" (UID: \"c0abdbfc-0312-4fd1-9ca4-098e3fa3e7e1\") " pod="openshift-dns/node-resolver-26hpb"
Jul 08 12:01:51.820230 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.818474    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c08cf95f-760d-49d9-8b97-b7cb64dc6c95-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wwvt9\" (UID: \"c08cf95f-760d-49d9-8b97-b7cb64dc6c95\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwvt9"
Jul 08 12:01:51.820230 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.818441    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/636b6729-2bf5-43f2-8db0-c1052576d023-os-release\") pod \"multus-vm6ql\" (UID: \"636b6729-2bf5-43f2-8db0-c1052576d023\") " pod="openshift-multus/multus-vm6ql"
Jul 08 12:01:51.820230 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.818487    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/636b6729-2bf5-43f2-8db0-c1052576d023-multus-conf-dir\") pod \"multus-vm6ql\" (UID: \"636b6729-2bf5-43f2-8db0-c1052576d023\") " pod="openshift-multus/multus-vm6ql"
Jul 08 12:01:51.820230 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.818488    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/24f4637c-726e-45ae-991a-c517b065e0cc-sys\") pod \"tuned-2jgc8\" (UID: \"24f4637c-726e-45ae-991a-c517b065e0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgc8"
Jul 08 12:01:51.820230 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.818520    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/636b6729-2bf5-43f2-8db0-c1052576d023-multus-conf-dir\") pod \"multus-vm6ql\" (UID: \"636b6729-2bf5-43f2-8db0-c1052576d023\") " pod="openshift-multus/multus-vm6ql"
Jul 08 12:01:51.820230 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.818517    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbt58\" (UniqueName: \"kubernetes.io/projected/ffda1903-e8ae-4139-bb4c-d01904b23925-kube-api-access-kbt58\") pod \"multus-additional-cni-plugins-hrzgb\" (UID: \"ffda1903-e8ae-4139-bb4c-d01904b23925\") " pod="openshift-multus/multus-additional-cni-plugins-hrzgb"
Jul 08 12:01:51.820230 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.818567    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b6d8afe0-1555-462f-921e-eaa3d7acdb7a-node-log\") pod \"ovnkube-node-xkp4b\" (UID: \"b6d8afe0-1555-462f-921e-eaa3d7acdb7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkp4b"
Jul 08 12:01:51.820230 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.818606    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0abdbfc-0312-4fd1-9ca4-098e3fa3e7e1-tmp-dir\") pod \"node-resolver-26hpb\" (UID: \"c0abdbfc-0312-4fd1-9ca4-098e3fa3e7e1\") " pod="openshift-dns/node-resolver-26hpb"
Jul 08 12:01:51.820230 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.818633    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/636b6729-2bf5-43f2-8db0-c1052576d023-host-run-k8s-cni-cncf-io\") pod \"multus-vm6ql\" (UID: \"636b6729-2bf5-43f2-8db0-c1052576d023\") " pod="openshift-multus/multus-vm6ql"
Jul 08 12:01:51.820230 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.818637    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b6d8afe0-1555-462f-921e-eaa3d7acdb7a-node-log\") pod \"ovnkube-node-xkp4b\" (UID: \"b6d8afe0-1555-462f-921e-eaa3d7acdb7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkp4b"
Jul 08 12:01:51.820230 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.818655    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ffda1903-e8ae-4139-bb4c-d01904b23925-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hrzgb\" (UID: \"ffda1903-e8ae-4139-bb4c-d01904b23925\") " pod="openshift-multus/multus-additional-cni-plugins-hrzgb"
Jul 08 12:01:51.820230 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.818666    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b6d8afe0-1555-462f-921e-eaa3d7acdb7a-ovn-node-metrics-cert\") pod \"ovnkube-node-xkp4b\" (UID: \"b6d8afe0-1555-462f-921e-eaa3d7acdb7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkp4b"
Jul 08 12:01:51.820230 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.818692    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/56220650-f743-4804-9e3e-5a0cffd4fd3a-agent-certs\") pod \"konnectivity-agent-pp86n\" (UID: \"56220650-f743-4804-9e3e-5a0cffd4fd3a\") " pod="kube-system/konnectivity-agent-pp86n"
Jul 08 12:01:51.820230 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.818725    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc6jr\" (UniqueName: \"kubernetes.io/projected/24f4637c-726e-45ae-991a-c517b065e0cc-kube-api-access-nc6jr\") pod \"tuned-2jgc8\" (UID: \"24f4637c-726e-45ae-991a-c517b065e0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgc8"
Jul 08 12:01:51.820230 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.818750    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48n2w\" (UniqueName: \"kubernetes.io/projected/636b6729-2bf5-43f2-8db0-c1052576d023-kube-api-access-48n2w\") pod \"multus-vm6ql\" (UID: \"636b6729-2bf5-43f2-8db0-c1052576d023\") " pod="openshift-multus/multus-vm6ql"
Jul 08 12:01:51.820230 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.818693    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/636b6729-2bf5-43f2-8db0-c1052576d023-host-run-k8s-cni-cncf-io\") pod \"multus-vm6ql\" (UID: \"636b6729-2bf5-43f2-8db0-c1052576d023\") " pod="openshift-multus/multus-vm6ql"
Jul 08 12:01:51.821049 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.818791    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b6d8afe0-1555-462f-921e-eaa3d7acdb7a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xkp4b\" (UID: \"b6d8afe0-1555-462f-921e-eaa3d7acdb7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkp4b"
Jul 08 12:01:51.821049 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.818817    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/56220650-f743-4804-9e3e-5a0cffd4fd3a-konnectivity-ca\") pod \"konnectivity-agent-pp86n\" (UID: \"56220650-f743-4804-9e3e-5a0cffd4fd3a\") " pod="kube-system/konnectivity-agent-pp86n"
Jul 08 12:01:51.821049 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.818829    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b6d8afe0-1555-462f-921e-eaa3d7acdb7a-env-overrides\") pod \"ovnkube-node-xkp4b\" (UID: \"b6d8afe0-1555-462f-921e-eaa3d7acdb7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkp4b"
Jul 08 12:01:51.821049 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.818840    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/24f4637c-726e-45ae-991a-c517b065e0cc-etc-modprobe-d\") pod \"tuned-2jgc8\" (UID: \"24f4637c-726e-45ae-991a-c517b065e0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgc8"
Jul 08 12:01:51.821049 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.818872    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ffda1903-e8ae-4139-bb4c-d01904b23925-os-release\") pod \"multus-additional-cni-plugins-hrzgb\" (UID: \"ffda1903-e8ae-4139-bb4c-d01904b23925\") " pod="openshift-multus/multus-additional-cni-plugins-hrzgb"
Jul 08 12:01:51.821049 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.818880    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b6d8afe0-1555-462f-921e-eaa3d7acdb7a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xkp4b\" (UID: \"b6d8afe0-1555-462f-921e-eaa3d7acdb7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkp4b"
Jul 08 12:01:51.821049 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.818891    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e37bfd95-6a30-453c-be17-846c687e4d9e-iptables-alerter-script\") pod \"iptables-alerter-5wpmj\" (UID: \"e37bfd95-6a30-453c-be17-846c687e4d9e\") " pod="openshift-network-operator/iptables-alerter-5wpmj"
Jul 08 12:01:51.821049 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.818897    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/641b596d-df5a-491a-a40e-e269de0a676e-host\") pod \"node-ca-k4jsh\" (UID: \"641b596d-df5a-491a-a40e-e269de0a676e\") " pod="openshift-image-registry/node-ca-k4jsh"
Jul 08 12:01:51.821049 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.818920    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b6d8afe0-1555-462f-921e-eaa3d7acdb7a-host-cni-bin\") pod \"ovnkube-node-xkp4b\" (UID: \"b6d8afe0-1555-462f-921e-eaa3d7acdb7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkp4b"
Jul 08 12:01:51.821049 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.818941    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/24f4637c-726e-45ae-991a-c517b065e0cc-var-lib-kubelet\") pod \"tuned-2jgc8\" (UID: \"24f4637c-726e-45ae-991a-c517b065e0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgc8"
Jul 08 12:01:51.821049 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.818952    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0abdbfc-0312-4fd1-9ca4-098e3fa3e7e1-tmp-dir\") pod \"node-resolver-26hpb\" (UID: \"c0abdbfc-0312-4fd1-9ca4-098e3fa3e7e1\") " pod="openshift-dns/node-resolver-26hpb"
Jul 08 12:01:51.821049 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.818962    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ffda1903-e8ae-4139-bb4c-d01904b23925-cnibin\") pod \"multus-additional-cni-plugins-hrzgb\" (UID: \"ffda1903-e8ae-4139-bb4c-d01904b23925\") " pod="openshift-multus/multus-additional-cni-plugins-hrzgb"
Jul 08 12:01:51.821049 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.818996    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b6d8afe0-1555-462f-921e-eaa3d7acdb7a-ovnkube-script-lib\") pod \"ovnkube-node-xkp4b\" (UID: \"b6d8afe0-1555-462f-921e-eaa3d7acdb7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkp4b"
Jul 08 12:01:51.821049 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.819000    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/641b596d-df5a-491a-a40e-e269de0a676e-host\") pod \"node-ca-k4jsh\" (UID: \"641b596d-df5a-491a-a40e-e269de0a676e\") " pod="openshift-image-registry/node-ca-k4jsh"
Jul 08 12:01:51.821049 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.819003    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/24f4637c-726e-45ae-991a-c517b065e0cc-etc-modprobe-d\") pod \"tuned-2jgc8\" (UID: \"24f4637c-726e-45ae-991a-c517b065e0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgc8"
Jul 08 12:01:51.821049 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.819000    2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 08 12:01:51.821049 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.819101    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/24f4637c-726e-45ae-991a-c517b065e0cc-var-lib-kubelet\") pod \"tuned-2jgc8\" (UID: \"24f4637c-726e-45ae-991a-c517b065e0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgc8"
Jul 08 12:01:51.821049 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.819140    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ffda1903-e8ae-4139-bb4c-d01904b23925-cnibin\") pod \"multus-additional-cni-plugins-hrzgb\" (UID: \"ffda1903-e8ae-4139-bb4c-d01904b23925\") " pod="openshift-multus/multus-additional-cni-plugins-hrzgb"
Jul 08 12:01:51.821917 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.819176    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b6d8afe0-1555-462f-921e-eaa3d7acdb7a-host-cni-bin\") pod \"ovnkube-node-xkp4b\" (UID: \"b6d8afe0-1555-462f-921e-eaa3d7acdb7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkp4b"
Jul 08 12:01:51.821917 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.819232    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ffda1903-e8ae-4139-bb4c-d01904b23925-os-release\") pod \"multus-additional-cni-plugins-hrzgb\" (UID: \"ffda1903-e8ae-4139-bb4c-d01904b23925\") " pod="openshift-multus/multus-additional-cni-plugins-hrzgb"
Jul 08 12:01:51.821917 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.819237    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b6d8afe0-1555-462f-921e-eaa3d7acdb7a-etc-openvswitch\") pod \"ovnkube-node-xkp4b\" (UID: \"b6d8afe0-1555-462f-921e-eaa3d7acdb7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkp4b"
Jul 08 12:01:51.821917 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.819259    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c08cf95f-760d-49d9-8b97-b7cb64dc6c95-registration-dir\") pod \"aws-ebs-csi-driver-node-wwvt9\" (UID: \"c08cf95f-760d-49d9-8b97-b7cb64dc6c95\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwvt9"
Jul 08 12:01:51.821917 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.819310    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c08cf95f-760d-49d9-8b97-b7cb64dc6c95-registration-dir\") pod \"aws-ebs-csi-driver-node-wwvt9\" (UID: \"c08cf95f-760d-49d9-8b97-b7cb64dc6c95\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwvt9"
Jul 08 12:01:51.821917 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.819310    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b6d8afe0-1555-462f-921e-eaa3d7acdb7a-etc-openvswitch\") pod \"ovnkube-node-xkp4b\" (UID: \"b6d8afe0-1555-462f-921e-eaa3d7acdb7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkp4b"
Jul 08 12:01:51.821917 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.819343    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e37bfd95-6a30-453c-be17-846c687e4d9e-host-slash\") pod \"iptables-alerter-5wpmj\" (UID: \"e37bfd95-6a30-453c-be17-846c687e4d9e\") " pod="openshift-network-operator/iptables-alerter-5wpmj"
Jul 08 12:01:51.821917 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.819408    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c0abdbfc-0312-4fd1-9ca4-098e3fa3e7e1-hosts-file\") pod \"node-resolver-26hpb\" (UID: \"c0abdbfc-0312-4fd1-9ca4-098e3fa3e7e1\") " pod="openshift-dns/node-resolver-26hpb"
Jul 08 12:01:51.821917 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.819433    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/56220650-f743-4804-9e3e-5a0cffd4fd3a-konnectivity-ca\") pod \"konnectivity-agent-pp86n\" (UID: \"56220650-f743-4804-9e3e-5a0cffd4fd3a\") " pod="kube-system/konnectivity-agent-pp86n"
Jul 08 12:01:51.821917 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.819447    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c0abdbfc-0312-4fd1-9ca4-098e3fa3e7e1-hosts-file\") pod \"node-resolver-26hpb\" (UID: \"c0abdbfc-0312-4fd1-9ca4-098e3fa3e7e1\") " pod="openshift-dns/node-resolver-26hpb"
Jul 08 12:01:51.821917 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.819449    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e37bfd95-6a30-453c-be17-846c687e4d9e-host-slash\") pod \"iptables-alerter-5wpmj\" (UID: \"e37bfd95-6a30-453c-be17-846c687e4d9e\") " pod="openshift-network-operator/iptables-alerter-5wpmj"
Jul 08 12:01:51.821917 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.819468    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/636b6729-2bf5-43f2-8db0-c1052576d023-system-cni-dir\") pod \"multus-vm6ql\" (UID: \"636b6729-2bf5-43f2-8db0-c1052576d023\") " pod="openshift-multus/multus-vm6ql"
Jul 08 12:01:51.821917 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.819490    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/636b6729-2bf5-43f2-8db0-c1052576d023-multus-cni-dir\") pod \"multus-vm6ql\" (UID: \"636b6729-2bf5-43f2-8db0-c1052576d023\") " pod="openshift-multus/multus-vm6ql"
Jul 08 12:01:51.821917 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.819511    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/636b6729-2bf5-43f2-8db0-c1052576d023-cnibin\") pod \"multus-vm6ql\" (UID: \"636b6729-2bf5-43f2-8db0-c1052576d023\") " pod="openshift-multus/multus-vm6ql"
Jul 08 12:01:51.821917 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.819529    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/24f4637c-726e-45ae-991a-c517b065e0cc-etc-sysctl-conf\") pod \"tuned-2jgc8\" (UID: \"24f4637c-726e-45ae-991a-c517b065e0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgc8"
Jul 08 12:01:51.821917 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.819549    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b6d8afe0-1555-462f-921e-eaa3d7acdb7a-host-run-ovn-kubernetes\") pod \"ovnkube-node-xkp4b\" (UID: \"b6d8afe0-1555-462f-921e-eaa3d7acdb7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkp4b"
Jul 08 12:01:51.821917 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.819562    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/636b6729-2bf5-43f2-8db0-c1052576d023-cnibin\") pod \"multus-vm6ql\" (UID: \"636b6729-2bf5-43f2-8db0-c1052576d023\") " pod="openshift-multus/multus-vm6ql"
Jul 08 12:01:51.821917 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.819567    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24f4637c-726e-45ae-991a-c517b065e0cc-tmp\") pod \"tuned-2jgc8\" (UID: \"24f4637c-726e-45ae-991a-c517b065e0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgc8"
Jul 08 12:01:51.822570 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.819587    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/636b6729-2bf5-43f2-8db0-c1052576d023-host-run-netns\") pod \"multus-vm6ql\" (UID: \"636b6729-2bf5-43f2-8db0-c1052576d023\") " pod="openshift-multus/multus-vm6ql"
Jul 08 12:01:51.822570 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.819605    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/636b6729-2bf5-43f2-8db0-c1052576d023-hostroot\") pod \"multus-vm6ql\" (UID: \"636b6729-2bf5-43f2-8db0-c1052576d023\") " pod="openshift-multus/multus-vm6ql"
Jul 08 12:01:51.822570 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.819612    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/636b6729-2bf5-43f2-8db0-c1052576d023-system-cni-dir\") pod \"multus-vm6ql\" (UID: \"636b6729-2bf5-43f2-8db0-c1052576d023\") " pod="openshift-multus/multus-vm6ql"
Jul 08 12:01:51.822570 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.819625    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ffda1903-e8ae-4139-bb4c-d01904b23925-cni-binary-copy\") pod \"multus-additional-cni-plugins-hrzgb\" (UID: \"ffda1903-e8ae-4139-bb4c-d01904b23925\") " pod="openshift-multus/multus-additional-cni-plugins-hrzgb"
Jul 08 12:01:51.822570 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.819642    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/24f4637c-726e-45ae-991a-c517b065e0cc-etc-sysctl-conf\") pod \"tuned-2jgc8\" (UID: \"24f4637c-726e-45ae-991a-c517b065e0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgc8"
Jul 08 12:01:51.822570 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.819641    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b6d8afe0-1555-462f-921e-eaa3d7acdb7a-host-run-ovn-kubernetes\") pod \"ovnkube-node-xkp4b\" (UID: \"b6d8afe0-1555-462f-921e-eaa3d7acdb7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkp4b"
Jul 08 12:01:51.822570 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.819644    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ffda1903-e8ae-4139-bb4c-d01904b23925-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hrzgb\" (UID: \"ffda1903-e8ae-4139-bb4c-d01904b23925\") " pod="openshift-multus/multus-additional-cni-plugins-hrzgb"
Jul 08 12:01:51.822570 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.819674    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/636b6729-2bf5-43f2-8db0-c1052576d023-host-var-lib-cni-bin\") pod \"multus-vm6ql\" (UID: \"636b6729-2bf5-43f2-8db0-c1052576d023\") " pod="openshift-multus/multus-vm6ql"
Jul 08 12:01:51.822570 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.819677    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/636b6729-2bf5-43f2-8db0-c1052576d023-hostroot\") pod \"multus-vm6ql\" (UID: \"636b6729-2bf5-43f2-8db0-c1052576d023\") " pod="openshift-multus/multus-vm6ql"
Jul 08 12:01:51.822570 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.819691    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/636b6729-2bf5-43f2-8db0-c1052576d023-etc-kubernetes\") pod \"multus-vm6ql\" (UID: \"636b6729-2bf5-43f2-8db0-c1052576d023\") " pod="openshift-multus/multus-vm6ql"
Jul 08 12:01:51.822570 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.819706    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq5n6\" (UniqueName: \"kubernetes.io/projected/c08cf95f-760d-49d9-8b97-b7cb64dc6c95-kube-api-access-cq5n6\") pod \"aws-ebs-csi-driver-node-wwvt9\" (UID: \"c08cf95f-760d-49d9-8b97-b7cb64dc6c95\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwvt9"
Jul 08 12:01:51.822570 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.819720    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/636b6729-2bf5-43f2-8db0-c1052576d023-host-run-netns\") pod \"multus-vm6ql\" (UID: \"636b6729-2bf5-43f2-8db0-c1052576d023\") " pod="openshift-multus/multus-vm6ql"
Jul 08 12:01:51.822570 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.819723    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z6tq\" (UniqueName: \"kubernetes.io/projected/641b596d-df5a-491a-a40e-e269de0a676e-kube-api-access-5z6tq\") pod \"node-ca-k4jsh\" (UID: \"641b596d-df5a-491a-a40e-e269de0a676e\") " pod="openshift-image-registry/node-ca-k4jsh"
Jul 08 12:01:51.822570 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.819746    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/636b6729-2bf5-43f2-8db0-c1052576d023-host-var-lib-cni-bin\") pod \"multus-vm6ql\" (UID: \"636b6729-2bf5-43f2-8db0-c1052576d023\") " pod="openshift-multus/multus-vm6ql"
Jul 08 12:01:51.822570 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.819758    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/24f4637c-726e-45ae-991a-c517b065e0cc-etc-systemd\") pod \"tuned-2jgc8\" (UID: \"24f4637c-726e-45ae-991a-c517b065e0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgc8"
Jul 08 12:01:51.822570 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.819791    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/24f4637c-726e-45ae-991a-c517b065e0cc-run\") pod \"tuned-2jgc8\" (UID: \"24f4637c-726e-45ae-991a-c517b065e0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgc8"
Jul 08 12:01:51.822570 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.819808    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/636b6729-2bf5-43f2-8db0-c1052576d023-multus-socket-dir-parent\") pod \"multus-vm6ql\" (UID: \"636b6729-2bf5-43f2-8db0-c1052576d023\") " pod="openshift-multus/multus-vm6ql"
Jul 08 12:01:51.822570 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.819831    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/636b6729-2bf5-43f2-8db0-c1052576d023-multus-daemon-config\") pod \"multus-vm6ql\" (UID: \"636b6729-2bf5-43f2-8db0-c1052576d023\") " pod="openshift-multus/multus-vm6ql"
Jul 08 12:01:51.823088 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.819846    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ffda1903-e8ae-4139-bb4c-d01904b23925-system-cni-dir\") pod \"multus-additional-cni-plugins-hrzgb\" (UID: \"ffda1903-e8ae-4139-bb4c-d01904b23925\") " pod="openshift-multus/multus-additional-cni-plugins-hrzgb"
Jul 08 12:01:51.823088 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.819860    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b6d8afe0-1555-462f-921e-eaa3d7acdb7a-ovnkube-config\") pod \"ovnkube-node-xkp4b\" (UID: \"b6d8afe0-1555-462f-921e-eaa3d7acdb7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkp4b"
Jul 08 12:01:51.823088 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.819875    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c08cf95f-760d-49d9-8b97-b7cb64dc6c95-socket-dir\") pod \"aws-ebs-csi-driver-node-wwvt9\" (UID: \"c08cf95f-760d-49d9-8b97-b7cb64dc6c95\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwvt9"
Jul 08 12:01:51.823088 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.819891    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c08cf95f-760d-49d9-8b97-b7cb64dc6c95-etc-selinux\") pod \"aws-ebs-csi-driver-node-wwvt9\" (UID: \"c08cf95f-760d-49d9-8b97-b7cb64dc6c95\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwvt9"
Jul 08 12:01:51.823088 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.819913    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/24f4637c-726e-45ae-991a-c517b065e0cc-host\") pod \"tuned-2jgc8\" (UID: \"24f4637c-726e-45ae-991a-c517b065e0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgc8"
Jul 08 12:01:51.823088 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.819936    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/24f4637c-726e-45ae-991a-c517b065e0cc-run\") pod \"tuned-2jgc8\" (UID: \"24f4637c-726e-45ae-991a-c517b065e0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgc8"
Jul 08 12:01:51.823088 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.819937    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdj4j\" (UniqueName: \"kubernetes.io/projected/ecbe5ccd-c4c5-4820-9238-72591f011842-kube-api-access-sdj4j\") pod \"network-check-target-842d7\" (UID: \"ecbe5ccd-c4c5-4820-9238-72591f011842\") " pod="openshift-network-diagnostics/network-check-target-842d7"
Jul 08 12:01:51.823088 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.819567    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/636b6729-2bf5-43f2-8db0-c1052576d023-multus-cni-dir\") pod \"multus-vm6ql\" (UID: \"636b6729-2bf5-43f2-8db0-c1052576d023\") " pod="openshift-multus/multus-vm6ql"
Jul 08 12:01:51.823088 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.819982    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c08cf95f-760d-49d9-8b97-b7cb64dc6c95-device-dir\") pod \"aws-ebs-csi-driver-node-wwvt9\" (UID: \"c08cf95f-760d-49d9-8b97-b7cb64dc6c95\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwvt9"
Jul 08 12:01:51.823088 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.820014    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/636b6729-2bf5-43f2-8db0-c1052576d023-cni-binary-copy\") pod \"multus-vm6ql\" (UID: \"636b6729-2bf5-43f2-8db0-c1052576d023\") " pod="openshift-multus/multus-vm6ql"
Jul 08 12:01:51.823088 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.820038    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9e4572a9-69d7-418c-b751-07c5d6759d5a-kubelet-config\") pod \"global-pull-secret-syncer-cslr6\" (UID: \"9e4572a9-69d7-418c-b751-07c5d6759d5a\") " pod="kube-system/global-pull-secret-syncer-cslr6"
Jul 08 12:01:51.823088 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.820063    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b6d8afe0-1555-462f-921e-eaa3d7acdb7a-log-socket\") pod \"ovnkube-node-xkp4b\" (UID: \"b6d8afe0-1555-462f-921e-eaa3d7acdb7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkp4b"
Jul 08 12:01:51.823088 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.820085    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/641b596d-df5a-491a-a40e-e269de0a676e-serviceca\") pod \"node-ca-k4jsh\" (UID: \"641b596d-df5a-491a-a40e-e269de0a676e\") " pod="openshift-image-registry/node-ca-k4jsh"
Jul 08 12:01:51.823088 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.820100    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ffda1903-e8ae-4139-bb4c-d01904b23925-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hrzgb\" (UID: \"ffda1903-e8ae-4139-bb4c-d01904b23925\") " pod="openshift-multus/multus-additional-cni-plugins-hrzgb"
Jul 08 12:01:51.823088 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.820109    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b6d8afe0-1555-462f-921e-eaa3d7acdb7a-systemd-units\") pod \"ovnkube-node-xkp4b\" (UID: \"b6d8afe0-1555-462f-921e-eaa3d7acdb7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkp4b"
Jul 08 12:01:51.823088 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.820132    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b6d8afe0-1555-462f-921e-eaa3d7acdb7a-run-systemd\") pod \"ovnkube-node-xkp4b\" (UID: \"b6d8afe0-1555-462f-921e-eaa3d7acdb7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkp4b"
Jul 08 12:01:51.823088 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.820135    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ffda1903-e8ae-4139-bb4c-d01904b23925-cni-binary-copy\") pod \"multus-additional-cni-plugins-hrzgb\" (UID: \"ffda1903-e8ae-4139-bb4c-d01904b23925\") " pod="openshift-multus/multus-additional-cni-plugins-hrzgb"
Jul 08 12:01:51.823626 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.820156    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c08cf95f-760d-49d9-8b97-b7cb64dc6c95-device-dir\") pod \"aws-ebs-csi-driver-node-wwvt9\" (UID: \"c08cf95f-760d-49d9-8b97-b7cb64dc6c95\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwvt9"
Jul 08 12:01:51.823626 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.820202    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/636b6729-2bf5-43f2-8db0-c1052576d023-etc-kubernetes\") pod \"multus-vm6ql\" (UID: \"636b6729-2bf5-43f2-8db0-c1052576d023\") " pod="openshift-multus/multus-vm6ql"
Jul 08 12:01:51.823626 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.820207    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b6d8afe0-1555-462f-921e-eaa3d7acdb7a-run-systemd\") pod \"ovnkube-node-xkp4b\" (UID: \"b6d8afe0-1555-462f-921e-eaa3d7acdb7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkp4b"
Jul 08 12:01:51.823626 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.820232    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b6d8afe0-1555-462f-921e-eaa3d7acdb7a-host-cni-netd\") pod \"ovnkube-node-xkp4b\" (UID: \"b6d8afe0-1555-462f-921e-eaa3d7acdb7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkp4b"
Jul 08 12:01:51.823626 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.820260    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/24f4637c-726e-45ae-991a-c517b065e0cc-etc-tuned\") pod \"tuned-2jgc8\" (UID: \"24f4637c-726e-45ae-991a-c517b065e0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgc8"
Jul 08 12:01:51.823626 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.820285    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/24f4637c-726e-45ae-991a-c517b065e0cc-etc-kubernetes\") pod \"tuned-2jgc8\" (UID: \"24f4637c-726e-45ae-991a-c517b065e0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgc8"
Jul 08 12:01:51.823626 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.820310    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a64d26d8-d849-4f64-b43d-3a9133970f52-metrics-certs\") pod \"network-metrics-daemon-m5628\" (UID: \"a64d26d8-d849-4f64-b43d-3a9133970f52\") " pod="openshift-multus/network-metrics-daemon-m5628"
Jul 08 12:01:51.823626 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.820333    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b6d8afe0-1555-462f-921e-eaa3d7acdb7a-host-kubelet\") pod \"ovnkube-node-xkp4b\" (UID: \"b6d8afe0-1555-462f-921e-eaa3d7acdb7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkp4b"
Jul 08 12:01:51.823626 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.820356    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b6d8afe0-1555-462f-921e-eaa3d7acdb7a-host-slash\") pod \"ovnkube-node-xkp4b\" (UID: \"b6d8afe0-1555-462f-921e-eaa3d7acdb7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkp4b"
Jul 08 12:01:51.823626 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.820380    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b6d8afe0-1555-462f-921e-eaa3d7acdb7a-host-run-netns\") pod \"ovnkube-node-xkp4b\" (UID: \"b6d8afe0-1555-462f-921e-eaa3d7acdb7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkp4b"
Jul 08 12:01:51.823626 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.820401    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b6d8afe0-1555-462f-921e-eaa3d7acdb7a-run-openvswitch\") pod \"ovnkube-node-xkp4b\" (UID: \"b6d8afe0-1555-462f-921e-eaa3d7acdb7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkp4b"
Jul 08 12:01:51.823626 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.820416    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b6d8afe0-1555-462f-921e-eaa3d7acdb7a-ovnkube-config\") pod \"ovnkube-node-xkp4b\" (UID: \"b6d8afe0-1555-462f-921e-eaa3d7acdb7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkp4b"
Jul 08 12:01:51.823626 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.820430    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqzjw\" (UniqueName: \"kubernetes.io/projected/b6d8afe0-1555-462f-921e-eaa3d7acdb7a-kube-api-access-cqzjw\") pod \"ovnkube-node-xkp4b\" (UID: \"b6d8afe0-1555-462f-921e-eaa3d7acdb7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkp4b"
Jul 08 12:01:51.823626 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.820453    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c08cf95f-760d-49d9-8b97-b7cb64dc6c95-sys-fs\") pod \"aws-ebs-csi-driver-node-wwvt9\" (UID: \"c08cf95f-760d-49d9-8b97-b7cb64dc6c95\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwvt9"
Jul 08 12:01:51.823626 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.820468    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ffda1903-e8ae-4139-bb4c-d01904b23925-system-cni-dir\") pod \"multus-additional-cni-plugins-hrzgb\" (UID: \"ffda1903-e8ae-4139-bb4c-d01904b23925\") " pod="openshift-multus/multus-additional-cni-plugins-hrzgb"
Jul 08 12:01:51.823626 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.820478    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/24f4637c-726e-45ae-991a-c517b065e0cc-etc-sysconfig\") pod \"tuned-2jgc8\" (UID: \"24f4637c-726e-45ae-991a-c517b065e0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgc8"
Jul 08 12:01:51.823626 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.820517    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/24f4637c-726e-45ae-991a-c517b065e0cc-etc-systemd\") pod \"tuned-2jgc8\" (UID: \"24f4637c-726e-45ae-991a-c517b065e0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgc8"
Jul 08 12:01:51.824205 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.820554    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b6d8afe0-1555-462f-921e-eaa3d7acdb7a-host-kubelet\") pod \"ovnkube-node-xkp4b\" (UID: \"b6d8afe0-1555-462f-921e-eaa3d7acdb7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkp4b"
Jul 08 12:01:51.824205 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.820556    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/24f4637c-726e-45ae-991a-c517b065e0cc-etc-sysconfig\") pod \"tuned-2jgc8\" (UID: \"24f4637c-726e-45ae-991a-c517b065e0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgc8"
Jul 08 12:01:51.824205 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.820552    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c08cf95f-760d-49d9-8b97-b7cb64dc6c95-etc-selinux\") pod \"aws-ebs-csi-driver-node-wwvt9\" (UID: \"c08cf95f-760d-49d9-8b97-b7cb64dc6c95\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwvt9"
Jul 08 12:01:51.824205 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.820593    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b6d8afe0-1555-462f-921e-eaa3d7acdb7a-log-socket\") pod \"ovnkube-node-xkp4b\" (UID: \"b6d8afe0-1555-462f-921e-eaa3d7acdb7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkp4b"
Jul 08 12:01:51.824205 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.820601    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b6d8afe0-1555-462f-921e-eaa3d7acdb7a-host-run-netns\") pod \"ovnkube-node-xkp4b\" (UID: \"b6d8afe0-1555-462f-921e-eaa3d7acdb7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkp4b"
Jul 08 12:01:51.824205 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.820639    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b6d8afe0-1555-462f-921e-eaa3d7acdb7a-run-openvswitch\") pod \"ovnkube-node-xkp4b\" (UID: \"b6d8afe0-1555-462f-921e-eaa3d7acdb7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkp4b"
Jul 08 12:01:51.824205 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.820652    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c08cf95f-760d-49d9-8b97-b7cb64dc6c95-socket-dir\") pod \"aws-ebs-csi-driver-node-wwvt9\" (UID: \"c08cf95f-760d-49d9-8b97-b7cb64dc6c95\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwvt9"
Jul 08 12:01:51.824205 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.820652    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9e4572a9-69d7-418c-b751-07c5d6759d5a-kubelet-config\") pod \"global-pull-secret-syncer-cslr6\" (UID: \"9e4572a9-69d7-418c-b751-07c5d6759d5a\") " pod="kube-system/global-pull-secret-syncer-cslr6"
Jul 08 12:01:51.824205 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.820696    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c08cf95f-760d-49d9-8b97-b7cb64dc6c95-sys-fs\") pod \"aws-ebs-csi-driver-node-wwvt9\" (UID: \"c08cf95f-760d-49d9-8b97-b7cb64dc6c95\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwvt9"
Jul 08 12:01:51.824205 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.820729    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b6d8afe0-1555-462f-921e-eaa3d7acdb7a-host-cni-netd\") pod \"ovnkube-node-xkp4b\" (UID: \"b6d8afe0-1555-462f-921e-eaa3d7acdb7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkp4b"
Jul 08 12:01:51.824205 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.820760    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/24f4637c-726e-45ae-991a-c517b065e0cc-host\") pod \"tuned-2jgc8\" (UID: \"24f4637c-726e-45ae-991a-c517b065e0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgc8"
Jul 08 12:01:51.824205 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.820791    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/636b6729-2bf5-43f2-8db0-c1052576d023-multus-daemon-config\") pod \"multus-vm6ql\" (UID: \"636b6729-2bf5-43f2-8db0-c1052576d023\") " pod="openshift-multus/multus-vm6ql"
Jul 08 12:01:51.824205 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.820818    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b6d8afe0-1555-462f-921e-eaa3d7acdb7a-host-slash\") pod \"ovnkube-node-xkp4b\" (UID: \"b6d8afe0-1555-462f-921e-eaa3d7acdb7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkp4b"
Jul 08 12:01:51.824205 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:01:51.820830    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 12:01:51.824205 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.820868    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/24f4637c-726e-45ae-991a-c517b065e0cc-etc-kubernetes\") pod \"tuned-2jgc8\" (UID: \"24f4637c-726e-45ae-991a-c517b065e0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgc8"
Jul 08 12:01:51.824205 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.820869    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/636b6729-2bf5-43f2-8db0-c1052576d023-multus-socket-dir-parent\") pod \"multus-vm6ql\" (UID: \"636b6729-2bf5-43f2-8db0-c1052576d023\") " pod="openshift-multus/multus-vm6ql"
Jul 08 12:01:51.824205 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:01:51.820885    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a64d26d8-d849-4f64-b43d-3a9133970f52-metrics-certs podName:a64d26d8-d849-4f64-b43d-3a9133970f52 nodeName:}" failed. No retries permitted until 2026-07-08 12:01:52.320869953 +0000 UTC m=+2.051244904 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a64d26d8-d849-4f64-b43d-3a9133970f52-metrics-certs") pod "network-metrics-daemon-m5628" (UID: "a64d26d8-d849-4f64-b43d-3a9133970f52") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 12:01:51.824695 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.820911    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b6d8afe0-1555-462f-921e-eaa3d7acdb7a-systemd-units\") pod \"ovnkube-node-xkp4b\" (UID: \"b6d8afe0-1555-462f-921e-eaa3d7acdb7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkp4b"
Jul 08 12:01:51.824695 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.821391    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/636b6729-2bf5-43f2-8db0-c1052576d023-cni-binary-copy\") pod \"multus-vm6ql\" (UID: \"636b6729-2bf5-43f2-8db0-c1052576d023\") " pod="openshift-multus/multus-vm6ql"
Jul 08 12:01:51.824695 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.821582    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/641b596d-df5a-491a-a40e-e269de0a676e-serviceca\") pod \"node-ca-k4jsh\" (UID: \"641b596d-df5a-491a-a40e-e269de0a676e\") " pod="openshift-image-registry/node-ca-k4jsh"
Jul 08 12:01:51.824695 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.822481    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b6d8afe0-1555-462f-921e-eaa3d7acdb7a-ovn-node-metrics-cert\") pod \"ovnkube-node-xkp4b\" (UID: \"b6d8afe0-1555-462f-921e-eaa3d7acdb7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkp4b"
Jul 08 12:01:51.824695 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.822634    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/56220650-f743-4804-9e3e-5a0cffd4fd3a-agent-certs\") pod \"konnectivity-agent-pp86n\" (UID: \"56220650-f743-4804-9e3e-5a0cffd4fd3a\") " pod="kube-system/konnectivity-agent-pp86n"
Jul 08 12:01:51.824695 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.823192    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24f4637c-726e-45ae-991a-c517b065e0cc-tmp\") pod \"tuned-2jgc8\" (UID: \"24f4637c-726e-45ae-991a-c517b065e0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgc8"
Jul 08 12:01:51.824695 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.823244    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/24f4637c-726e-45ae-991a-c517b065e0cc-etc-tuned\") pod \"tuned-2jgc8\" (UID: \"24f4637c-726e-45ae-991a-c517b065e0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgc8"
Jul 08 12:01:51.832505 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:01:51.832476    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 12:01:51.832696 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:01:51.832663    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 12:01:51.832845 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:01:51.832702    2579 projected.go:196] Error preparing data for projected volume kube-api-access-sdj4j for pod openshift-network-diagnostics/network-check-target-842d7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 12:01:51.832997 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:01:51.832951    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ecbe5ccd-c4c5-4820-9238-72591f011842-kube-api-access-sdj4j podName:ecbe5ccd-c4c5-4820-9238-72591f011842 nodeName:}" failed. No retries permitted until 2026-07-08 12:01:52.332763842 +0000 UTC m=+2.063138815 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-sdj4j" (UniqueName: "kubernetes.io/projected/ecbe5ccd-c4c5-4820-9238-72591f011842-kube-api-access-sdj4j") pod "network-check-target-842d7" (UID: "ecbe5ccd-c4c5-4820-9238-72591f011842") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 12:01:51.833246 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.833191    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc6jr\" (UniqueName: \"kubernetes.io/projected/24f4637c-726e-45ae-991a-c517b065e0cc-kube-api-access-nc6jr\") pod \"tuned-2jgc8\" (UID: \"24f4637c-726e-45ae-991a-c517b065e0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgc8"
Jul 08 12:01:51.833861 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.833839    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9f9c\" (UniqueName: \"kubernetes.io/projected/e37bfd95-6a30-453c-be17-846c687e4d9e-kube-api-access-h9f9c\") pod \"iptables-alerter-5wpmj\" (UID: \"e37bfd95-6a30-453c-be17-846c687e4d9e\") " pod="openshift-network-operator/iptables-alerter-5wpmj"
Jul 08 12:01:51.835167 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.835137    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbt58\" (UniqueName: \"kubernetes.io/projected/ffda1903-e8ae-4139-bb4c-d01904b23925-kube-api-access-kbt58\") pod \"multus-additional-cni-plugins-hrzgb\" (UID: \"ffda1903-e8ae-4139-bb4c-d01904b23925\") " pod="openshift-multus/multus-additional-cni-plugins-hrzgb"
Jul 08 12:01:51.835678 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.835277    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwwxw\" (UniqueName: \"kubernetes.io/projected/c0abdbfc-0312-4fd1-9ca4-098e3fa3e7e1-kube-api-access-fwwxw\") pod \"node-resolver-26hpb\" (UID: \"c0abdbfc-0312-4fd1-9ca4-098e3fa3e7e1\") " pod="openshift-dns/node-resolver-26hpb"
Jul 08 12:01:51.835678 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.835480    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq5n6\" (UniqueName: \"kubernetes.io/projected/c08cf95f-760d-49d9-8b97-b7cb64dc6c95-kube-api-access-cq5n6\") pod \"aws-ebs-csi-driver-node-wwvt9\" (UID: \"c08cf95f-760d-49d9-8b97-b7cb64dc6c95\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwvt9"
Jul 08 12:01:51.835678 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.835512    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hckqt\" (UniqueName: \"kubernetes.io/projected/a64d26d8-d849-4f64-b43d-3a9133970f52-kube-api-access-hckqt\") pod \"network-metrics-daemon-m5628\" (UID: \"a64d26d8-d849-4f64-b43d-3a9133970f52\") " pod="openshift-multus/network-metrics-daemon-m5628"
Jul 08 12:01:51.835678 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.835635    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqzjw\" (UniqueName: \"kubernetes.io/projected/b6d8afe0-1555-462f-921e-eaa3d7acdb7a-kube-api-access-cqzjw\") pod \"ovnkube-node-xkp4b\" (UID: \"b6d8afe0-1555-462f-921e-eaa3d7acdb7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkp4b"
Jul 08 12:01:51.835916 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.835843    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48n2w\" (UniqueName: \"kubernetes.io/projected/636b6729-2bf5-43f2-8db0-c1052576d023-kube-api-access-48n2w\") pod \"multus-vm6ql\" (UID: \"636b6729-2bf5-43f2-8db0-c1052576d023\") " pod="openshift-multus/multus-vm6ql"
Jul 08 12:01:51.836395 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.836381    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z6tq\" (UniqueName: \"kubernetes.io/projected/641b596d-df5a-491a-a40e-e269de0a676e-kube-api-access-5z6tq\") pod \"node-ca-k4jsh\" (UID: \"641b596d-df5a-491a-a40e-e269de0a676e\") " pod="openshift-image-registry/node-ca-k4jsh"
Jul 08 12:01:51.894712 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.894676    2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 08 12:01:51.934652 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.934571    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hrzgb"
Jul 08 12:01:51.943871 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:51.943844    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffda1903_e8ae_4139_bb4c_d01904b23925.slice/crio-c17e1a5443c7b2b4ca28e359c5de055249204446a4501608450d8f2407f4538f WatchSource:0}: Error finding container c17e1a5443c7b2b4ca28e359c5de055249204446a4501608450d8f2407f4538f: Status 404 returned error can't find the container with id c17e1a5443c7b2b4ca28e359c5de055249204446a4501608450d8f2407f4538f
Jul 08 12:01:51.966295 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.966261    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwvt9"
Jul 08 12:01:51.974036 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:51.974011    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc08cf95f_760d_49d9_8b97_b7cb64dc6c95.slice/crio-52d44dd27783050959f685309af6885059ebe379dcbbe1b3a422bf937e593dc9 WatchSource:0}: Error finding container 52d44dd27783050959f685309af6885059ebe379dcbbe1b3a422bf937e593dc9: Status 404 returned error can't find the container with id 52d44dd27783050959f685309af6885059ebe379dcbbe1b3a422bf937e593dc9
Jul 08 12:01:51.981306 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.981282    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2jgc8"
Jul 08 12:01:51.988900 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:51.988877    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24f4637c_726e_45ae_991a_c517b065e0cc.slice/crio-802c4ec8a208092607b5618efd46dbb91550ec172d224d8a8d79819a3c0e563c WatchSource:0}: Error finding container 802c4ec8a208092607b5618efd46dbb91550ec172d224d8a8d79819a3c0e563c: Status 404 returned error can't find the container with id 802c4ec8a208092607b5618efd46dbb91550ec172d224d8a8d79819a3c0e563c
Jul 08 12:01:51.998101 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:51.998079    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5wpmj"
Jul 08 12:01:52.005631 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:52.005596    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode37bfd95_6a30_453c_be17_846c687e4d9e.slice/crio-c4ef7390c3804248e162f7d58025bb9e42af1ad42279ac9925211894bc08f272 WatchSource:0}: Error finding container c4ef7390c3804248e162f7d58025bb9e42af1ad42279ac9925211894bc08f272: Status 404 returned error can't find the container with id c4ef7390c3804248e162f7d58025bb9e42af1ad42279ac9925211894bc08f272
Jul 08 12:01:52.024509 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:52.024474    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-k4jsh"
Jul 08 12:01:52.031134 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:52.031114    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-26hpb"
Jul 08 12:01:52.033107 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:52.033073    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod641b596d_df5a_491a_a40e_e269de0a676e.slice/crio-b5b009017e57f3bf36b2b4bb3447a98c45ea726a5058d1b89fb5f79e5f68b385 WatchSource:0}: Error finding container b5b009017e57f3bf36b2b4bb3447a98c45ea726a5058d1b89fb5f79e5f68b385: Status 404 returned error can't find the container with id b5b009017e57f3bf36b2b4bb3447a98c45ea726a5058d1b89fb5f79e5f68b385
Jul 08 12:01:52.035508 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:52.035492    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xkp4b"
Jul 08 12:01:52.038798 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:52.038751    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0abdbfc_0312_4fd1_9ca4_098e3fa3e7e1.slice/crio-36074ae4ae2b1ca192f0fa9ccf3ee838710f9990caebea4076b45e85b7733c40 WatchSource:0}: Error finding container 36074ae4ae2b1ca192f0fa9ccf3ee838710f9990caebea4076b45e85b7733c40: Status 404 returned error can't find the container with id 36074ae4ae2b1ca192f0fa9ccf3ee838710f9990caebea4076b45e85b7733c40
Jul 08 12:01:52.041994 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:52.041974    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pp86n"
Jul 08 12:01:52.043611 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:52.043589    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6d8afe0_1555_462f_921e_eaa3d7acdb7a.slice/crio-06ed1edf81749c4a0a1f28abadbd948e0994e5e4246bc15896218df5a2042bf4 WatchSource:0}: Error finding container 06ed1edf81749c4a0a1f28abadbd948e0994e5e4246bc15896218df5a2042bf4: Status 404 returned error can't find the container with id 06ed1edf81749c4a0a1f28abadbd948e0994e5e4246bc15896218df5a2042bf4
Jul 08 12:01:52.045675 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:52.045655    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vm6ql"
Jul 08 12:01:52.052442 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:52.052416    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56220650_f743_4804_9e3e_5a0cffd4fd3a.slice/crio-946ef7989ea156ca563a291969c1990fb0362a37ac96d242b8cdffd2e9436809 WatchSource:0}: Error finding container 946ef7989ea156ca563a291969c1990fb0362a37ac96d242b8cdffd2e9436809: Status 404 returned error can't find the container with id 946ef7989ea156ca563a291969c1990fb0362a37ac96d242b8cdffd2e9436809
Jul 08 12:01:52.054650 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:01:52.054625    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod636b6729_2bf5_43f2_8db0_c1052576d023.slice/crio-9beb8cca8d6e184def33f21fb9ee51c3ea392422bb4f369e19d1708e02b26604 WatchSource:0}: Error finding container 9beb8cca8d6e184def33f21fb9ee51c3ea392422bb4f369e19d1708e02b26604: Status 404 returned error can't find the container with id 9beb8cca8d6e184def33f21fb9ee51c3ea392422bb4f369e19d1708e02b26604
Jul 08 12:01:52.326092 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:52.326006    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a64d26d8-d849-4f64-b43d-3a9133970f52-metrics-certs\") pod \"network-metrics-daemon-m5628\" (UID: \"a64d26d8-d849-4f64-b43d-3a9133970f52\") " pod="openshift-multus/network-metrics-daemon-m5628"
Jul 08 12:01:52.326092 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:52.326064    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9e4572a9-69d7-418c-b751-07c5d6759d5a-original-pull-secret\") pod \"global-pull-secret-syncer-cslr6\" (UID: \"9e4572a9-69d7-418c-b751-07c5d6759d5a\") " pod="kube-system/global-pull-secret-syncer-cslr6"
Jul 08 12:01:52.326319 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:01:52.326216    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 12:01:52.326319 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:01:52.326278    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e4572a9-69d7-418c-b751-07c5d6759d5a-original-pull-secret podName:9e4572a9-69d7-418c-b751-07c5d6759d5a nodeName:}" failed. No retries permitted until 2026-07-08 12:01:53.326259215 +0000 UTC m=+3.056634180 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9e4572a9-69d7-418c-b751-07c5d6759d5a-original-pull-secret") pod "global-pull-secret-syncer-cslr6" (UID: "9e4572a9-69d7-418c-b751-07c5d6759d5a") : object "kube-system"/"original-pull-secret" not registered
Jul 08 12:01:52.326644 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:01:52.326625    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 12:01:52.326716 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:01:52.326687    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a64d26d8-d849-4f64-b43d-3a9133970f52-metrics-certs podName:a64d26d8-d849-4f64-b43d-3a9133970f52 nodeName:}" failed. No retries permitted until 2026-07-08 12:01:53.326670059 +0000 UTC m=+3.057045022 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a64d26d8-d849-4f64-b43d-3a9133970f52-metrics-certs") pod "network-metrics-daemon-m5628" (UID: "a64d26d8-d849-4f64-b43d-3a9133970f52") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 12:01:52.427071 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:52.427037    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdj4j\" (UniqueName: \"kubernetes.io/projected/ecbe5ccd-c4c5-4820-9238-72591f011842-kube-api-access-sdj4j\") pod \"network-check-target-842d7\" (UID: \"ecbe5ccd-c4c5-4820-9238-72591f011842\") " pod="openshift-network-diagnostics/network-check-target-842d7"
Jul 08 12:01:52.427246 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:01:52.427221    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 12:01:52.427246 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:01:52.427239    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 12:01:52.427246 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:01:52.427249    2579 projected.go:196] Error preparing data for projected volume kube-api-access-sdj4j for pod openshift-network-diagnostics/network-check-target-842d7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 12:01:52.427476 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:01:52.427299    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ecbe5ccd-c4c5-4820-9238-72591f011842-kube-api-access-sdj4j podName:ecbe5ccd-c4c5-4820-9238-72591f011842 nodeName:}" failed. No retries permitted until 2026-07-08 12:01:53.427285925 +0000 UTC m=+3.157660876 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-sdj4j" (UniqueName: "kubernetes.io/projected/ecbe5ccd-c4c5-4820-9238-72591f011842-kube-api-access-sdj4j") pod "network-check-target-842d7" (UID: "ecbe5ccd-c4c5-4820-9238-72591f011842") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 12:01:52.530515 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:52.530479    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 08 12:01:52.662202 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:52.662049    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-07 11:56:52 +0000 UTC" deadline="2028-01-21 04:55:01.63523213 +0000 UTC"
Jul 08 12:01:52.662202 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:52.662127    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13480h53m8.973121778s"
Jul 08 12:01:52.719373 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:52.719282    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-89.ec2.internal" event={"ID":"8e5e073dfe04675ff90daa0b47b9f43e","Type":"ContainerStarted","Data":"aad6178b3249d6654253e4810ee9be022c0dc31507ef0a855e02f0afc061ce23"}
Jul 08 12:01:52.725233 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:52.725190    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vm6ql" event={"ID":"636b6729-2bf5-43f2-8db0-c1052576d023","Type":"ContainerStarted","Data":"9beb8cca8d6e184def33f21fb9ee51c3ea392422bb4f369e19d1708e02b26604"}
Jul 08 12:01:52.737977 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:52.737935    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xkp4b" event={"ID":"b6d8afe0-1555-462f-921e-eaa3d7acdb7a","Type":"ContainerStarted","Data":"06ed1edf81749c4a0a1f28abadbd948e0994e5e4246bc15896218df5a2042bf4"}
Jul 08 12:01:52.740486 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:52.740449    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-26hpb" event={"ID":"c0abdbfc-0312-4fd1-9ca4-098e3fa3e7e1","Type":"ContainerStarted","Data":"36074ae4ae2b1ca192f0fa9ccf3ee838710f9990caebea4076b45e85b7733c40"}
Jul 08 12:01:52.743660 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:52.743624    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5wpmj" event={"ID":"e37bfd95-6a30-453c-be17-846c687e4d9e","Type":"ContainerStarted","Data":"c4ef7390c3804248e162f7d58025bb9e42af1ad42279ac9925211894bc08f272"}
Jul 08 12:01:52.748864 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:52.748822    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-89.ec2.internal" event={"ID":"01a7cd28e5cb68af10b158e7471bca6f","Type":"ContainerStarted","Data":"f490e103b2188be561e65a2499a02db4c19b5f0b9dc66a892b425cca1099986c"}
Jul 08 12:01:52.750621 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:52.750590    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pp86n" event={"ID":"56220650-f743-4804-9e3e-5a0cffd4fd3a","Type":"ContainerStarted","Data":"946ef7989ea156ca563a291969c1990fb0362a37ac96d242b8cdffd2e9436809"}
Jul 08 12:01:52.756064 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:52.756028    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-k4jsh" event={"ID":"641b596d-df5a-491a-a40e-e269de0a676e","Type":"ContainerStarted","Data":"b5b009017e57f3bf36b2b4bb3447a98c45ea726a5058d1b89fb5f79e5f68b385"}
Jul 08 12:01:52.765083 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:52.764988    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2jgc8" event={"ID":"24f4637c-726e-45ae-991a-c517b065e0cc","Type":"ContainerStarted","Data":"802c4ec8a208092607b5618efd46dbb91550ec172d224d8a8d79819a3c0e563c"}
Jul 08 12:01:52.770716 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:52.770638    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwvt9" event={"ID":"c08cf95f-760d-49d9-8b97-b7cb64dc6c95","Type":"ContainerStarted","Data":"52d44dd27783050959f685309af6885059ebe379dcbbe1b3a422bf937e593dc9"}
Jul 08 12:01:52.777461 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:52.777425    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hrzgb" event={"ID":"ffda1903-e8ae-4139-bb4c-d01904b23925","Type":"ContainerStarted","Data":"c17e1a5443c7b2b4ca28e359c5de055249204446a4501608450d8f2407f4538f"}
Jul 08 12:01:53.336070 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:53.336026    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a64d26d8-d849-4f64-b43d-3a9133970f52-metrics-certs\") pod \"network-metrics-daemon-m5628\" (UID: \"a64d26d8-d849-4f64-b43d-3a9133970f52\") " pod="openshift-multus/network-metrics-daemon-m5628"
Jul 08 12:01:53.336281 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:53.336090    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9e4572a9-69d7-418c-b751-07c5d6759d5a-original-pull-secret\") pod \"global-pull-secret-syncer-cslr6\" (UID: \"9e4572a9-69d7-418c-b751-07c5d6759d5a\") " pod="kube-system/global-pull-secret-syncer-cslr6"
Jul 08 12:01:53.336281 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:01:53.336242    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 12:01:53.336392 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:01:53.336315    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e4572a9-69d7-418c-b751-07c5d6759d5a-original-pull-secret podName:9e4572a9-69d7-418c-b751-07c5d6759d5a nodeName:}" failed. No retries permitted until 2026-07-08 12:01:55.33629447 +0000 UTC m=+5.066669445 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9e4572a9-69d7-418c-b751-07c5d6759d5a-original-pull-secret") pod "global-pull-secret-syncer-cslr6" (UID: "9e4572a9-69d7-418c-b751-07c5d6759d5a") : object "kube-system"/"original-pull-secret" not registered
Jul 08 12:01:53.336781 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:01:53.336760    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 12:01:53.336889 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:01:53.336819    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a64d26d8-d849-4f64-b43d-3a9133970f52-metrics-certs podName:a64d26d8-d849-4f64-b43d-3a9133970f52 nodeName:}" failed. No retries permitted until 2026-07-08 12:01:55.336803235 +0000 UTC m=+5.067178192 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a64d26d8-d849-4f64-b43d-3a9133970f52-metrics-certs") pod "network-metrics-daemon-m5628" (UID: "a64d26d8-d849-4f64-b43d-3a9133970f52") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 12:01:53.437559 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:53.437494    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdj4j\" (UniqueName: \"kubernetes.io/projected/ecbe5ccd-c4c5-4820-9238-72591f011842-kube-api-access-sdj4j\") pod \"network-check-target-842d7\" (UID: \"ecbe5ccd-c4c5-4820-9238-72591f011842\") " pod="openshift-network-diagnostics/network-check-target-842d7"
Jul 08 12:01:53.437775 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:01:53.437755    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 12:01:53.437832 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:01:53.437809    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 12:01:53.437911 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:01:53.437843    2579 projected.go:196] Error preparing data for projected volume kube-api-access-sdj4j for pod openshift-network-diagnostics/network-check-target-842d7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 12:01:53.437911 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:01:53.437908    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ecbe5ccd-c4c5-4820-9238-72591f011842-kube-api-access-sdj4j podName:ecbe5ccd-c4c5-4820-9238-72591f011842 nodeName:}" failed. No retries permitted until 2026-07-08 12:01:55.437887594 +0000 UTC m=+5.168262553 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-sdj4j" (UniqueName: "kubernetes.io/projected/ecbe5ccd-c4c5-4820-9238-72591f011842-kube-api-access-sdj4j") pod "network-check-target-842d7" (UID: "ecbe5ccd-c4c5-4820-9238-72591f011842") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 12:01:53.662788 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:53.662687    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-07 11:56:52 +0000 UTC" deadline="2027-12-04 00:04:19.310991243 +0000 UTC"
Jul 08 12:01:53.662788 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:53.662735    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12324h2m25.648261731s"
Jul 08 12:01:53.710866 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:53.710759    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cslr6"
Jul 08 12:01:53.711050 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:01:53.710973    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cslr6" podUID="9e4572a9-69d7-418c-b751-07c5d6759d5a"
Jul 08 12:01:53.711133 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:53.711080    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m5628"
Jul 08 12:01:53.711794 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:01:53.711182    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m5628" podUID="a64d26d8-d849-4f64-b43d-3a9133970f52"
Jul 08 12:01:53.711794 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:53.711244    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-842d7"
Jul 08 12:01:53.711794 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:01:53.711334    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-842d7" podUID="ecbe5ccd-c4c5-4820-9238-72591f011842"
Jul 08 12:01:54.069564 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:54.069465    2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 08 12:01:55.355908 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:55.355675    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a64d26d8-d849-4f64-b43d-3a9133970f52-metrics-certs\") pod \"network-metrics-daemon-m5628\" (UID: \"a64d26d8-d849-4f64-b43d-3a9133970f52\") " pod="openshift-multus/network-metrics-daemon-m5628"
Jul 08 12:01:55.356418 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:01:55.355934    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 12:01:55.356418 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:01:55.356018    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a64d26d8-d849-4f64-b43d-3a9133970f52-metrics-certs podName:a64d26d8-d849-4f64-b43d-3a9133970f52 nodeName:}" failed. No retries permitted until 2026-07-08 12:01:59.355995382 +0000 UTC m=+9.086370348 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a64d26d8-d849-4f64-b43d-3a9133970f52-metrics-certs") pod "network-metrics-daemon-m5628" (UID: "a64d26d8-d849-4f64-b43d-3a9133970f52") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 12:01:55.356418 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:01:55.356022    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 12:01:55.356418 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:01:55.356082    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e4572a9-69d7-418c-b751-07c5d6759d5a-original-pull-secret podName:9e4572a9-69d7-418c-b751-07c5d6759d5a nodeName:}" failed. No retries permitted until 2026-07-08 12:01:59.356062149 +0000 UTC m=+9.086437103 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9e4572a9-69d7-418c-b751-07c5d6759d5a-original-pull-secret") pod "global-pull-secret-syncer-cslr6" (UID: "9e4572a9-69d7-418c-b751-07c5d6759d5a") : object "kube-system"/"original-pull-secret" not registered
Jul 08 12:01:55.356418 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:55.355934    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9e4572a9-69d7-418c-b751-07c5d6759d5a-original-pull-secret\") pod \"global-pull-secret-syncer-cslr6\" (UID: \"9e4572a9-69d7-418c-b751-07c5d6759d5a\") " pod="kube-system/global-pull-secret-syncer-cslr6"
Jul 08 12:01:55.457001 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:55.456957    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdj4j\" (UniqueName: \"kubernetes.io/projected/ecbe5ccd-c4c5-4820-9238-72591f011842-kube-api-access-sdj4j\") pod \"network-check-target-842d7\" (UID: \"ecbe5ccd-c4c5-4820-9238-72591f011842\") " pod="openshift-network-diagnostics/network-check-target-842d7"
Jul 08 12:01:55.457179 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:01:55.457161    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 12:01:55.457239 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:01:55.457185    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 12:01:55.457239 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:01:55.457198    2579 projected.go:196] Error preparing data for projected volume kube-api-access-sdj4j for pod openshift-network-diagnostics/network-check-target-842d7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 12:01:55.457355 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:01:55.457261    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ecbe5ccd-c4c5-4820-9238-72591f011842-kube-api-access-sdj4j podName:ecbe5ccd-c4c5-4820-9238-72591f011842 nodeName:}" failed. No retries permitted until 2026-07-08 12:01:59.457241904 +0000 UTC m=+9.187616872 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-sdj4j" (UniqueName: "kubernetes.io/projected/ecbe5ccd-c4c5-4820-9238-72591f011842-kube-api-access-sdj4j") pod "network-check-target-842d7" (UID: "ecbe5ccd-c4c5-4820-9238-72591f011842") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 12:01:55.711173 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:55.711090    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cslr6"
Jul 08 12:01:55.711330 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:01:55.711239    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cslr6" podUID="9e4572a9-69d7-418c-b751-07c5d6759d5a"
Jul 08 12:01:55.711654 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:55.711636    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m5628"
Jul 08 12:01:55.711754 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:01:55.711735    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m5628" podUID="a64d26d8-d849-4f64-b43d-3a9133970f52"
Jul 08 12:01:55.711809 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:55.711796    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-842d7"
Jul 08 12:01:55.711893 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:01:55.711878    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-842d7" podUID="ecbe5ccd-c4c5-4820-9238-72591f011842"
Jul 08 12:01:57.711316 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:57.711273    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-842d7"
Jul 08 12:01:57.711721 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:01:57.711394    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-842d7" podUID="ecbe5ccd-c4c5-4820-9238-72591f011842"
Jul 08 12:01:57.711721 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:57.711695    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cslr6"
Jul 08 12:01:57.711839 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:01:57.711759    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cslr6" podUID="9e4572a9-69d7-418c-b751-07c5d6759d5a"
Jul 08 12:01:57.711839 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:57.711794    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m5628"
Jul 08 12:01:57.711923 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:01:57.711883    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m5628" podUID="a64d26d8-d849-4f64-b43d-3a9133970f52"
Jul 08 12:01:59.394199 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:59.394146    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a64d26d8-d849-4f64-b43d-3a9133970f52-metrics-certs\") pod \"network-metrics-daemon-m5628\" (UID: \"a64d26d8-d849-4f64-b43d-3a9133970f52\") " pod="openshift-multus/network-metrics-daemon-m5628"
Jul 08 12:01:59.394700 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:59.394209    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9e4572a9-69d7-418c-b751-07c5d6759d5a-original-pull-secret\") pod \"global-pull-secret-syncer-cslr6\" (UID: \"9e4572a9-69d7-418c-b751-07c5d6759d5a\") " pod="kube-system/global-pull-secret-syncer-cslr6"
Jul 08 12:01:59.394700 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:01:59.394347    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 12:01:59.394700 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:01:59.394350    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 12:01:59.394700 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:01:59.394415    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e4572a9-69d7-418c-b751-07c5d6759d5a-original-pull-secret podName:9e4572a9-69d7-418c-b751-07c5d6759d5a nodeName:}" failed. No retries permitted until 2026-07-08 12:02:07.394396225 +0000 UTC m=+17.124771180 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9e4572a9-69d7-418c-b751-07c5d6759d5a-original-pull-secret") pod "global-pull-secret-syncer-cslr6" (UID: "9e4572a9-69d7-418c-b751-07c5d6759d5a") : object "kube-system"/"original-pull-secret" not registered
Jul 08 12:01:59.394700 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:01:59.394442    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a64d26d8-d849-4f64-b43d-3a9133970f52-metrics-certs podName:a64d26d8-d849-4f64-b43d-3a9133970f52 nodeName:}" failed. No retries permitted until 2026-07-08 12:02:07.394423051 +0000 UTC m=+17.124798003 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a64d26d8-d849-4f64-b43d-3a9133970f52-metrics-certs") pod "network-metrics-daemon-m5628" (UID: "a64d26d8-d849-4f64-b43d-3a9133970f52") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 12:01:59.494883 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:59.494835    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdj4j\" (UniqueName: \"kubernetes.io/projected/ecbe5ccd-c4c5-4820-9238-72591f011842-kube-api-access-sdj4j\") pod \"network-check-target-842d7\" (UID: \"ecbe5ccd-c4c5-4820-9238-72591f011842\") " pod="openshift-network-diagnostics/network-check-target-842d7"
Jul 08 12:01:59.495108 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:01:59.495089    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 12:01:59.495173 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:01:59.495114    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 12:01:59.495173 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:01:59.495127    2579 projected.go:196] Error preparing data for projected volume kube-api-access-sdj4j for pod openshift-network-diagnostics/network-check-target-842d7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 12:01:59.495280 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:01:59.495190    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ecbe5ccd-c4c5-4820-9238-72591f011842-kube-api-access-sdj4j podName:ecbe5ccd-c4c5-4820-9238-72591f011842 nodeName:}" failed. No retries permitted until 2026-07-08 12:02:07.495170149 +0000 UTC m=+17.225545118 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-sdj4j" (UniqueName: "kubernetes.io/projected/ecbe5ccd-c4c5-4820-9238-72591f011842-kube-api-access-sdj4j") pod "network-check-target-842d7" (UID: "ecbe5ccd-c4c5-4820-9238-72591f011842") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 12:01:59.711326 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:59.711246    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-842d7"
Jul 08 12:01:59.711494 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:01:59.711373    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-842d7" podUID="ecbe5ccd-c4c5-4820-9238-72591f011842"
Jul 08 12:01:59.711813 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:59.711791    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cslr6"
Jul 08 12:01:59.711920 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:01:59.711903    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cslr6" podUID="9e4572a9-69d7-418c-b751-07c5d6759d5a"
Jul 08 12:01:59.711986 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:01:59.711972    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m5628"
Jul 08 12:01:59.712085 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:01:59.712060    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m5628" podUID="a64d26d8-d849-4f64-b43d-3a9133970f52"
Jul 08 12:02:01.711347 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:01.711310    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cslr6"
Jul 08 12:02:01.711347 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:01.711347    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m5628"
Jul 08 12:02:01.711842 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:02:01.711461    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cslr6" podUID="9e4572a9-69d7-418c-b751-07c5d6759d5a"
Jul 08 12:02:01.711842 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:02:01.711527    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m5628" podUID="a64d26d8-d849-4f64-b43d-3a9133970f52"
Jul 08 12:02:01.711842 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:01.711556    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-842d7"
Jul 08 12:02:01.711842 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:02:01.711657    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-842d7" podUID="ecbe5ccd-c4c5-4820-9238-72591f011842"
Jul 08 12:02:03.710464 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:03.710431    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-842d7"
Jul 08 12:02:03.710963 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:03.710538    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cslr6"
Jul 08 12:02:03.710963 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:02:03.710550    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-842d7" podUID="ecbe5ccd-c4c5-4820-9238-72591f011842"
Jul 08 12:02:03.710963 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:03.710574    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m5628"
Jul 08 12:02:03.710963 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:02:03.710671    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cslr6" podUID="9e4572a9-69d7-418c-b751-07c5d6759d5a"
Jul 08 12:02:03.710963 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:02:03.710759    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m5628" podUID="a64d26d8-d849-4f64-b43d-3a9133970f52"
Jul 08 12:02:03.807558 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:03.807523    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-89.ec2.internal" event={"ID":"01a7cd28e5cb68af10b158e7471bca6f","Type":"ContainerStarted","Data":"b32180bec399bd9626e13791cf8b6839e2cd61ea6fa80678f278446fbc083848"}
Jul 08 12:02:03.809014 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:03.808970    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2jgc8" event={"ID":"24f4637c-726e-45ae-991a-c517b065e0cc","Type":"ContainerStarted","Data":"7a482f1e1e57bf559f7f86430792dfcdc7e863a9e6be05dbd080039120ac5f1e"}
Jul 08 12:02:03.823409 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:03.823364    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-89.ec2.internal" podStartSLOduration=11.823350662 podStartE2EDuration="11.823350662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 12:01:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 12:02:03.823204677 +0000 UTC m=+13.553579649" watchObservedRunningTime="2026-07-08 12:02:03.823350662 +0000 UTC m=+13.553725632"
Jul 08 12:02:05.711170 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:05.710896    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cslr6"
Jul 08 12:02:05.711545 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:02:05.711225    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cslr6" podUID="9e4572a9-69d7-418c-b751-07c5d6759d5a"
Jul 08 12:02:05.711545 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:05.710966    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-842d7"
Jul 08 12:02:05.711545 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:02:05.711302    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-842d7" podUID="ecbe5ccd-c4c5-4820-9238-72591f011842"
Jul 08 12:02:05.711545 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:05.710934    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m5628"
Jul 08 12:02:05.711545 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:02:05.711376    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m5628" podUID="a64d26d8-d849-4f64-b43d-3a9133970f52"
Jul 08 12:02:05.814686 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:05.814647    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-26hpb" event={"ID":"c0abdbfc-0312-4fd1-9ca4-098e3fa3e7e1","Type":"ContainerStarted","Data":"5bdd40b7b19dd8ab5716843cdd5072fd81a0df77c3ef90ff654620c68c893b80"}
Jul 08 12:02:05.816681 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:05.816650    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5wpmj" event={"ID":"e37bfd95-6a30-453c-be17-846c687e4d9e","Type":"ContainerStarted","Data":"ef05c293718332d2b43a5449fd82314f8fad917794d604f66a6c13617e6206ef"}
Jul 08 12:02:05.818139 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:05.818093    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pp86n" event={"ID":"56220650-f743-4804-9e3e-5a0cffd4fd3a","Type":"ContainerStarted","Data":"9183778e14cf40f30879e18ac3858ff7e2c36d84a7dc0559d2b467934b141ce7"}
Jul 08 12:02:05.819545 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:05.819508    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-k4jsh" event={"ID":"641b596d-df5a-491a-a40e-e269de0a676e","Type":"ContainerStarted","Data":"3114edb96a5f88f267545bcacf17ecfcd4486d72ba767ec93510d93b6306916c"}
Jul 08 12:02:05.821131 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:05.821109    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwvt9" event={"ID":"c08cf95f-760d-49d9-8b97-b7cb64dc6c95","Type":"ContainerStarted","Data":"8f9d1a568bc4eee53252ce94ee82e6d0cce460b52cee42349b8a69487fbf5044"}
Jul 08 12:02:05.822544 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:05.822520    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hrzgb" event={"ID":"ffda1903-e8ae-4139-bb4c-d01904b23925","Type":"ContainerStarted","Data":"2c88bda9cabd66ea994c17a06b5e41e6ea232d4edc54a3821c6a0a361ee7e534"}
Jul 08 12:02:05.824061 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:05.824033    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-89.ec2.internal" event={"ID":"8e5e073dfe04675ff90daa0b47b9f43e","Type":"ContainerStarted","Data":"5982e432b4cf7bd2207a8a9d6b71db5f96ecb963e390e669404ae94a8208ff3e"}
Jul 08 12:02:05.833072 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:05.833023    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-26hpb" podStartSLOduration=3.44714717 podStartE2EDuration="14.833007017s" totalImagesPullingTime="11.385859847s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 12:01:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 12:02:05.832795571 +0000 UTC m=+15.563170544" watchObservedRunningTime="2026-07-08 12:02:05.833007017 +0000 UTC m=+15.563381991"
Jul 08 12:02:05.833403 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:05.833374    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-2jgc8" podStartSLOduration=3.403111193 podStartE2EDuration="14.833363235s" totalImagesPullingTime="11.430252042s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 12:01:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 12:02:03.841496802 +0000 UTC m=+13.571871774" watchObservedRunningTime="2026-07-08 12:02:05.833363235 +0000 UTC m=+15.563738205"
Jul 08 12:02:05.850987 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:05.850931    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-5wpmj" podStartSLOduration=3.440928952 podStartE2EDuration="14.85091659s" totalImagesPullingTime="11.409987638s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 12:01:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 12:02:05.850399528 +0000 UTC m=+15.580774501" watchObservedRunningTime="2026-07-08 12:02:05.85091659 +0000 UTC m=+15.581291643"
Jul 08 12:02:05.871535 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:05.871480    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-k4jsh" podStartSLOduration=3.490250303 podStartE2EDuration="14.87146418s" totalImagesPullingTime="11.381213877s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 12:01:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 12:02:05.87098419 +0000 UTC m=+15.601359173" watchObservedRunningTime="2026-07-08 12:02:05.87146418 +0000 UTC m=+15.601839183"
Jul 08 12:02:05.887671 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:05.887619    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-pp86n" podStartSLOduration=3.542253158 podStartE2EDuration="14.887603701s" totalImagesPullingTime="11.345350543s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 12:01:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 12:02:05.8874135 +0000 UTC m=+15.617788471" watchObservedRunningTime="2026-07-08 12:02:05.887603701 +0000 UTC m=+15.617978672"
Jul 08 12:02:07.225998 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:07.225954    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-pp86n"
Jul 08 12:02:07.226617 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:07.226598    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-pp86n"
Jul 08 12:02:07.461807 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:07.461752    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a64d26d8-d849-4f64-b43d-3a9133970f52-metrics-certs\") pod \"network-metrics-daemon-m5628\" (UID: \"a64d26d8-d849-4f64-b43d-3a9133970f52\") " pod="openshift-multus/network-metrics-daemon-m5628"
Jul 08 12:02:07.461984 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:07.461825    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9e4572a9-69d7-418c-b751-07c5d6759d5a-original-pull-secret\") pod \"global-pull-secret-syncer-cslr6\" (UID: \"9e4572a9-69d7-418c-b751-07c5d6759d5a\") " pod="kube-system/global-pull-secret-syncer-cslr6"
Jul 08 12:02:07.461984 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:02:07.461972    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 12:02:07.461984 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:02:07.461972    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 12:02:07.462081 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:02:07.462031    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e4572a9-69d7-418c-b751-07c5d6759d5a-original-pull-secret podName:9e4572a9-69d7-418c-b751-07c5d6759d5a nodeName:}" failed. No retries permitted until 2026-07-08 12:02:23.462014118 +0000 UTC m=+33.192389067 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9e4572a9-69d7-418c-b751-07c5d6759d5a-original-pull-secret") pod "global-pull-secret-syncer-cslr6" (UID: "9e4572a9-69d7-418c-b751-07c5d6759d5a") : object "kube-system"/"original-pull-secret" not registered
Jul 08 12:02:07.462081 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:02:07.462048    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a64d26d8-d849-4f64-b43d-3a9133970f52-metrics-certs podName:a64d26d8-d849-4f64-b43d-3a9133970f52 nodeName:}" failed. No retries permitted until 2026-07-08 12:02:23.462040419 +0000 UTC m=+33.192415369 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a64d26d8-d849-4f64-b43d-3a9133970f52-metrics-certs") pod "network-metrics-daemon-m5628" (UID: "a64d26d8-d849-4f64-b43d-3a9133970f52") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 12:02:07.562925 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:07.562839    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdj4j\" (UniqueName: \"kubernetes.io/projected/ecbe5ccd-c4c5-4820-9238-72591f011842-kube-api-access-sdj4j\") pod \"network-check-target-842d7\" (UID: \"ecbe5ccd-c4c5-4820-9238-72591f011842\") " pod="openshift-network-diagnostics/network-check-target-842d7"
Jul 08 12:02:07.563110 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:02:07.562984    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 12:02:07.563110 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:02:07.563005    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 12:02:07.563110 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:02:07.563018    2579 projected.go:196] Error preparing data for projected volume kube-api-access-sdj4j for pod openshift-network-diagnostics/network-check-target-842d7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 12:02:07.563110 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:02:07.563079    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ecbe5ccd-c4c5-4820-9238-72591f011842-kube-api-access-sdj4j podName:ecbe5ccd-c4c5-4820-9238-72591f011842 nodeName:}" failed. No retries permitted until 2026-07-08 12:02:23.563061197 +0000 UTC m=+33.293436148 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-sdj4j" (UniqueName: "kubernetes.io/projected/ecbe5ccd-c4c5-4820-9238-72591f011842-kube-api-access-sdj4j") pod "network-check-target-842d7" (UID: "ecbe5ccd-c4c5-4820-9238-72591f011842") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 12:02:07.710784 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:07.710737    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m5628"
Jul 08 12:02:07.710986 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:07.710800    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-842d7"
Jul 08 12:02:07.710986 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:07.710737    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cslr6"
Jul 08 12:02:07.710986 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:02:07.710868    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m5628" podUID="a64d26d8-d849-4f64-b43d-3a9133970f52"
Jul 08 12:02:07.710986 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:02:07.710945    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cslr6" podUID="9e4572a9-69d7-418c-b751-07c5d6759d5a"
Jul 08 12:02:07.711152 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:02:07.711037    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-842d7" podUID="ecbe5ccd-c4c5-4820-9238-72591f011842"
Jul 08 12:02:07.829328 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:07.829251    2579 generic.go:356] "Generic (PLEG): container finished" podID="ffda1903-e8ae-4139-bb4c-d01904b23925" containerID="2c88bda9cabd66ea994c17a06b5e41e6ea232d4edc54a3821c6a0a361ee7e534" exitCode=0
Jul 08 12:02:07.829328 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:07.829315    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hrzgb" event={"ID":"ffda1903-e8ae-4139-bb4c-d01904b23925","Type":"ContainerDied","Data":"2c88bda9cabd66ea994c17a06b5e41e6ea232d4edc54a3821c6a0a361ee7e534"}
Jul 08 12:02:07.829627 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:07.829611    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-pp86n"
Jul 08 12:02:07.830400 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:07.830380    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-pp86n"
Jul 08 12:02:09.711280 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:09.711234    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cslr6"
Jul 08 12:02:09.711981 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:09.711332    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-842d7"
Jul 08 12:02:09.711981 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:02:09.711464    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cslr6" podUID="9e4572a9-69d7-418c-b751-07c5d6759d5a"
Jul 08 12:02:09.711981 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:09.711525    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m5628"
Jul 08 12:02:09.711981 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:02:09.711664    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-842d7" podUID="ecbe5ccd-c4c5-4820-9238-72591f011842"
Jul 08 12:02:09.711981 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:02:09.711791    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m5628" podUID="a64d26d8-d849-4f64-b43d-3a9133970f52"
Jul 08 12:02:11.710651 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:11.710614    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cslr6"
Jul 08 12:02:11.711102 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:11.710629    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m5628"
Jul 08 12:02:11.711102 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:02:11.710735    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cslr6" podUID="9e4572a9-69d7-418c-b751-07c5d6759d5a"
Jul 08 12:02:11.711102 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:11.710629    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-842d7"
Jul 08 12:02:11.711102 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:02:11.710855    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m5628" podUID="a64d26d8-d849-4f64-b43d-3a9133970f52"
Jul 08 12:02:11.711102 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:02:11.710945    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-842d7" podUID="ecbe5ccd-c4c5-4820-9238-72591f011842"
Jul 08 12:02:12.762170 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:12.761819    2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 08 12:02:12.848410 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:12.848378    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vm6ql" event={"ID":"636b6729-2bf5-43f2-8db0-c1052576d023","Type":"ContainerStarted","Data":"44d04f185d792c7c6731edb43352f4b71ddbd98d370c602cde69d8ae2f606487"}
Jul 08 12:02:12.850818 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:12.850796    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xkp4b" event={"ID":"b6d8afe0-1555-462f-921e-eaa3d7acdb7a","Type":"ContainerStarted","Data":"3093b133cbf1ca37bdc21728806aa482dec3c916c865c9bf89f7cbd6937f0e34"}
Jul 08 12:02:12.850947 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:12.850870    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xkp4b" event={"ID":"b6d8afe0-1555-462f-921e-eaa3d7acdb7a","Type":"ContainerStarted","Data":"464cc7603bca7cf233c821ce8f8084241d91568c732aee8da9055d3f683108bd"}
Jul 08 12:02:12.850947 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:12.850921    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xkp4b" event={"ID":"b6d8afe0-1555-462f-921e-eaa3d7acdb7a","Type":"ContainerStarted","Data":"2e239d0a82d1a80d52ebbdb31c0cc542157acaf77cafadf24e40a9447edab15d"}
Jul 08 12:02:12.852639 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:12.852618    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwvt9" event={"ID":"c08cf95f-760d-49d9-8b97-b7cb64dc6c95","Type":"ContainerStarted","Data":"711795764ddb28d2b8c5b5eb2282cb2ce0d84617482ee96ed8aca6a951fb94f5"}
Jul 08 12:02:12.854168 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:12.854149    2579 generic.go:356] "Generic (PLEG): container finished" podID="8e5e073dfe04675ff90daa0b47b9f43e" containerID="5982e432b4cf7bd2207a8a9d6b71db5f96ecb963e390e669404ae94a8208ff3e" exitCode=0
Jul 08 12:02:12.854243 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:12.854174    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-89.ec2.internal" event={"ID":"8e5e073dfe04675ff90daa0b47b9f43e","Type":"ContainerDied","Data":"5982e432b4cf7bd2207a8a9d6b71db5f96ecb963e390e669404ae94a8208ff3e"}
Jul 08 12:02:12.892373 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:12.892324    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-vm6ql" podStartSLOduration=1.424565426 podStartE2EDuration="21.892304726s" totalImagesPullingTime="20.4677393s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 12:01:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 12:02:12.873503692 +0000 UTC m=+22.603878663" watchObservedRunningTime="2026-07-08 12:02:12.892304726 +0000 UTC m=+22.622679698"
Jul 08 12:02:13.689825 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:13.689699    2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-08T12:02:12.762168735Z","UUID":"38417645-bfd2-4784-af77-0c2353fa402e","Handler":null,"Name":"","Endpoint":""}
Jul 08 12:02:13.691303 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:13.691281    2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 08 12:02:13.691432 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:13.691323    2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 08 12:02:13.710516 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:13.710484    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cslr6"
Jul 08 12:02:13.710516 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:13.710527    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m5628"
Jul 08 12:02:13.710755 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:13.710527    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-842d7"
Jul 08 12:02:13.710755 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:02:13.710614    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cslr6" podUID="9e4572a9-69d7-418c-b751-07c5d6759d5a"
Jul 08 12:02:13.710755 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:02:13.710733    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m5628" podUID="a64d26d8-d849-4f64-b43d-3a9133970f52"
Jul 08 12:02:13.710932 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:02:13.710835    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-842d7" podUID="ecbe5ccd-c4c5-4820-9238-72591f011842"
Jul 08 12:02:13.859049 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:13.859013    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-89.ec2.internal" event={"ID":"8e5e073dfe04675ff90daa0b47b9f43e","Type":"ContainerStarted","Data":"e265c0e3f2c0094866ed94c3b3e75e6892488dbf52b4c055ed02fa2812d9e4d7"}
Jul 08 12:02:13.863404 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:13.863378    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xkp4b" event={"ID":"b6d8afe0-1555-462f-921e-eaa3d7acdb7a","Type":"ContainerStarted","Data":"d4b261a3d8e825964dfed80dd06b38ac27cd6fde00c09283f1e18648df69256a"}
Jul 08 12:02:13.863533 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:13.863458    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xkp4b" event={"ID":"b6d8afe0-1555-462f-921e-eaa3d7acdb7a","Type":"ContainerStarted","Data":"a4471bea851f3a4f94419291428428fd78d6e8d919a058e9baf26be166d59c5f"}
Jul 08 12:02:13.866924 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:13.863899    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xkp4b" event={"ID":"b6d8afe0-1555-462f-921e-eaa3d7acdb7a","Type":"ContainerStarted","Data":"9a6f88786fef5b1655fa146fdd9d9d79cbef9ea409075f05307629a6fc19b727"}
Jul 08 12:02:14.868106 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:14.868067    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwvt9" event={"ID":"c08cf95f-760d-49d9-8b97-b7cb64dc6c95","Type":"ContainerStarted","Data":"196f2262ac87a7f5aabbfaf6b7e5e40798556113aea30d07b374ea23ff9f2c7d"}
Jul 08 12:02:14.888202 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:14.888159    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwvt9" podStartSLOduration=3.691383827 podStartE2EDuration="23.888144824s" totalImagesPullingTime="20.196760997s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 12:01:51 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 12:02:14.88788053 +0000 UTC m=+24.618255500" watchObservedRunningTime="2026-07-08 12:02:14.888144824 +0000 UTC m=+24.618519816"
Jul 08 12:02:14.888349 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:14.888304    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-89.ec2.internal" podStartSLOduration=22.88830103 podStartE2EDuration="22.88830103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 12:01:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 12:02:13.878381582 +0000 UTC m=+23.608756566" watchObservedRunningTime="2026-07-08 12:02:14.88830103 +0000 UTC m=+24.618676001"
Jul 08 12:02:15.711019 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:15.710983    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m5628"
Jul 08 12:02:15.711019 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:15.711019    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cslr6"
Jul 08 12:02:15.711252 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:15.710983    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-842d7"
Jul 08 12:02:15.711252 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:02:15.711122    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m5628" podUID="a64d26d8-d849-4f64-b43d-3a9133970f52"
Jul 08 12:02:15.711252 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:02:15.711194    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-842d7" podUID="ecbe5ccd-c4c5-4820-9238-72591f011842"
Jul 08 12:02:15.711366 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:02:15.711261    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cslr6" podUID="9e4572a9-69d7-418c-b751-07c5d6759d5a"
Jul 08 12:02:15.875186 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:15.875146    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xkp4b" event={"ID":"b6d8afe0-1555-462f-921e-eaa3d7acdb7a","Type":"ContainerStarted","Data":"afaf8a3575af536895c88b82536fb3845e806a47ca1c05ac0ce5260046627e1b"}
Jul 08 12:02:17.711380 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:17.711205    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m5628"
Jul 08 12:02:17.711948 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:17.711205    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cslr6"
Jul 08 12:02:17.711948 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:02:17.711464    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m5628" podUID="a64d26d8-d849-4f64-b43d-3a9133970f52"
Jul 08 12:02:17.711948 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:17.711206    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-842d7"
Jul 08 12:02:17.711948 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:02:17.711519    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cslr6" podUID="9e4572a9-69d7-418c-b751-07c5d6759d5a"
Jul 08 12:02:17.711948 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:02:17.711615    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-842d7" podUID="ecbe5ccd-c4c5-4820-9238-72591f011842"
Jul 08 12:02:17.882281 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:17.882192    2579 generic.go:356] "Generic (PLEG): container finished" podID="ffda1903-e8ae-4139-bb4c-d01904b23925" containerID="c39d4ca09f2b4b98d6f0cb5f79c14475120cc041cd8800148f44e262424e11a1" exitCode=0
Jul 08 12:02:17.882457 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:17.882268    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hrzgb" event={"ID":"ffda1903-e8ae-4139-bb4c-d01904b23925","Type":"ContainerDied","Data":"c39d4ca09f2b4b98d6f0cb5f79c14475120cc041cd8800148f44e262424e11a1"}
Jul 08 12:02:17.886331 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:17.886305    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xkp4b" event={"ID":"b6d8afe0-1555-462f-921e-eaa3d7acdb7a","Type":"ContainerStarted","Data":"fee3431ed0e41dc762b1898c14604dc63cd633e79d8f28b416f86e82ca081db1"}
Jul 08 12:02:17.886833 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:17.886685    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xkp4b"
Jul 08 12:02:17.886833 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:17.886749    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xkp4b"
Jul 08 12:02:17.886833 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:17.886802    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xkp4b"
Jul 08 12:02:17.903335 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:17.903311    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xkp4b"
Jul 08 12:02:17.903461 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:17.903409    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xkp4b"
Jul 08 12:02:17.933453 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:17.933403    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-xkp4b" podStartSLOduration=6.454322212 podStartE2EDuration="26.933388919s" totalImagesPullingTime="20.479066707s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 12:01:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 12:02:17.932200939 +0000 UTC m=+27.662575910" watchObservedRunningTime="2026-07-08 12:02:17.933388919 +0000 UTC m=+27.663763889"
Jul 08 12:02:19.466015 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:19.465986    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cslr6"]
Jul 08 12:02:19.466458 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:19.466103    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cslr6"
Jul 08 12:02:19.466458 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:02:19.466193    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cslr6" podUID="9e4572a9-69d7-418c-b751-07c5d6759d5a"
Jul 08 12:02:19.469541 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:19.469511    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-m5628"]
Jul 08 12:02:19.469671 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:19.469620    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m5628"
Jul 08 12:02:19.469738 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:02:19.469716    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m5628" podUID="a64d26d8-d849-4f64-b43d-3a9133970f52"
Jul 08 12:02:19.470726 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:19.470483    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-842d7"]
Jul 08 12:02:19.470726 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:19.470579    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-842d7"
Jul 08 12:02:19.470726 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:02:19.470704    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-842d7" podUID="ecbe5ccd-c4c5-4820-9238-72591f011842"
Jul 08 12:02:19.895233 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:19.895146    2579 generic.go:356] "Generic (PLEG): container finished" podID="ffda1903-e8ae-4139-bb4c-d01904b23925" containerID="49d61c98cc25d2b755f6ffe16fff00dc82a4b1844793e9befea10b61bdd53e93" exitCode=0
Jul 08 12:02:19.895233 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:19.895213    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hrzgb" event={"ID":"ffda1903-e8ae-4139-bb4c-d01904b23925","Type":"ContainerDied","Data":"49d61c98cc25d2b755f6ffe16fff00dc82a4b1844793e9befea10b61bdd53e93"}
Jul 08 12:02:21.710478 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:21.710280    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m5628"
Jul 08 12:02:21.710882 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:21.710281    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cslr6"
Jul 08 12:02:21.710882 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:21.710280    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-842d7"
Jul 08 12:02:21.710882 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:02:21.710622    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m5628" podUID="a64d26d8-d849-4f64-b43d-3a9133970f52"
Jul 08 12:02:21.710882 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:02:21.710703    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cslr6" podUID="9e4572a9-69d7-418c-b751-07c5d6759d5a"
Jul 08 12:02:21.710882 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:02:21.710783    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-842d7" podUID="ecbe5ccd-c4c5-4820-9238-72591f011842"
Jul 08 12:02:23.473827 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:23.473793    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a64d26d8-d849-4f64-b43d-3a9133970f52-metrics-certs\") pod \"network-metrics-daemon-m5628\" (UID: \"a64d26d8-d849-4f64-b43d-3a9133970f52\") " pod="openshift-multus/network-metrics-daemon-m5628"
Jul 08 12:02:23.473827 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:23.473831    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9e4572a9-69d7-418c-b751-07c5d6759d5a-original-pull-secret\") pod \"global-pull-secret-syncer-cslr6\" (UID: \"9e4572a9-69d7-418c-b751-07c5d6759d5a\") " pod="kube-system/global-pull-secret-syncer-cslr6"
Jul 08 12:02:23.474289 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:02:23.473927    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 12:02:23.474289 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:02:23.473929    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 12:02:23.474289 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:02:23.473981    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e4572a9-69d7-418c-b751-07c5d6759d5a-original-pull-secret podName:9e4572a9-69d7-418c-b751-07c5d6759d5a nodeName:}" failed. No retries permitted until 2026-07-08 12:02:55.473967168 +0000 UTC m=+65.204342135 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9e4572a9-69d7-418c-b751-07c5d6759d5a-original-pull-secret") pod "global-pull-secret-syncer-cslr6" (UID: "9e4572a9-69d7-418c-b751-07c5d6759d5a") : object "kube-system"/"original-pull-secret" not registered
Jul 08 12:02:23.474289 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:02:23.473996    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a64d26d8-d849-4f64-b43d-3a9133970f52-metrics-certs podName:a64d26d8-d849-4f64-b43d-3a9133970f52 nodeName:}" failed. No retries permitted until 2026-07-08 12:02:55.473989974 +0000 UTC m=+65.204364923 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a64d26d8-d849-4f64-b43d-3a9133970f52-metrics-certs") pod "network-metrics-daemon-m5628" (UID: "a64d26d8-d849-4f64-b43d-3a9133970f52") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 12:02:23.574679 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:23.574590    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdj4j\" (UniqueName: \"kubernetes.io/projected/ecbe5ccd-c4c5-4820-9238-72591f011842-kube-api-access-sdj4j\") pod \"network-check-target-842d7\" (UID: \"ecbe5ccd-c4c5-4820-9238-72591f011842\") " pod="openshift-network-diagnostics/network-check-target-842d7"
Jul 08 12:02:23.574846 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:02:23.574756    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 12:02:23.574846 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:02:23.574793    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 12:02:23.574846 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:02:23.574803    2579 projected.go:196] Error preparing data for projected volume kube-api-access-sdj4j for pod openshift-network-diagnostics/network-check-target-842d7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 12:02:23.574952 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:02:23.574852    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ecbe5ccd-c4c5-4820-9238-72591f011842-kube-api-access-sdj4j podName:ecbe5ccd-c4c5-4820-9238-72591f011842 nodeName:}" failed. No retries permitted until 2026-07-08 12:02:55.574839813 +0000 UTC m=+65.305214762 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-sdj4j" (UniqueName: "kubernetes.io/projected/ecbe5ccd-c4c5-4820-9238-72591f011842-kube-api-access-sdj4j") pod "network-check-target-842d7" (UID: "ecbe5ccd-c4c5-4820-9238-72591f011842") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 12:02:23.711310 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:23.711272    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-842d7"
Jul 08 12:02:23.711475 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:23.711288    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cslr6"
Jul 08 12:02:23.711475 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:02:23.711408    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-842d7" podUID="ecbe5ccd-c4c5-4820-9238-72591f011842"
Jul 08 12:02:23.711475 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:23.711458    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m5628"
Jul 08 12:02:23.711574 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:02:23.711536    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cslr6" podUID="9e4572a9-69d7-418c-b751-07c5d6759d5a"
Jul 08 12:02:23.711635 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:02:23.711618    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m5628" podUID="a64d26d8-d849-4f64-b43d-3a9133970f52"
Jul 08 12:02:23.906439 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:23.906365    2579 generic.go:356] "Generic (PLEG): container finished" podID="ffda1903-e8ae-4139-bb4c-d01904b23925" containerID="69934fe881fbd04c44b08da5209caaebe12fb539b9afd85a3d9e53dbee2e628d" exitCode=0
Jul 08 12:02:23.906439 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:23.906416    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hrzgb" event={"ID":"ffda1903-e8ae-4139-bb4c-d01904b23925","Type":"ContainerDied","Data":"69934fe881fbd04c44b08da5209caaebe12fb539b9afd85a3d9e53dbee2e628d"}
Jul 08 12:02:24.123702 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:24.123671    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-89.ec2.internal" event="NodeReady"
Jul 08 12:02:24.123918 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:24.123855    2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 08 12:02:24.168229 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:24.168155    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-jprx6"]
Jul 08 12:02:24.191539 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:24.191508    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-nn7z8"]
Jul 08 12:02:24.191686 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:24.191635    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jprx6"
Jul 08 12:02:24.194037 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:24.194013    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 08 12:02:24.194037 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:24.194023    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 08 12:02:24.194221 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:24.194147    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5pzr4\""
Jul 08 12:02:24.208204 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:24.208178    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jprx6"]
Jul 08 12:02:24.208348 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:24.208224    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nn7z8"]
Jul 08 12:02:24.208348 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:24.208302    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nn7z8"
Jul 08 12:02:24.210561 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:24.210543    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 08 12:02:24.210698 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:24.210612    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 08 12:02:24.210698 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:24.210632    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 08 12:02:24.210698 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:24.210616    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-twvk9\""
Jul 08 12:02:24.280733 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:24.280694    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb2tv\" (UniqueName: \"kubernetes.io/projected/d8e14b6c-7fd3-46b6-8e03-f8c03d924e2b-kube-api-access-rb2tv\") pod \"ingress-canary-nn7z8\" (UID: \"d8e14b6c-7fd3-46b6-8e03-f8c03d924e2b\") " pod="openshift-ingress-canary/ingress-canary-nn7z8"
Jul 08 12:02:24.280933 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:24.280755    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9785\" (UniqueName: \"kubernetes.io/projected/6fb2ecb5-0ba5-4b15-9654-76648d18ed3b-kube-api-access-t9785\") pod \"dns-default-jprx6\" (UID: \"6fb2ecb5-0ba5-4b15-9654-76648d18ed3b\") " pod="openshift-dns/dns-default-jprx6"
Jul 08 12:02:24.280933 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:24.280793    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6fb2ecb5-0ba5-4b15-9654-76648d18ed3b-metrics-tls\") pod \"dns-default-jprx6\" (UID: \"6fb2ecb5-0ba5-4b15-9654-76648d18ed3b\") " pod="openshift-dns/dns-default-jprx6"
Jul 08 12:02:24.280933 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:24.280817    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fb2ecb5-0ba5-4b15-9654-76648d18ed3b-tmp-dir\") pod \"dns-default-jprx6\" (UID: \"6fb2ecb5-0ba5-4b15-9654-76648d18ed3b\") " pod="openshift-dns/dns-default-jprx6"
Jul 08 12:02:24.280933 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:24.280848    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6fb2ecb5-0ba5-4b15-9654-76648d18ed3b-config-volume\") pod \"dns-default-jprx6\" (UID: \"6fb2ecb5-0ba5-4b15-9654-76648d18ed3b\") " pod="openshift-dns/dns-default-jprx6"
Jul 08 12:02:24.280933 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:24.280861    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d8e14b6c-7fd3-46b6-8e03-f8c03d924e2b-cert\") pod \"ingress-canary-nn7z8\" (UID: \"d8e14b6c-7fd3-46b6-8e03-f8c03d924e2b\") " pod="openshift-ingress-canary/ingress-canary-nn7z8"
Jul 08 12:02:24.381544 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:24.381506    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb2tv\" (UniqueName: \"kubernetes.io/projected/d8e14b6c-7fd3-46b6-8e03-f8c03d924e2b-kube-api-access-rb2tv\") pod \"ingress-canary-nn7z8\" (UID: \"d8e14b6c-7fd3-46b6-8e03-f8c03d924e2b\") " pod="openshift-ingress-canary/ingress-canary-nn7z8"
Jul 08 12:02:24.381655 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:24.381637    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9785\" (UniqueName: \"kubernetes.io/projected/6fb2ecb5-0ba5-4b15-9654-76648d18ed3b-kube-api-access-t9785\") pod \"dns-default-jprx6\" (UID: \"6fb2ecb5-0ba5-4b15-9654-76648d18ed3b\") " pod="openshift-dns/dns-default-jprx6"
Jul 08 12:02:24.381705 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:24.381666    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6fb2ecb5-0ba5-4b15-9654-76648d18ed3b-metrics-tls\") pod \"dns-default-jprx6\" (UID: \"6fb2ecb5-0ba5-4b15-9654-76648d18ed3b\") " pod="openshift-dns/dns-default-jprx6"
Jul 08 12:02:24.381705 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:24.381684    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fb2ecb5-0ba5-4b15-9654-76648d18ed3b-tmp-dir\") pod \"dns-default-jprx6\" (UID: \"6fb2ecb5-0ba5-4b15-9654-76648d18ed3b\") " pod="openshift-dns/dns-default-jprx6"
Jul 08 12:02:24.381823 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:24.381718    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6fb2ecb5-0ba5-4b15-9654-76648d18ed3b-config-volume\") pod \"dns-default-jprx6\" (UID: \"6fb2ecb5-0ba5-4b15-9654-76648d18ed3b\") " pod="openshift-dns/dns-default-jprx6"
Jul 08 12:02:24.381823 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:24.381732    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d8e14b6c-7fd3-46b6-8e03-f8c03d924e2b-cert\") pod \"ingress-canary-nn7z8\" (UID: \"d8e14b6c-7fd3-46b6-8e03-f8c03d924e2b\") " pod="openshift-ingress-canary/ingress-canary-nn7z8"
Jul 08 12:02:24.381923 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:02:24.381837    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 08 12:02:24.381923 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:02:24.381909    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fb2ecb5-0ba5-4b15-9654-76648d18ed3b-metrics-tls podName:6fb2ecb5-0ba5-4b15-9654-76648d18ed3b nodeName:}" failed. No retries permitted until 2026-07-08 12:02:24.881888768 +0000 UTC m=+34.612263718 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6fb2ecb5-0ba5-4b15-9654-76648d18ed3b-metrics-tls") pod "dns-default-jprx6" (UID: "6fb2ecb5-0ba5-4b15-9654-76648d18ed3b") : secret "dns-default-metrics-tls" not found
Jul 08 12:02:24.382146 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:24.382124    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fb2ecb5-0ba5-4b15-9654-76648d18ed3b-tmp-dir\") pod \"dns-default-jprx6\" (UID: \"6fb2ecb5-0ba5-4b15-9654-76648d18ed3b\") " pod="openshift-dns/dns-default-jprx6"
Jul 08 12:02:24.382345 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:24.382319    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6fb2ecb5-0ba5-4b15-9654-76648d18ed3b-config-volume\") pod \"dns-default-jprx6\" (UID: \"6fb2ecb5-0ba5-4b15-9654-76648d18ed3b\") " pod="openshift-dns/dns-default-jprx6"
Jul 08 12:02:24.386000 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:24.385978    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d8e14b6c-7fd3-46b6-8e03-f8c03d924e2b-cert\") pod \"ingress-canary-nn7z8\" (UID: \"d8e14b6c-7fd3-46b6-8e03-f8c03d924e2b\") " pod="openshift-ingress-canary/ingress-canary-nn7z8"
Jul 08 12:02:24.390795 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:24.390750    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb2tv\" (UniqueName: \"kubernetes.io/projected/d8e14b6c-7fd3-46b6-8e03-f8c03d924e2b-kube-api-access-rb2tv\") pod \"ingress-canary-nn7z8\" (UID: \"d8e14b6c-7fd3-46b6-8e03-f8c03d924e2b\") " pod="openshift-ingress-canary/ingress-canary-nn7z8"
Jul 08 12:02:24.391004 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:24.390982    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9785\" (UniqueName: \"kubernetes.io/projected/6fb2ecb5-0ba5-4b15-9654-76648d18ed3b-kube-api-access-t9785\") pod \"dns-default-jprx6\" (UID: \"6fb2ecb5-0ba5-4b15-9654-76648d18ed3b\") " pod="openshift-dns/dns-default-jprx6"
Jul 08 12:02:24.519230 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:24.519191    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nn7z8"
Jul 08 12:02:24.724217 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:24.724061    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nn7z8"]
Jul 08 12:02:24.727901 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:02:24.727879    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8e14b6c_7fd3_46b6_8e03_f8c03d924e2b.slice/crio-70a37e6e616c998107d3cb5c649033b133d537aa5c40ab5b4cada9b1b993577e WatchSource:0}: Error finding container 70a37e6e616c998107d3cb5c649033b133d537aa5c40ab5b4cada9b1b993577e: Status 404 returned error can't find the container with id 70a37e6e616c998107d3cb5c649033b133d537aa5c40ab5b4cada9b1b993577e
Jul 08 12:02:24.886838 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:24.886723    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6fb2ecb5-0ba5-4b15-9654-76648d18ed3b-metrics-tls\") pod \"dns-default-jprx6\" (UID: \"6fb2ecb5-0ba5-4b15-9654-76648d18ed3b\") " pod="openshift-dns/dns-default-jprx6"
Jul 08 12:02:24.886978 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:02:24.886841    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 08 12:02:24.886978 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:02:24.886893    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fb2ecb5-0ba5-4b15-9654-76648d18ed3b-metrics-tls podName:6fb2ecb5-0ba5-4b15-9654-76648d18ed3b nodeName:}" failed. No retries permitted until 2026-07-08 12:02:25.886880671 +0000 UTC m=+35.617255623 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6fb2ecb5-0ba5-4b15-9654-76648d18ed3b-metrics-tls") pod "dns-default-jprx6" (UID: "6fb2ecb5-0ba5-4b15-9654-76648d18ed3b") : secret "dns-default-metrics-tls" not found
Jul 08 12:02:24.909976 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:24.909944    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nn7z8" event={"ID":"d8e14b6c-7fd3-46b6-8e03-f8c03d924e2b","Type":"ContainerStarted","Data":"70a37e6e616c998107d3cb5c649033b133d537aa5c40ab5b4cada9b1b993577e"}
Jul 08 12:02:25.711231 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:25.711199    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-842d7"
Jul 08 12:02:25.711231 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:25.711225    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cslr6"
Jul 08 12:02:25.711793 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:25.711209    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m5628"
Jul 08 12:02:25.717569 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:25.717543    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 08 12:02:25.717848 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:25.717808    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 08 12:02:25.718055 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:25.718040    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 08 12:02:25.718246 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:25.718227    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 08 12:02:25.719001 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:25.718985    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hvrx5\""
Jul 08 12:02:25.719106 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:25.719021    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rdkjt\""
Jul 08 12:02:25.894607 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:25.894562    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6fb2ecb5-0ba5-4b15-9654-76648d18ed3b-metrics-tls\") pod \"dns-default-jprx6\" (UID: \"6fb2ecb5-0ba5-4b15-9654-76648d18ed3b\") " pod="openshift-dns/dns-default-jprx6"
Jul 08 12:02:25.894804 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:02:25.894722    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 08 12:02:25.894867 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:02:25.894822    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fb2ecb5-0ba5-4b15-9654-76648d18ed3b-metrics-tls podName:6fb2ecb5-0ba5-4b15-9654-76648d18ed3b nodeName:}" failed. No retries permitted until 2026-07-08 12:02:27.894804639 +0000 UTC m=+37.625179588 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6fb2ecb5-0ba5-4b15-9654-76648d18ed3b-metrics-tls") pod "dns-default-jprx6" (UID: "6fb2ecb5-0ba5-4b15-9654-76648d18ed3b") : secret "dns-default-metrics-tls" not found
Jul 08 12:02:27.911444 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:27.911414    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6fb2ecb5-0ba5-4b15-9654-76648d18ed3b-metrics-tls\") pod \"dns-default-jprx6\" (UID: \"6fb2ecb5-0ba5-4b15-9654-76648d18ed3b\") " pod="openshift-dns/dns-default-jprx6"
Jul 08 12:02:27.911854 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:02:27.911526    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 08 12:02:27.911854 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:02:27.911583    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fb2ecb5-0ba5-4b15-9654-76648d18ed3b-metrics-tls podName:6fb2ecb5-0ba5-4b15-9654-76648d18ed3b nodeName:}" failed. No retries permitted until 2026-07-08 12:02:31.911564986 +0000 UTC m=+41.641939935 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6fb2ecb5-0ba5-4b15-9654-76648d18ed3b-metrics-tls") pod "dns-default-jprx6" (UID: "6fb2ecb5-0ba5-4b15-9654-76648d18ed3b") : secret "dns-default-metrics-tls" not found
Jul 08 12:02:28.922722 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:28.922684    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nn7z8" event={"ID":"d8e14b6c-7fd3-46b6-8e03-f8c03d924e2b","Type":"ContainerStarted","Data":"fe89aca5ca7074e74566e35a45417b77d758c519affc102af2b1d35d93e494c2"}
Jul 08 12:02:28.941021 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:28.940966    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-nn7z8" podStartSLOduration=1.584400054 podStartE2EDuration="4.940950055s" totalImagesPullingTime="3.356550001s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 12:02:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 12:02:28.940728887 +0000 UTC m=+38.671103860" watchObservedRunningTime="2026-07-08 12:02:28.940950055 +0000 UTC m=+38.671325025"
Jul 08 12:02:31.932763 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:31.932589    2579 generic.go:356] "Generic (PLEG): container finished" podID="ffda1903-e8ae-4139-bb4c-d01904b23925" containerID="3fbe2ce91ca78b0cafa1e096dce506042fa1f02a66609009d4fdbc06d714bc9d" exitCode=0
Jul 08 12:02:31.932763 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:31.932657    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hrzgb" event={"ID":"ffda1903-e8ae-4139-bb4c-d01904b23925","Type":"ContainerDied","Data":"3fbe2ce91ca78b0cafa1e096dce506042fa1f02a66609009d4fdbc06d714bc9d"}
Jul 08 12:02:31.942026 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:31.942002    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6fb2ecb5-0ba5-4b15-9654-76648d18ed3b-metrics-tls\") pod \"dns-default-jprx6\" (UID: \"6fb2ecb5-0ba5-4b15-9654-76648d18ed3b\") " pod="openshift-dns/dns-default-jprx6"
Jul 08 12:02:31.942157 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:02:31.942112    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 08 12:02:31.942200 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:02:31.942158    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fb2ecb5-0ba5-4b15-9654-76648d18ed3b-metrics-tls podName:6fb2ecb5-0ba5-4b15-9654-76648d18ed3b nodeName:}" failed. No retries permitted until 2026-07-08 12:02:39.942144846 +0000 UTC m=+49.672519795 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6fb2ecb5-0ba5-4b15-9654-76648d18ed3b-metrics-tls") pod "dns-default-jprx6" (UID: "6fb2ecb5-0ba5-4b15-9654-76648d18ed3b") : secret "dns-default-metrics-tls" not found
Jul 08 12:02:32.938007 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:32.937969    2579 generic.go:356] "Generic (PLEG): container finished" podID="ffda1903-e8ae-4139-bb4c-d01904b23925" containerID="8f49b987ea6bb9d578257496f8242505f2eac0f40bef640b08fa4da6569e3642" exitCode=0
Jul 08 12:02:32.938351 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:32.938030    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hrzgb" event={"ID":"ffda1903-e8ae-4139-bb4c-d01904b23925","Type":"ContainerDied","Data":"8f49b987ea6bb9d578257496f8242505f2eac0f40bef640b08fa4da6569e3642"}
Jul 08 12:02:33.943399 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:33.943365    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hrzgb" event={"ID":"ffda1903-e8ae-4139-bb4c-d01904b23925","Type":"ContainerStarted","Data":"2f2e2f68ac940bd928e973ed88524f8a91fde38cb6b052169e29a3b3c03b80c1"}
Jul 08 12:02:33.972415 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:33.972362    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-hrzgb" podStartSLOduration=11.213919552 podStartE2EDuration="42.97232992s" totalImagesPullingTime="31.758410368s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 12:01:51 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 12:02:33.970894733 +0000 UTC m=+43.701269704" watchObservedRunningTime="2026-07-08 12:02:33.97232992 +0000 UTC m=+43.702704892"
Jul 08 12:02:39.997575 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:39.997531    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6fb2ecb5-0ba5-4b15-9654-76648d18ed3b-metrics-tls\") pod \"dns-default-jprx6\" (UID: \"6fb2ecb5-0ba5-4b15-9654-76648d18ed3b\") " pod="openshift-dns/dns-default-jprx6"
Jul 08 12:02:39.998079 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:02:39.997657    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 08 12:02:39.998079 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:02:39.997715    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fb2ecb5-0ba5-4b15-9654-76648d18ed3b-metrics-tls podName:6fb2ecb5-0ba5-4b15-9654-76648d18ed3b nodeName:}" failed. No retries permitted until 2026-07-08 12:02:55.997699307 +0000 UTC m=+65.728074265 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6fb2ecb5-0ba5-4b15-9654-76648d18ed3b-metrics-tls") pod "dns-default-jprx6" (UID: "6fb2ecb5-0ba5-4b15-9654-76648d18ed3b") : secret "dns-default-metrics-tls" not found
Jul 08 12:02:49.916494 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:49.916466    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xkp4b"
Jul 08 12:02:55.506405 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:55.506348    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a64d26d8-d849-4f64-b43d-3a9133970f52-metrics-certs\") pod \"network-metrics-daemon-m5628\" (UID: \"a64d26d8-d849-4f64-b43d-3a9133970f52\") " pod="openshift-multus/network-metrics-daemon-m5628"
Jul 08 12:02:55.506405 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:55.506408    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9e4572a9-69d7-418c-b751-07c5d6759d5a-original-pull-secret\") pod \"global-pull-secret-syncer-cslr6\" (UID: \"9e4572a9-69d7-418c-b751-07c5d6759d5a\") " pod="kube-system/global-pull-secret-syncer-cslr6"
Jul 08 12:02:55.509371 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:55.509349    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 08 12:02:55.509441 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:55.509362    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 08 12:02:55.517071 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:02:55.517043    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 08 12:02:55.517190 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:02:55.517127    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a64d26d8-d849-4f64-b43d-3a9133970f52-metrics-certs podName:a64d26d8-d849-4f64-b43d-3a9133970f52 nodeName:}" failed. No retries permitted until 2026-07-08 12:03:59.517105217 +0000 UTC m=+129.247480167 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a64d26d8-d849-4f64-b43d-3a9133970f52-metrics-certs") pod "network-metrics-daemon-m5628" (UID: "a64d26d8-d849-4f64-b43d-3a9133970f52") : secret "metrics-daemon-secret" not found
Jul 08 12:02:55.520870 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:55.520843    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9e4572a9-69d7-418c-b751-07c5d6759d5a-original-pull-secret\") pod \"global-pull-secret-syncer-cslr6\" (UID: \"9e4572a9-69d7-418c-b751-07c5d6759d5a\") " pod="kube-system/global-pull-secret-syncer-cslr6"
Jul 08 12:02:55.607750 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:55.607703    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdj4j\" (UniqueName: \"kubernetes.io/projected/ecbe5ccd-c4c5-4820-9238-72591f011842-kube-api-access-sdj4j\") pod \"network-check-target-842d7\" (UID: \"ecbe5ccd-c4c5-4820-9238-72591f011842\") " pod="openshift-network-diagnostics/network-check-target-842d7"
Jul 08 12:02:55.610643 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:55.610625    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 08 12:02:55.621695 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:55.621668    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 08 12:02:55.632412 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:55.632381    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdj4j\" (UniqueName: \"kubernetes.io/projected/ecbe5ccd-c4c5-4820-9238-72591f011842-kube-api-access-sdj4j\") pod \"network-check-target-842d7\" (UID: \"ecbe5ccd-c4c5-4820-9238-72591f011842\") " pod="openshift-network-diagnostics/network-check-target-842d7"
Jul 08 12:02:55.726025 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:55.725991    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hvrx5\""
Jul 08 12:02:55.733683 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:55.733662    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-842d7"
Jul 08 12:02:55.735952 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:55.735930    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cslr6"
Jul 08 12:02:55.871549 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:55.871517    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cslr6"]
Jul 08 12:02:55.874818 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:02:55.874794    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e4572a9_69d7_418c_b751_07c5d6759d5a.slice/crio-6244a0f44ed433a9fcd6359f2893826e7b8521e2dcc02b506c8c18e9eb681533 WatchSource:0}: Error finding container 6244a0f44ed433a9fcd6359f2893826e7b8521e2dcc02b506c8c18e9eb681533: Status 404 returned error can't find the container with id 6244a0f44ed433a9fcd6359f2893826e7b8521e2dcc02b506c8c18e9eb681533
Jul 08 12:02:55.890897 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:55.890874    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-842d7"]
Jul 08 12:02:55.893885 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:02:55.893859    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecbe5ccd_c4c5_4820_9238_72591f011842.slice/crio-c710dc604a7055ef7ce3496ee929dd631edea3ec3f63c8d94944799b3cb6b960 WatchSource:0}: Error finding container c710dc604a7055ef7ce3496ee929dd631edea3ec3f63c8d94944799b3cb6b960: Status 404 returned error can't find the container with id c710dc604a7055ef7ce3496ee929dd631edea3ec3f63c8d94944799b3cb6b960
Jul 08 12:02:55.995959 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:55.995924    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cslr6" event={"ID":"9e4572a9-69d7-418c-b751-07c5d6759d5a","Type":"ContainerStarted","Data":"6244a0f44ed433a9fcd6359f2893826e7b8521e2dcc02b506c8c18e9eb681533"}
Jul 08 12:02:55.996977 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:55.996953    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-842d7" event={"ID":"ecbe5ccd-c4c5-4820-9238-72591f011842","Type":"ContainerStarted","Data":"c710dc604a7055ef7ce3496ee929dd631edea3ec3f63c8d94944799b3cb6b960"}
Jul 08 12:02:56.009514 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:02:56.009486    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6fb2ecb5-0ba5-4b15-9654-76648d18ed3b-metrics-tls\") pod \"dns-default-jprx6\" (UID: \"6fb2ecb5-0ba5-4b15-9654-76648d18ed3b\") " pod="openshift-dns/dns-default-jprx6"
Jul 08 12:02:56.009608 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:02:56.009596    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 08 12:02:56.009651 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:02:56.009647    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fb2ecb5-0ba5-4b15-9654-76648d18ed3b-metrics-tls podName:6fb2ecb5-0ba5-4b15-9654-76648d18ed3b nodeName:}" failed. No retries permitted until 2026-07-08 12:03:28.009633264 +0000 UTC m=+97.740008213 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6fb2ecb5-0ba5-4b15-9654-76648d18ed3b-metrics-tls") pod "dns-default-jprx6" (UID: "6fb2ecb5-0ba5-4b15-9654-76648d18ed3b") : secret "dns-default-metrics-tls" not found
Jul 08 12:03:00.008521 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:00.008488    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-842d7" event={"ID":"ecbe5ccd-c4c5-4820-9238-72591f011842","Type":"ContainerStarted","Data":"515f62dc0a2c9fd1ddb96d4db3a796980f5b172f196f35d45bb3c0502d6b1f48"}
Jul 08 12:03:00.008979 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:00.008608    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-842d7"
Jul 08 12:03:00.027746 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:00.027683    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-842d7" podStartSLOduration=65.773431867 podStartE2EDuration="1m9.027661188s" totalImagesPullingTime="3.254229321s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 12:01:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 12:03:00.026573305 +0000 UTC m=+69.756948280" watchObservedRunningTime="2026-07-08 12:03:00.027661188 +0000 UTC m=+69.758036137"
Jul 08 12:03:02.014831 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:02.014796    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cslr6" event={"ID":"9e4572a9-69d7-418c-b751-07c5d6759d5a","Type":"ContainerStarted","Data":"0963541df95a96cbb23acf7104e69d462a9f5d929503e0c69fc0e3e7582d5e86"}
Jul 08 12:03:02.033739 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:02.033686    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-cslr6" podStartSLOduration=65.557089847 podStartE2EDuration="1m11.033668575s" totalImagesPullingTime="5.476578728s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 12:01:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 12:03:02.032751376 +0000 UTC m=+71.763126347" watchObservedRunningTime="2026-07-08 12:03:02.033668575 +0000 UTC m=+71.764043546"
Jul 08 12:03:23.490266 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:23.490125    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-9bgrq"]
Jul 08 12:03:23.492470 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:23.492447    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-9bgrq"
Jul 08 12:03:23.495329 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:23.495304    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 08 12:03:23.495451 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:23.495328    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 08 12:03:23.495451 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:23.495348    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 08 12:03:23.496410 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:23.496394    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-lgrhs\""
Jul 08 12:03:23.496473 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:23.496399    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 08 12:03:23.501759 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:23.501743    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 08 12:03:23.504844 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:23.504825    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-9bgrq"]
Jul 08 12:03:23.591236 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:23.591198    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ebb356ca-384f-4018-bf6d-9c9a1ee748bd-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-9bgrq\" (UID: \"ebb356ca-384f-4018-bf6d-9c9a1ee748bd\") " pod="openshift-insights/insights-operator-5c847dcfc4-9bgrq"
Jul 08 12:03:23.591419 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:23.591248    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ebb356ca-384f-4018-bf6d-9c9a1ee748bd-serving-cert\") pod \"insights-operator-5c847dcfc4-9bgrq\" (UID: \"ebb356ca-384f-4018-bf6d-9c9a1ee748bd\") " pod="openshift-insights/insights-operator-5c847dcfc4-9bgrq"
Jul 08 12:03:23.591419 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:23.591268    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ebb356ca-384f-4018-bf6d-9c9a1ee748bd-tmp\") pod \"insights-operator-5c847dcfc4-9bgrq\" (UID: \"ebb356ca-384f-4018-bf6d-9c9a1ee748bd\") " pod="openshift-insights/insights-operator-5c847dcfc4-9bgrq"
Jul 08 12:03:23.591419 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:23.591291    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ebb356ca-384f-4018-bf6d-9c9a1ee748bd-snapshots\") pod \"insights-operator-5c847dcfc4-9bgrq\" (UID: \"ebb356ca-384f-4018-bf6d-9c9a1ee748bd\") " pod="openshift-insights/insights-operator-5c847dcfc4-9bgrq"
Jul 08 12:03:23.591419 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:23.591313    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ebb356ca-384f-4018-bf6d-9c9a1ee748bd-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-9bgrq\" (UID: \"ebb356ca-384f-4018-bf6d-9c9a1ee748bd\") " pod="openshift-insights/insights-operator-5c847dcfc4-9bgrq"
Jul 08 12:03:23.591419 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:23.591328    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh5bv\" (UniqueName: \"kubernetes.io/projected/ebb356ca-384f-4018-bf6d-9c9a1ee748bd-kube-api-access-sh5bv\") pod \"insights-operator-5c847dcfc4-9bgrq\" (UID: \"ebb356ca-384f-4018-bf6d-9c9a1ee748bd\") " pod="openshift-insights/insights-operator-5c847dcfc4-9bgrq"
Jul 08 12:03:23.622403 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:23.622368    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-697fbc8d47-wlscv"]
Jul 08 12:03:23.624635 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:23.624615    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-xzlcm"]
Jul 08 12:03:23.624835 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:23.624818    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-697fbc8d47-wlscv"
Jul 08 12:03:23.626025 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:23.626009    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-xzlcm"
Jul 08 12:03:23.643919 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:23.643891    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 08 12:03:23.644215 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:23.644199    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 08 12:03:23.644291 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:23.644200    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 08 12:03:23.644291 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:23.644201    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 08 12:03:23.652150 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:23.652126    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 08 12:03:23.654506 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:23.654489    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 08 12:03:23.655116 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:23.655101    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 08 12:03:23.655371 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:23.655360    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 08 12:03:23.655681 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:23.655665    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 08 12:03:23.656040 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:23.655996    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-stvvn\""
Jul 08 12:03:23.656144 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:23.656107    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 08 12:03:23.656207 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:23.656178    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-4g47q\""
Jul 08 12:03:23.666945 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:23.666913    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-697fbc8d47-wlscv"]
Jul 08 12:03:23.667616 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:23.667592    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-xzlcm"]
Jul 08 12:03:23.692259 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:23.692219    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67af172b-92fa-43c3-a989-4d42968b3d0e-metrics-certs\") pod \"router-default-697fbc8d47-wlscv\" (UID: \"67af172b-92fa-43c3-a989-4d42968b3d0e\") " pod="openshift-ingress/router-default-697fbc8d47-wlscv"
Jul 08 12:03:23.692413 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:23.692275    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ebb356ca-384f-4018-bf6d-9c9a1ee748bd-serving-cert\") pod \"insights-operator-5c847dcfc4-9bgrq\" (UID: \"ebb356ca-384f-4018-bf6d-9c9a1ee748bd\") " pod="openshift-insights/insights-operator-5c847dcfc4-9bgrq"
Jul 08 12:03:23.692413 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:23.692305    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ebb356ca-384f-4018-bf6d-9c9a1ee748bd-tmp\") pod \"insights-operator-5c847dcfc4-9bgrq\" (UID: \"ebb356ca-384f-4018-bf6d-9c9a1ee748bd\") " pod="openshift-insights/insights-operator-5c847dcfc4-9bgrq"
Jul 08 12:03:23.692413 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:23.692326    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ebb356ca-384f-4018-bf6d-9c9a1ee748bd-snapshots\") pod \"insights-operator-5c847dcfc4-9bgrq\" (UID: \"ebb356ca-384f-4018-bf6d-9c9a1ee748bd\") " pod="openshift-insights/insights-operator-5c847dcfc4-9bgrq"
Jul 08 12:03:23.692413 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:23.692349    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ebb356ca-384f-4018-bf6d-9c9a1ee748bd-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-9bgrq\" (UID: \"ebb356ca-384f-4018-bf6d-9c9a1ee748bd\") " pod="openshift-insights/insights-operator-5c847dcfc4-9bgrq"
Jul 08 12:03:23.692413 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:23.692370    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh5bv\" (UniqueName: \"kubernetes.io/projected/ebb356ca-384f-4018-bf6d-9c9a1ee748bd-kube-api-access-sh5bv\") pod \"insights-operator-5c847dcfc4-9bgrq\" (UID: \"ebb356ca-384f-4018-bf6d-9c9a1ee748bd\") " pod="openshift-insights/insights-operator-5c847dcfc4-9bgrq"
Jul 08 12:03:23.692661 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:23.692486    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/67af172b-92fa-43c3-a989-4d42968b3d0e-default-certificate\") pod \"router-default-697fbc8d47-wlscv\" (UID: \"67af172b-92fa-43c3-a989-4d42968b3d0e\") " pod="openshift-ingress/router-default-697fbc8d47-wlscv"
Jul 08 12:03:23.692661 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:23.692531    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/79addf06-f231-4993-8ae6-a9bec4194e1a-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-xzlcm\" (UID: \"79addf06-f231-4993-8ae6-a9bec4194e1a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-xzlcm"
Jul 08 12:03:23.692661 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:23.692558    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl5dp\" (UniqueName: \"kubernetes.io/projected/79addf06-f231-4993-8ae6-a9bec4194e1a-kube-api-access-tl5dp\") pod \"kube-storage-version-migrator-operator-846bf668f7-xzlcm\" (UID: \"79addf06-f231-4993-8ae6-a9bec4194e1a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-xzlcm"
Jul 08 12:03:23.692661 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:23.692593    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/67af172b-92fa-43c3-a989-4d42968b3d0e-stats-auth\") pod \"router-default-697fbc8d47-wlscv\" (UID: \"67af172b-92fa-43c3-a989-4d42968b3d0e\") " pod="openshift-ingress/router-default-697fbc8d47-wlscv"
Jul 08 12:03:23.692661 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:23.692619    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9bbn\" (UniqueName: \"kubernetes.io/projected/67af172b-92fa-43c3-a989-4d42968b3d0e-kube-api-access-h9bbn\") pod \"router-default-697fbc8d47-wlscv\" (UID: \"67af172b-92fa-43c3-a989-4d42968b3d0e\") " pod="openshift-ingress/router-default-697fbc8d47-wlscv"
Jul 08 12:03:23.692661 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:23.692650    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67af172b-92fa-43c3-a989-4d42968b3d0e-service-ca-bundle\") pod \"router-default-697fbc8d47-wlscv\" (UID: \"67af172b-92fa-43c3-a989-4d42968b3d0e\") " pod="openshift-ingress/router-default-697fbc8d47-wlscv"
Jul 08 12:03:23.692985 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:23.692758    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ebb356ca-384f-4018-bf6d-9c9a1ee748bd-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-9bgrq\" (UID: \"ebb356ca-384f-4018-bf6d-9c9a1ee748bd\") " pod="openshift-insights/insights-operator-5c847dcfc4-9bgrq"
Jul 08 12:03:23.692985 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:23.692825    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/79addf06-f231-4993-8ae6-a9bec4194e1a-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-xzlcm\" (UID: \"79addf06-f231-4993-8ae6-a9bec4194e1a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-xzlcm"
Jul 08 12:03:23.693100 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:23.693006    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ebb356ca-384f-4018-bf6d-9c9a1ee748bd-tmp\") pod \"insights-operator-5c847dcfc4-9bgrq\" (UID: \"ebb356ca-384f-4018-bf6d-9c9a1ee748bd\") " pod="openshift-insights/insights-operator-5c847dcfc4-9bgrq"
Jul 08 12:03:23.693100 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:23.693028    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ebb356ca-384f-4018-bf6d-9c9a1ee748bd-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-9bgrq\" (UID: \"ebb356ca-384f-4018-bf6d-9c9a1ee748bd\") " pod="openshift-insights/insights-operator-5c847dcfc4-9bgrq"
Jul 08 12:03:23.693341 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:23.693316    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ebb356ca-384f-4018-bf6d-9c9a1ee748bd-snapshots\") pod \"insights-operator-5c847dcfc4-9bgrq\" (UID: \"ebb356ca-384f-4018-bf6d-9c9a1ee748bd\") " pod="openshift-insights/insights-operator-5c847dcfc4-9bgrq"
Jul 08 12:03:23.693757 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:23.693739    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ebb356ca-384f-4018-bf6d-9c9a1ee748bd-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-9bgrq\" (UID: \"ebb356ca-384f-4018-bf6d-9c9a1ee748bd\") " pod="openshift-insights/insights-operator-5c847dcfc4-9bgrq"
Jul 08 12:03:23.694941 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:23.694917    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ebb356ca-384f-4018-bf6d-9c9a1ee748bd-serving-cert\") pod \"insights-operator-5c847dcfc4-9bgrq\" (UID: \"ebb356ca-384f-4018-bf6d-9c9a1ee748bd\") " pod="openshift-insights/insights-operator-5c847dcfc4-9bgrq"
Jul 08 12:03:23.709230 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:23.709198    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh5bv\" (UniqueName: \"kubernetes.io/projected/ebb356ca-384f-4018-bf6d-9c9a1ee748bd-kube-api-access-sh5bv\") pod \"insights-operator-5c847dcfc4-9bgrq\" (UID: \"ebb356ca-384f-4018-bf6d-9c9a1ee748bd\") " pod="openshift-insights/insights-operator-5c847dcfc4-9bgrq"
Jul 08 12:03:23.794121 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:23.794004    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/67af172b-92fa-43c3-a989-4d42968b3d0e-default-certificate\") pod \"router-default-697fbc8d47-wlscv\" (UID: \"67af172b-92fa-43c3-a989-4d42968b3d0e\") " pod="openshift-ingress/router-default-697fbc8d47-wlscv"
Jul 08 12:03:23.794121 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:23.794058    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/79addf06-f231-4993-8ae6-a9bec4194e1a-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-xzlcm\" (UID: \"79addf06-f231-4993-8ae6-a9bec4194e1a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-xzlcm"
Jul 08 12:03:23.794121 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:23.794075    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl5dp\" (UniqueName: \"kubernetes.io/projected/79addf06-f231-4993-8ae6-a9bec4194e1a-kube-api-access-tl5dp\") pod \"kube-storage-version-migrator-operator-846bf668f7-xzlcm\" (UID: \"79addf06-f231-4993-8ae6-a9bec4194e1a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-xzlcm"
Jul 08 12:03:23.794121 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:23.794093    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/67af172b-92fa-43c3-a989-4d42968b3d0e-stats-auth\") pod \"router-default-697fbc8d47-wlscv\" (UID: \"67af172b-92fa-43c3-a989-4d42968b3d0e\") " pod="openshift-ingress/router-default-697fbc8d47-wlscv"
Jul 08 12:03:23.794121 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:23.794119    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9bbn\" (UniqueName: \"kubernetes.io/projected/67af172b-92fa-43c3-a989-4d42968b3d0e-kube-api-access-h9bbn\") pod \"router-default-697fbc8d47-wlscv\" (UID: \"67af172b-92fa-43c3-a989-4d42968b3d0e\") " pod="openshift-ingress/router-default-697fbc8d47-wlscv"
Jul 08 12:03:23.794496 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:23.794333    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67af172b-92fa-43c3-a989-4d42968b3d0e-service-ca-bundle\") pod \"router-default-697fbc8d47-wlscv\" (UID: \"67af172b-92fa-43c3-a989-4d42968b3d0e\") " pod="openshift-ingress/router-default-697fbc8d47-wlscv"
Jul 08 12:03:23.794496 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:23.794435    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/79addf06-f231-4993-8ae6-a9bec4194e1a-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-xzlcm\" (UID: \"79addf06-f231-4993-8ae6-a9bec4194e1a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-xzlcm"
Jul 08 12:03:23.794496 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:03:23.794465    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/67af172b-92fa-43c3-a989-4d42968b3d0e-service-ca-bundle podName:67af172b-92fa-43c3-a989-4d42968b3d0e nodeName:}" failed. No retries permitted until 2026-07-08 12:03:24.294444289 +0000 UTC m=+94.024819242 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/67af172b-92fa-43c3-a989-4d42968b3d0e-service-ca-bundle") pod "router-default-697fbc8d47-wlscv" (UID: "67af172b-92fa-43c3-a989-4d42968b3d0e") : configmap references non-existent config key: service-ca.crt
Jul 08 12:03:23.794658 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:23.794512    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67af172b-92fa-43c3-a989-4d42968b3d0e-metrics-certs\") pod \"router-default-697fbc8d47-wlscv\" (UID: \"67af172b-92fa-43c3-a989-4d42968b3d0e\") " pod="openshift-ingress/router-default-697fbc8d47-wlscv"
Jul 08 12:03:23.794658 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:03:23.794604    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 08 12:03:23.794742 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:03:23.794717    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67af172b-92fa-43c3-a989-4d42968b3d0e-metrics-certs podName:67af172b-92fa-43c3-a989-4d42968b3d0e nodeName:}" failed. No retries permitted until 2026-07-08 12:03:24.294700384 +0000 UTC m=+94.025075337 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/67af172b-92fa-43c3-a989-4d42968b3d0e-metrics-certs") pod "router-default-697fbc8d47-wlscv" (UID: "67af172b-92fa-43c3-a989-4d42968b3d0e") : secret "router-metrics-certs-default" not found
Jul 08 12:03:23.795173 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:23.795146    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/79addf06-f231-4993-8ae6-a9bec4194e1a-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-xzlcm\" (UID: \"79addf06-f231-4993-8ae6-a9bec4194e1a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-xzlcm"
Jul 08 12:03:23.796784 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:23.796739    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/79addf06-f231-4993-8ae6-a9bec4194e1a-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-xzlcm\" (UID: \"79addf06-f231-4993-8ae6-a9bec4194e1a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-xzlcm"
Jul 08 12:03:23.796890 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:23.796866    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/67af172b-92fa-43c3-a989-4d42968b3d0e-stats-auth\") pod \"router-default-697fbc8d47-wlscv\" (UID: \"67af172b-92fa-43c3-a989-4d42968b3d0e\") " pod="openshift-ingress/router-default-697fbc8d47-wlscv"
Jul 08 12:03:23.796932 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:23.796896    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/67af172b-92fa-43c3-a989-4d42968b3d0e-default-certificate\") pod \"router-default-697fbc8d47-wlscv\" (UID: \"67af172b-92fa-43c3-a989-4d42968b3d0e\") " pod="openshift-ingress/router-default-697fbc8d47-wlscv"
Jul 08 12:03:23.802356 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:23.802329    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-9bgrq"
Jul 08 12:03:23.804442 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:23.804422    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9bbn\" (UniqueName: \"kubernetes.io/projected/67af172b-92fa-43c3-a989-4d42968b3d0e-kube-api-access-h9bbn\") pod \"router-default-697fbc8d47-wlscv\" (UID: \"67af172b-92fa-43c3-a989-4d42968b3d0e\") " pod="openshift-ingress/router-default-697fbc8d47-wlscv"
Jul 08 12:03:23.804442 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:23.804444    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl5dp\" (UniqueName: \"kubernetes.io/projected/79addf06-f231-4993-8ae6-a9bec4194e1a-kube-api-access-tl5dp\") pod \"kube-storage-version-migrator-operator-846bf668f7-xzlcm\" (UID: \"79addf06-f231-4993-8ae6-a9bec4194e1a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-xzlcm"
Jul 08 12:03:23.927999 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:23.927944    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-9bgrq"]
Jul 08 12:03:23.930144 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:03:23.930116    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebb356ca_384f_4018_bf6d_9c9a1ee748bd.slice/crio-166a89bfa7052e7dd2f5318da3d61b384faf878b43946f564ca167a7c85966ba WatchSource:0}: Error finding container 166a89bfa7052e7dd2f5318da3d61b384faf878b43946f564ca167a7c85966ba: Status 404 returned error can't find the container with id 166a89bfa7052e7dd2f5318da3d61b384faf878b43946f564ca167a7c85966ba
Jul 08 12:03:23.937875 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:23.937852    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-xzlcm"
Jul 08 12:03:24.063591 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:24.063513    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-xzlcm"]
Jul 08 12:03:24.067223 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:03:24.067198    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79addf06_f231_4993_8ae6_a9bec4194e1a.slice/crio-c4c8463946679de743719a627a49a610c5e5fb73f6f38d5914e04b5a46b23d31 WatchSource:0}: Error finding container c4c8463946679de743719a627a49a610c5e5fb73f6f38d5914e04b5a46b23d31: Status 404 returned error can't find the container with id c4c8463946679de743719a627a49a610c5e5fb73f6f38d5914e04b5a46b23d31
Jul 08 12:03:24.070120 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:24.070096    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-9bgrq" event={"ID":"ebb356ca-384f-4018-bf6d-9c9a1ee748bd","Type":"ContainerStarted","Data":"166a89bfa7052e7dd2f5318da3d61b384faf878b43946f564ca167a7c85966ba"}
Jul 08 12:03:24.298319 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:24.298274    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67af172b-92fa-43c3-a989-4d42968b3d0e-metrics-certs\") pod \"router-default-697fbc8d47-wlscv\" (UID: \"67af172b-92fa-43c3-a989-4d42968b3d0e\") " pod="openshift-ingress/router-default-697fbc8d47-wlscv"
Jul 08 12:03:24.298503 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:24.298348    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67af172b-92fa-43c3-a989-4d42968b3d0e-service-ca-bundle\") pod \"router-default-697fbc8d47-wlscv\" (UID: \"67af172b-92fa-43c3-a989-4d42968b3d0e\") " pod="openshift-ingress/router-default-697fbc8d47-wlscv"
Jul 08 12:03:24.298503 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:03:24.298436    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 08 12:03:24.298503 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:03:24.298479    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/67af172b-92fa-43c3-a989-4d42968b3d0e-service-ca-bundle podName:67af172b-92fa-43c3-a989-4d42968b3d0e nodeName:}" failed. No retries permitted until 2026-07-08 12:03:25.298459603 +0000 UTC m=+95.028834553 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/67af172b-92fa-43c3-a989-4d42968b3d0e-service-ca-bundle") pod "router-default-697fbc8d47-wlscv" (UID: "67af172b-92fa-43c3-a989-4d42968b3d0e") : configmap references non-existent config key: service-ca.crt
Jul 08 12:03:24.298503 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:03:24.298496    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67af172b-92fa-43c3-a989-4d42968b3d0e-metrics-certs podName:67af172b-92fa-43c3-a989-4d42968b3d0e nodeName:}" failed. No retries permitted until 2026-07-08 12:03:25.298489074 +0000 UTC m=+95.028864023 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/67af172b-92fa-43c3-a989-4d42968b3d0e-metrics-certs") pod "router-default-697fbc8d47-wlscv" (UID: "67af172b-92fa-43c3-a989-4d42968b3d0e") : secret "router-metrics-certs-default" not found
Jul 08 12:03:25.074432 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:25.074395    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-xzlcm" event={"ID":"79addf06-f231-4993-8ae6-a9bec4194e1a","Type":"ContainerStarted","Data":"c4c8463946679de743719a627a49a610c5e5fb73f6f38d5914e04b5a46b23d31"}
Jul 08 12:03:25.306672 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:25.306635    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67af172b-92fa-43c3-a989-4d42968b3d0e-metrics-certs\") pod \"router-default-697fbc8d47-wlscv\" (UID: \"67af172b-92fa-43c3-a989-4d42968b3d0e\") " pod="openshift-ingress/router-default-697fbc8d47-wlscv"
Jul 08 12:03:25.306893 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:25.306716    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67af172b-92fa-43c3-a989-4d42968b3d0e-service-ca-bundle\") pod \"router-default-697fbc8d47-wlscv\" (UID: \"67af172b-92fa-43c3-a989-4d42968b3d0e\") " pod="openshift-ingress/router-default-697fbc8d47-wlscv"
Jul 08 12:03:25.306893 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:03:25.306819    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 08 12:03:25.306893 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:03:25.306865    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/67af172b-92fa-43c3-a989-4d42968b3d0e-service-ca-bundle podName:67af172b-92fa-43c3-a989-4d42968b3d0e nodeName:}" failed. No retries permitted until 2026-07-08 12:03:27.306843759 +0000 UTC m=+97.037218713 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/67af172b-92fa-43c3-a989-4d42968b3d0e-service-ca-bundle") pod "router-default-697fbc8d47-wlscv" (UID: "67af172b-92fa-43c3-a989-4d42968b3d0e") : configmap references non-existent config key: service-ca.crt
Jul 08 12:03:25.306893 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:03:25.306890    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67af172b-92fa-43c3-a989-4d42968b3d0e-metrics-certs podName:67af172b-92fa-43c3-a989-4d42968b3d0e nodeName:}" failed. No retries permitted until 2026-07-08 12:03:27.306872876 +0000 UTC m=+97.037247828 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/67af172b-92fa-43c3-a989-4d42968b3d0e-metrics-certs") pod "router-default-697fbc8d47-wlscv" (UID: "67af172b-92fa-43c3-a989-4d42968b3d0e") : secret "router-metrics-certs-default" not found
Jul 08 12:03:27.081541 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:27.081500    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-xzlcm" event={"ID":"79addf06-f231-4993-8ae6-a9bec4194e1a","Type":"ContainerStarted","Data":"273d2f0ac5773b8f47d932ddd401e1ec9e206f93cf71e57b5ed99fe93792bfa6"}
Jul 08 12:03:27.083027 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:27.083004    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-9bgrq" event={"ID":"ebb356ca-384f-4018-bf6d-9c9a1ee748bd","Type":"ContainerStarted","Data":"50bf9dd6f0102b82bb2fef0d20b344abce2b9ce87aa66dcb123f649a4adfbad0"}
Jul 08 12:03:27.104712 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:27.104657    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-xzlcm" podStartSLOduration=1.2575231470000001 podStartE2EDuration="4.104643361s" totalImagesPullingTime="2.847120214s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 12:03:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 12:03:27.103734638 +0000 UTC m=+96.834109611" watchObservedRunningTime="2026-07-08 12:03:27.104643361 +0000 UTC m=+96.835018330"
Jul 08 12:03:27.121067 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:27.121009    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5c847dcfc4-9bgrq" podStartSLOduration=1.138404228 podStartE2EDuration="4.12098977s" totalImagesPullingTime="2.982585542s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 12:03:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 12:03:27.120371794 +0000 UTC m=+96.850746763" watchObservedRunningTime="2026-07-08 12:03:27.12098977 +0000 UTC m=+96.851364745"
Jul 08 12:03:27.322478 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:27.322416    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67af172b-92fa-43c3-a989-4d42968b3d0e-service-ca-bundle\") pod \"router-default-697fbc8d47-wlscv\" (UID: \"67af172b-92fa-43c3-a989-4d42968b3d0e\") " pod="openshift-ingress/router-default-697fbc8d47-wlscv"
Jul 08 12:03:27.322693 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:27.322517    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67af172b-92fa-43c3-a989-4d42968b3d0e-metrics-certs\") pod \"router-default-697fbc8d47-wlscv\" (UID: \"67af172b-92fa-43c3-a989-4d42968b3d0e\") " pod="openshift-ingress/router-default-697fbc8d47-wlscv"
Jul 08 12:03:27.322693 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:03:27.322591    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/67af172b-92fa-43c3-a989-4d42968b3d0e-service-ca-bundle podName:67af172b-92fa-43c3-a989-4d42968b3d0e nodeName:}" failed. No retries permitted until 2026-07-08 12:03:31.322570756 +0000 UTC m=+101.052945705 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/67af172b-92fa-43c3-a989-4d42968b3d0e-service-ca-bundle") pod "router-default-697fbc8d47-wlscv" (UID: "67af172b-92fa-43c3-a989-4d42968b3d0e") : configmap references non-existent config key: service-ca.crt
Jul 08 12:03:27.322693 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:03:27.322613    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 08 12:03:27.322693 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:03:27.322661    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67af172b-92fa-43c3-a989-4d42968b3d0e-metrics-certs podName:67af172b-92fa-43c3-a989-4d42968b3d0e nodeName:}" failed. No retries permitted until 2026-07-08 12:03:31.322644806 +0000 UTC m=+101.053019763 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/67af172b-92fa-43c3-a989-4d42968b3d0e-metrics-certs") pod "router-default-697fbc8d47-wlscv" (UID: "67af172b-92fa-43c3-a989-4d42968b3d0e") : secret "router-metrics-certs-default" not found
Jul 08 12:03:28.028752 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:28.028712    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6fb2ecb5-0ba5-4b15-9654-76648d18ed3b-metrics-tls\") pod \"dns-default-jprx6\" (UID: \"6fb2ecb5-0ba5-4b15-9654-76648d18ed3b\") " pod="openshift-dns/dns-default-jprx6"
Jul 08 12:03:28.028933 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:03:28.028883    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 08 12:03:28.028999 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:03:28.028949    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fb2ecb5-0ba5-4b15-9654-76648d18ed3b-metrics-tls podName:6fb2ecb5-0ba5-4b15-9654-76648d18ed3b nodeName:}" failed. No retries permitted until 2026-07-08 12:04:32.028932488 +0000 UTC m=+161.759307437 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6fb2ecb5-0ba5-4b15-9654-76648d18ed3b-metrics-tls") pod "dns-default-jprx6" (UID: "6fb2ecb5-0ba5-4b15-9654-76648d18ed3b") : secret "dns-default-metrics-tls" not found
Jul 08 12:03:28.075200 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:28.075167    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-8mcpn"]
Jul 08 12:03:28.077275 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:28.077260    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8mcpn"
Jul 08 12:03:28.082837 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:28.082815    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 08 12:03:28.083234 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:28.083054    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 08 12:03:28.084128 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:28.084104    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6nfnf\""
Jul 08 12:03:28.091948 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:28.091921    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8mcpn"]
Jul 08 12:03:28.129727 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:28.129684    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/52314101-63ee-448f-963f-16b1dd3cdedb-crio-socket\") pod \"insights-runtime-extractor-8mcpn\" (UID: \"52314101-63ee-448f-963f-16b1dd3cdedb\") " pod="openshift-insights/insights-runtime-extractor-8mcpn"
Jul 08 12:03:28.129930 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:28.129821    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/52314101-63ee-448f-963f-16b1dd3cdedb-data-volume\") pod \"insights-runtime-extractor-8mcpn\" (UID: \"52314101-63ee-448f-963f-16b1dd3cdedb\") " pod="openshift-insights/insights-runtime-extractor-8mcpn"
Jul 08 12:03:28.129930 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:28.129852    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/52314101-63ee-448f-963f-16b1dd3cdedb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8mcpn\" (UID: \"52314101-63ee-448f-963f-16b1dd3cdedb\") " pod="openshift-insights/insights-runtime-extractor-8mcpn"
Jul 08 12:03:28.129930 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:28.129879    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qghg6\" (UniqueName: \"kubernetes.io/projected/52314101-63ee-448f-963f-16b1dd3cdedb-kube-api-access-qghg6\") pod \"insights-runtime-extractor-8mcpn\" (UID: \"52314101-63ee-448f-963f-16b1dd3cdedb\") " pod="openshift-insights/insights-runtime-extractor-8mcpn"
Jul 08 12:03:28.130074 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:28.130015    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/52314101-63ee-448f-963f-16b1dd3cdedb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8mcpn\" (UID: \"52314101-63ee-448f-963f-16b1dd3cdedb\") " pod="openshift-insights/insights-runtime-extractor-8mcpn"
Jul 08 12:03:28.231091 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:28.231054    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/52314101-63ee-448f-963f-16b1dd3cdedb-crio-socket\") pod \"insights-runtime-extractor-8mcpn\" (UID: \"52314101-63ee-448f-963f-16b1dd3cdedb\") " pod="openshift-insights/insights-runtime-extractor-8mcpn"
Jul 08 12:03:28.231091 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:28.231098    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/52314101-63ee-448f-963f-16b1dd3cdedb-data-volume\") pod \"insights-runtime-extractor-8mcpn\" (UID: \"52314101-63ee-448f-963f-16b1dd3cdedb\") " pod="openshift-insights/insights-runtime-extractor-8mcpn"
Jul 08 12:03:28.231348 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:28.231115    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/52314101-63ee-448f-963f-16b1dd3cdedb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8mcpn\" (UID: \"52314101-63ee-448f-963f-16b1dd3cdedb\") " pod="openshift-insights/insights-runtime-extractor-8mcpn"
Jul 08 12:03:28.231348 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:03:28.231182    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 08 12:03:28.231348 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:28.231194    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/52314101-63ee-448f-963f-16b1dd3cdedb-crio-socket\") pod \"insights-runtime-extractor-8mcpn\" (UID: \"52314101-63ee-448f-963f-16b1dd3cdedb\") " pod="openshift-insights/insights-runtime-extractor-8mcpn"
Jul 08 12:03:28.231348 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:03:28.231244    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/52314101-63ee-448f-963f-16b1dd3cdedb-kube-rbac-proxy-cm podName:52314101-63ee-448f-963f-16b1dd3cdedb nodeName:}" failed. No retries permitted until 2026-07-08 12:03:28.731226229 +0000 UTC m=+98.461601198 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/52314101-63ee-448f-963f-16b1dd3cdedb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8mcpn" (UID: "52314101-63ee-448f-963f-16b1dd3cdedb") : configmap "kube-rbac-proxy" not found
Jul 08 12:03:28.231348 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:28.231259    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qghg6\" (UniqueName: \"kubernetes.io/projected/52314101-63ee-448f-963f-16b1dd3cdedb-kube-api-access-qghg6\") pod \"insights-runtime-extractor-8mcpn\" (UID: \"52314101-63ee-448f-963f-16b1dd3cdedb\") " pod="openshift-insights/insights-runtime-extractor-8mcpn"
Jul 08 12:03:28.231348 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:28.231322    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/52314101-63ee-448f-963f-16b1dd3cdedb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8mcpn\" (UID: \"52314101-63ee-448f-963f-16b1dd3cdedb\") " pod="openshift-insights/insights-runtime-extractor-8mcpn"
Jul 08 12:03:28.231727 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:03:28.231437    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 08 12:03:28.231727 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:28.231457    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/52314101-63ee-448f-963f-16b1dd3cdedb-data-volume\") pod \"insights-runtime-extractor-8mcpn\" (UID: \"52314101-63ee-448f-963f-16b1dd3cdedb\") " pod="openshift-insights/insights-runtime-extractor-8mcpn"
Jul 08 12:03:28.231727 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:03:28.231482    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52314101-63ee-448f-963f-16b1dd3cdedb-insights-runtime-extractor-tls podName:52314101-63ee-448f-963f-16b1dd3cdedb nodeName:}" failed. No retries permitted until 2026-07-08 12:03:28.731471402 +0000 UTC m=+98.461846352 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/52314101-63ee-448f-963f-16b1dd3cdedb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8mcpn" (UID: "52314101-63ee-448f-963f-16b1dd3cdedb") : secret "insights-runtime-extractor-tls" not found
Jul 08 12:03:28.245219 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:28.245191    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qghg6\" (UniqueName: \"kubernetes.io/projected/52314101-63ee-448f-963f-16b1dd3cdedb-kube-api-access-qghg6\") pod \"insights-runtime-extractor-8mcpn\" (UID: \"52314101-63ee-448f-963f-16b1dd3cdedb\") " pod="openshift-insights/insights-runtime-extractor-8mcpn"
Jul 08 12:03:28.734429 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:28.734397    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/52314101-63ee-448f-963f-16b1dd3cdedb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8mcpn\" (UID: \"52314101-63ee-448f-963f-16b1dd3cdedb\") " pod="openshift-insights/insights-runtime-extractor-8mcpn"
Jul 08 12:03:28.734429 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:28.734446    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/52314101-63ee-448f-963f-16b1dd3cdedb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8mcpn\" (UID: \"52314101-63ee-448f-963f-16b1dd3cdedb\") " pod="openshift-insights/insights-runtime-extractor-8mcpn"
Jul 08 12:03:28.734832 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:03:28.734531    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 08 12:03:28.734832 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:03:28.734579    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 08 12:03:28.734832 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:03:28.734601    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/52314101-63ee-448f-963f-16b1dd3cdedb-kube-rbac-proxy-cm podName:52314101-63ee-448f-963f-16b1dd3cdedb nodeName:}" failed. No retries permitted until 2026-07-08 12:03:29.734581841 +0000 UTC m=+99.464956796 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/52314101-63ee-448f-963f-16b1dd3cdedb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8mcpn" (UID: "52314101-63ee-448f-963f-16b1dd3cdedb") : configmap "kube-rbac-proxy" not found
Jul 08 12:03:28.734832 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:03:28.734619    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52314101-63ee-448f-963f-16b1dd3cdedb-insights-runtime-extractor-tls podName:52314101-63ee-448f-963f-16b1dd3cdedb nodeName:}" failed. No retries permitted until 2026-07-08 12:03:29.734607741 +0000 UTC m=+99.464982703 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/52314101-63ee-448f-963f-16b1dd3cdedb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8mcpn" (UID: "52314101-63ee-448f-963f-16b1dd3cdedb") : secret "insights-runtime-extractor-tls" not found
Jul 08 12:03:28.781165 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:28.781131    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-crpsg"]
Jul 08 12:03:28.783761 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:28.783732    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-crpsg"
Jul 08 12:03:28.787554 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:28.787529    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-vw26j\""
Jul 08 12:03:28.787685 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:28.787529    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 08 12:03:28.787685 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:28.787586    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 08 12:03:28.794634 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:28.794610    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-crpsg"]
Jul 08 12:03:28.835353 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:28.835315    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sn65\" (UniqueName: \"kubernetes.io/projected/738db72c-994e-43df-a7e7-53fba16a575a-kube-api-access-4sn65\") pod \"migrator-d57d58cc9-crpsg\" (UID: \"738db72c-994e-43df-a7e7-53fba16a575a\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-crpsg"
Jul 08 12:03:28.936643 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:28.936602    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sn65\" (UniqueName: \"kubernetes.io/projected/738db72c-994e-43df-a7e7-53fba16a575a-kube-api-access-4sn65\") pod \"migrator-d57d58cc9-crpsg\" (UID: \"738db72c-994e-43df-a7e7-53fba16a575a\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-crpsg"
Jul 08 12:03:28.947281 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:28.947246    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sn65\" (UniqueName: \"kubernetes.io/projected/738db72c-994e-43df-a7e7-53fba16a575a-kube-api-access-4sn65\") pod \"migrator-d57d58cc9-crpsg\" (UID: \"738db72c-994e-43df-a7e7-53fba16a575a\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-crpsg"
Jul 08 12:03:29.092738 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:29.092699    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-crpsg"
Jul 08 12:03:29.216493 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:29.216462    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-crpsg"]
Jul 08 12:03:29.219410 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:03:29.219379    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod738db72c_994e_43df_a7e7_53fba16a575a.slice/crio-46188b4564781c8e234688b1595e2518213308381468ebc8f2b354cc6d1142ae WatchSource:0}: Error finding container 46188b4564781c8e234688b1595e2518213308381468ebc8f2b354cc6d1142ae: Status 404 returned error can't find the container with id 46188b4564781c8e234688b1595e2518213308381468ebc8f2b354cc6d1142ae
Jul 08 12:03:29.743926 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:29.743884    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/52314101-63ee-448f-963f-16b1dd3cdedb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8mcpn\" (UID: \"52314101-63ee-448f-963f-16b1dd3cdedb\") " pod="openshift-insights/insights-runtime-extractor-8mcpn"
Jul 08 12:03:29.744121 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:03:29.743973    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 08 12:03:29.744121 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:29.743978    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/52314101-63ee-448f-963f-16b1dd3cdedb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8mcpn\" (UID: \"52314101-63ee-448f-963f-16b1dd3cdedb\") " pod="openshift-insights/insights-runtime-extractor-8mcpn"
Jul 08 12:03:29.744121 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:03:29.744027    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/52314101-63ee-448f-963f-16b1dd3cdedb-kube-rbac-proxy-cm podName:52314101-63ee-448f-963f-16b1dd3cdedb nodeName:}" failed. No retries permitted until 2026-07-08 12:03:31.744013564 +0000 UTC m=+101.474388513 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/52314101-63ee-448f-963f-16b1dd3cdedb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8mcpn" (UID: "52314101-63ee-448f-963f-16b1dd3cdedb") : configmap "kube-rbac-proxy" not found
Jul 08 12:03:29.744121 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:03:29.744100    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 08 12:03:29.744288 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:03:29.744146    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52314101-63ee-448f-963f-16b1dd3cdedb-insights-runtime-extractor-tls podName:52314101-63ee-448f-963f-16b1dd3cdedb nodeName:}" failed. No retries permitted until 2026-07-08 12:03:31.744135067 +0000 UTC m=+101.474510017 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/52314101-63ee-448f-963f-16b1dd3cdedb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8mcpn" (UID: "52314101-63ee-448f-963f-16b1dd3cdedb") : secret "insights-runtime-extractor-tls" not found
Jul 08 12:03:29.851886 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:29.851857    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-26hpb_c0abdbfc-0312-4fd1-9ca4-098e3fa3e7e1/dns-node-resolver/0.log"
Jul 08 12:03:30.094598 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:30.094504    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-crpsg" event={"ID":"738db72c-994e-43df-a7e7-53fba16a575a","Type":"ContainerStarted","Data":"46188b4564781c8e234688b1595e2518213308381468ebc8f2b354cc6d1142ae"}
Jul 08 12:03:31.012980 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:31.012954    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-842d7"
Jul 08 12:03:31.051028 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:31.050999    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-k4jsh_641b596d-df5a-491a-a40e-e269de0a676e/node-ca/0.log"
Jul 08 12:03:31.098653 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:31.098621    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-crpsg" event={"ID":"738db72c-994e-43df-a7e7-53fba16a575a","Type":"ContainerStarted","Data":"ede3693d752e51526552a6b0059c1eb4cb2cbb9e713ece58e0a08a3aff2c0abc"}
Jul 08 12:03:31.099040 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:31.098671    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-crpsg" event={"ID":"738db72c-994e-43df-a7e7-53fba16a575a","Type":"ContainerStarted","Data":"9a82396f7aeb236c2a3837aff66909d8cd4c284d5035cc76fe597d1bef98b765"}
Jul 08 12:03:31.117358 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:31.117311    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-crpsg" podStartSLOduration=1.727212953 podStartE2EDuration="3.117295307s" totalImagesPullingTime="1.390082354s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 12:03:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 12:03:31.116589981 +0000 UTC m=+100.846964951" watchObservedRunningTime="2026-07-08 12:03:31.117295307 +0000 UTC m=+100.847670277"
Jul 08 12:03:31.356603 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:31.356514    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67af172b-92fa-43c3-a989-4d42968b3d0e-service-ca-bundle\") pod \"router-default-697fbc8d47-wlscv\" (UID: \"67af172b-92fa-43c3-a989-4d42968b3d0e\") " pod="openshift-ingress/router-default-697fbc8d47-wlscv"
Jul 08 12:03:31.356603 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:31.356593    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67af172b-92fa-43c3-a989-4d42968b3d0e-metrics-certs\") pod \"router-default-697fbc8d47-wlscv\" (UID: \"67af172b-92fa-43c3-a989-4d42968b3d0e\") " pod="openshift-ingress/router-default-697fbc8d47-wlscv"
Jul 08 12:03:31.356839 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:03:31.356687    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/67af172b-92fa-43c3-a989-4d42968b3d0e-service-ca-bundle podName:67af172b-92fa-43c3-a989-4d42968b3d0e nodeName:}" failed. No retries permitted until 2026-07-08 12:03:39.356670576 +0000 UTC m=+109.087045525 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/67af172b-92fa-43c3-a989-4d42968b3d0e-service-ca-bundle") pod "router-default-697fbc8d47-wlscv" (UID: "67af172b-92fa-43c3-a989-4d42968b3d0e") : configmap references non-existent config key: service-ca.crt
Jul 08 12:03:31.356839 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:03:31.356693    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 08 12:03:31.356839 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:03:31.356741    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67af172b-92fa-43c3-a989-4d42968b3d0e-metrics-certs podName:67af172b-92fa-43c3-a989-4d42968b3d0e nodeName:}" failed. No retries permitted until 2026-07-08 12:03:39.356728791 +0000 UTC m=+109.087103740 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/67af172b-92fa-43c3-a989-4d42968b3d0e-metrics-certs") pod "router-default-697fbc8d47-wlscv" (UID: "67af172b-92fa-43c3-a989-4d42968b3d0e") : secret "router-metrics-certs-default" not found
Jul 08 12:03:31.759536 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:31.759493    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/52314101-63ee-448f-963f-16b1dd3cdedb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8mcpn\" (UID: \"52314101-63ee-448f-963f-16b1dd3cdedb\") " pod="openshift-insights/insights-runtime-extractor-8mcpn"
Jul 08 12:03:31.759746 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:31.759559    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/52314101-63ee-448f-963f-16b1dd3cdedb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8mcpn\" (UID: \"52314101-63ee-448f-963f-16b1dd3cdedb\") " pod="openshift-insights/insights-runtime-extractor-8mcpn"
Jul 08 12:03:31.759746 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:03:31.759612    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 08 12:03:31.759746 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:03:31.759628    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 08 12:03:31.759746 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:03:31.759672    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/52314101-63ee-448f-963f-16b1dd3cdedb-kube-rbac-proxy-cm podName:52314101-63ee-448f-963f-16b1dd3cdedb nodeName:}" failed. No retries permitted until 2026-07-08 12:03:35.759657029 +0000 UTC m=+105.490031978 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/52314101-63ee-448f-963f-16b1dd3cdedb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8mcpn" (UID: "52314101-63ee-448f-963f-16b1dd3cdedb") : configmap "kube-rbac-proxy" not found
Jul 08 12:03:31.759746 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:03:31.759686    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52314101-63ee-448f-963f-16b1dd3cdedb-insights-runtime-extractor-tls podName:52314101-63ee-448f-963f-16b1dd3cdedb nodeName:}" failed. No retries permitted until 2026-07-08 12:03:35.75968029 +0000 UTC m=+105.490055239 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/52314101-63ee-448f-963f-16b1dd3cdedb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8mcpn" (UID: "52314101-63ee-448f-963f-16b1dd3cdedb") : secret "insights-runtime-extractor-tls" not found
Jul 08 12:03:31.851353 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:31.851272    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-nn7z8_d8e14b6c-7fd3-46b6-8e03-f8c03d924e2b/serve-healthcheck-canary/0.log"
Jul 08 12:03:35.791829 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:35.791789    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/52314101-63ee-448f-963f-16b1dd3cdedb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8mcpn\" (UID: \"52314101-63ee-448f-963f-16b1dd3cdedb\") " pod="openshift-insights/insights-runtime-extractor-8mcpn"
Jul 08 12:03:35.792304 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:35.791914    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/52314101-63ee-448f-963f-16b1dd3cdedb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8mcpn\" (UID: \"52314101-63ee-448f-963f-16b1dd3cdedb\") " pod="openshift-insights/insights-runtime-extractor-8mcpn"
Jul 08 12:03:35.792304 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:03:35.791953    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 08 12:03:35.792304 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:03:35.792020    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52314101-63ee-448f-963f-16b1dd3cdedb-insights-runtime-extractor-tls podName:52314101-63ee-448f-963f-16b1dd3cdedb nodeName:}" failed. No retries permitted until 2026-07-08 12:03:43.792002736 +0000 UTC m=+113.522377685 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/52314101-63ee-448f-963f-16b1dd3cdedb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8mcpn" (UID: "52314101-63ee-448f-963f-16b1dd3cdedb") : secret "insights-runtime-extractor-tls" not found
Jul 08 12:03:35.792304 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:03:35.792029    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 08 12:03:35.792304 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:03:35.792079    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/52314101-63ee-448f-963f-16b1dd3cdedb-kube-rbac-proxy-cm podName:52314101-63ee-448f-963f-16b1dd3cdedb nodeName:}" failed. No retries permitted until 2026-07-08 12:03:43.792064017 +0000 UTC m=+113.522438970 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/52314101-63ee-448f-963f-16b1dd3cdedb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8mcpn" (UID: "52314101-63ee-448f-963f-16b1dd3cdedb") : configmap "kube-rbac-proxy" not found
Jul 08 12:03:39.421164 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:39.421103    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67af172b-92fa-43c3-a989-4d42968b3d0e-service-ca-bundle\") pod \"router-default-697fbc8d47-wlscv\" (UID: \"67af172b-92fa-43c3-a989-4d42968b3d0e\") " pod="openshift-ingress/router-default-697fbc8d47-wlscv"
Jul 08 12:03:39.421560 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:39.421205    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67af172b-92fa-43c3-a989-4d42968b3d0e-metrics-certs\") pod \"router-default-697fbc8d47-wlscv\" (UID: \"67af172b-92fa-43c3-a989-4d42968b3d0e\") " pod="openshift-ingress/router-default-697fbc8d47-wlscv"
Jul 08 12:03:39.421763 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:39.421742    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67af172b-92fa-43c3-a989-4d42968b3d0e-service-ca-bundle\") pod \"router-default-697fbc8d47-wlscv\" (UID: \"67af172b-92fa-43c3-a989-4d42968b3d0e\") " pod="openshift-ingress/router-default-697fbc8d47-wlscv"
Jul 08 12:03:39.423593 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:39.423574    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67af172b-92fa-43c3-a989-4d42968b3d0e-metrics-certs\") pod \"router-default-697fbc8d47-wlscv\" (UID: \"67af172b-92fa-43c3-a989-4d42968b3d0e\") " pod="openshift-ingress/router-default-697fbc8d47-wlscv"
Jul 08 12:03:39.533959 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:39.533911    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-697fbc8d47-wlscv"
Jul 08 12:03:39.665235 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:39.665204    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-697fbc8d47-wlscv"]
Jul 08 12:03:39.668050 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:03:39.668020    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67af172b_92fa_43c3_a989_4d42968b3d0e.slice/crio-ebacdcc25b1162c7c5f66ffd9407ce613252ad80e963ebc7929f8560b1f7307e WatchSource:0}: Error finding container ebacdcc25b1162c7c5f66ffd9407ce613252ad80e963ebc7929f8560b1f7307e: Status 404 returned error can't find the container with id ebacdcc25b1162c7c5f66ffd9407ce613252ad80e963ebc7929f8560b1f7307e
Jul 08 12:03:40.126598 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:40.126509    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-697fbc8d47-wlscv" event={"ID":"67af172b-92fa-43c3-a989-4d42968b3d0e","Type":"ContainerStarted","Data":"6f696cfba355d2fe5903e8c890d22827e6a5452629e27662b6d56ec0ccd85f8d"}
Jul 08 12:03:40.126757 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:40.126606    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-697fbc8d47-wlscv" event={"ID":"67af172b-92fa-43c3-a989-4d42968b3d0e","Type":"ContainerStarted","Data":"ebacdcc25b1162c7c5f66ffd9407ce613252ad80e963ebc7929f8560b1f7307e"}
Jul 08 12:03:40.149486 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:40.149428    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-697fbc8d47-wlscv" podStartSLOduration=17.149413256 podStartE2EDuration="17.149413256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 12:03:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 12:03:40.148494741 +0000 UTC m=+109.878869712" watchObservedRunningTime="2026-07-08 12:03:40.149413256 +0000 UTC m=+109.879788227"
Jul 08 12:03:40.534660 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:40.534622    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-697fbc8d47-wlscv"
Jul 08 12:03:40.537062 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:40.537037    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-697fbc8d47-wlscv"
Jul 08 12:03:41.129319 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:41.129277    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-697fbc8d47-wlscv"
Jul 08 12:03:41.130574 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:41.130551    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-697fbc8d47-wlscv"
Jul 08 12:03:43.857191 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:43.857138    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/52314101-63ee-448f-963f-16b1dd3cdedb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8mcpn\" (UID: \"52314101-63ee-448f-963f-16b1dd3cdedb\") " pod="openshift-insights/insights-runtime-extractor-8mcpn"
Jul 08 12:03:43.857677 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:43.857217    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/52314101-63ee-448f-963f-16b1dd3cdedb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8mcpn\" (UID: \"52314101-63ee-448f-963f-16b1dd3cdedb\") " pod="openshift-insights/insights-runtime-extractor-8mcpn"
Jul 08 12:03:43.857824 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:43.857801    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/52314101-63ee-448f-963f-16b1dd3cdedb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8mcpn\" (UID: \"52314101-63ee-448f-963f-16b1dd3cdedb\") " pod="openshift-insights/insights-runtime-extractor-8mcpn"
Jul 08 12:03:43.859649 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:43.859630    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/52314101-63ee-448f-963f-16b1dd3cdedb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8mcpn\" (UID: \"52314101-63ee-448f-963f-16b1dd3cdedb\") " pod="openshift-insights/insights-runtime-extractor-8mcpn"
Jul 08 12:03:43.985965 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:43.985925    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8mcpn"
Jul 08 12:03:44.122859 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:44.122751    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8mcpn"]
Jul 08 12:03:44.125465 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:03:44.125438    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52314101_63ee_448f_963f_16b1dd3cdedb.slice/crio-7c511f73eeaffdb7d563e2ea0ef8514aedc4d2fadb6e3600d76b1b58719cdacb WatchSource:0}: Error finding container 7c511f73eeaffdb7d563e2ea0ef8514aedc4d2fadb6e3600d76b1b58719cdacb: Status 404 returned error can't find the container with id 7c511f73eeaffdb7d563e2ea0ef8514aedc4d2fadb6e3600d76b1b58719cdacb
Jul 08 12:03:44.140225 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:44.140194    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8mcpn" event={"ID":"52314101-63ee-448f-963f-16b1dd3cdedb","Type":"ContainerStarted","Data":"7c511f73eeaffdb7d563e2ea0ef8514aedc4d2fadb6e3600d76b1b58719cdacb"}
Jul 08 12:03:48.160305 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:48.160269    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8mcpn" event={"ID":"52314101-63ee-448f-963f-16b1dd3cdedb","Type":"ContainerStarted","Data":"9ebeed9c979e3bba379e2cd175ccd5124c7db5eb167542b7706a35f0b77665a8"}
Jul 08 12:03:49.166240 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:49.166198    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8mcpn" event={"ID":"52314101-63ee-448f-963f-16b1dd3cdedb","Type":"ContainerStarted","Data":"ca18e7f446147d389e711e62211af98baf2396d06834bedcdb232f9f948d56eb"}
Jul 08 12:03:51.174381 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:51.174338    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8mcpn" event={"ID":"52314101-63ee-448f-963f-16b1dd3cdedb","Type":"ContainerStarted","Data":"6a3f7a36920868560a5b5a89648c39f2f5b104964744ab8c19b0ea27e830a3a6"}
Jul 08 12:03:51.195124 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:51.195003    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-8mcpn" podStartSLOduration=16.480811768 podStartE2EDuration="23.194988407s" totalImagesPullingTime="6.714176639s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 12:03:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 12:03:51.194179428 +0000 UTC m=+120.924554410" watchObservedRunningTime="2026-07-08 12:03:51.194988407 +0000 UTC m=+120.925363377"
Jul 08 12:03:54.275676 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:54.275641    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-86bdbc87b6-lrh8q"]
Jul 08 12:03:54.277588 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:54.277572    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86bdbc87b6-lrh8q"
Jul 08 12:03:54.280612 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:54.280588    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 08 12:03:54.280922 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:54.280905    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-j8q6l\""
Jul 08 12:03:54.281677 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:54.281661    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 08 12:03:54.281727 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:54.281664    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 08 12:03:54.290520 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:54.290500    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 08 12:03:54.299388 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:54.299363    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-86bdbc87b6-lrh8q"]
Jul 08 12:03:54.445366 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:54.445330    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d0f99899-5a9a-443c-9ca3-3033d5699f2a-installation-pull-secrets\") pod \"image-registry-86bdbc87b6-lrh8q\" (UID: \"d0f99899-5a9a-443c-9ca3-3033d5699f2a\") " pod="openshift-image-registry/image-registry-86bdbc87b6-lrh8q"
Jul 08 12:03:54.445366 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:54.445369    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d0f99899-5a9a-443c-9ca3-3033d5699f2a-trusted-ca\") pod \"image-registry-86bdbc87b6-lrh8q\" (UID: \"d0f99899-5a9a-443c-9ca3-3033d5699f2a\") " pod="openshift-image-registry/image-registry-86bdbc87b6-lrh8q"
Jul 08 12:03:54.445583 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:54.445392    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d0f99899-5a9a-443c-9ca3-3033d5699f2a-registry-tls\") pod \"image-registry-86bdbc87b6-lrh8q\" (UID: \"d0f99899-5a9a-443c-9ca3-3033d5699f2a\") " pod="openshift-image-registry/image-registry-86bdbc87b6-lrh8q"
Jul 08 12:03:54.445583 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:54.445487    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d0f99899-5a9a-443c-9ca3-3033d5699f2a-ca-trust-extracted\") pod \"image-registry-86bdbc87b6-lrh8q\" (UID: \"d0f99899-5a9a-443c-9ca3-3033d5699f2a\") " pod="openshift-image-registry/image-registry-86bdbc87b6-lrh8q"
Jul 08 12:03:54.445583 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:54.445528    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d0f99899-5a9a-443c-9ca3-3033d5699f2a-registry-certificates\") pod \"image-registry-86bdbc87b6-lrh8q\" (UID: \"d0f99899-5a9a-443c-9ca3-3033d5699f2a\") " pod="openshift-image-registry/image-registry-86bdbc87b6-lrh8q"
Jul 08 12:03:54.445583 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:54.445571    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wx8xx\" (UniqueName: \"kubernetes.io/projected/d0f99899-5a9a-443c-9ca3-3033d5699f2a-kube-api-access-wx8xx\") pod \"image-registry-86bdbc87b6-lrh8q\" (UID: \"d0f99899-5a9a-443c-9ca3-3033d5699f2a\") " pod="openshift-image-registry/image-registry-86bdbc87b6-lrh8q"
Jul 08 12:03:54.445700 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:54.445593    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d0f99899-5a9a-443c-9ca3-3033d5699f2a-bound-sa-token\") pod \"image-registry-86bdbc87b6-lrh8q\" (UID: \"d0f99899-5a9a-443c-9ca3-3033d5699f2a\") " pod="openshift-image-registry/image-registry-86bdbc87b6-lrh8q"
Jul 08 12:03:54.445700 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:54.445642    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d0f99899-5a9a-443c-9ca3-3033d5699f2a-image-registry-private-configuration\") pod \"image-registry-86bdbc87b6-lrh8q\" (UID: \"d0f99899-5a9a-443c-9ca3-3033d5699f2a\") " pod="openshift-image-registry/image-registry-86bdbc87b6-lrh8q"
Jul 08 12:03:54.546899 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:54.546862    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d0f99899-5a9a-443c-9ca3-3033d5699f2a-registry-tls\") pod \"image-registry-86bdbc87b6-lrh8q\" (UID: \"d0f99899-5a9a-443c-9ca3-3033d5699f2a\") " pod="openshift-image-registry/image-registry-86bdbc87b6-lrh8q"
Jul 08 12:03:54.546899 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:54.546902    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d0f99899-5a9a-443c-9ca3-3033d5699f2a-ca-trust-extracted\") pod \"image-registry-86bdbc87b6-lrh8q\" (UID: \"d0f99899-5a9a-443c-9ca3-3033d5699f2a\") " pod="openshift-image-registry/image-registry-86bdbc87b6-lrh8q"
Jul 08 12:03:54.547136 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:54.546935    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d0f99899-5a9a-443c-9ca3-3033d5699f2a-registry-certificates\") pod \"image-registry-86bdbc87b6-lrh8q\" (UID: \"d0f99899-5a9a-443c-9ca3-3033d5699f2a\") " pod="openshift-image-registry/image-registry-86bdbc87b6-lrh8q"
Jul 08 12:03:54.547136 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:54.546958    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx8xx\" (UniqueName: \"kubernetes.io/projected/d0f99899-5a9a-443c-9ca3-3033d5699f2a-kube-api-access-wx8xx\") pod \"image-registry-86bdbc87b6-lrh8q\" (UID: \"d0f99899-5a9a-443c-9ca3-3033d5699f2a\") " pod="openshift-image-registry/image-registry-86bdbc87b6-lrh8q"
Jul 08 12:03:54.547136 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:54.546973    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d0f99899-5a9a-443c-9ca3-3033d5699f2a-bound-sa-token\") pod \"image-registry-86bdbc87b6-lrh8q\" (UID: \"d0f99899-5a9a-443c-9ca3-3033d5699f2a\") " pod="openshift-image-registry/image-registry-86bdbc87b6-lrh8q"
Jul 08 12:03:54.547136 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:54.546999    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d0f99899-5a9a-443c-9ca3-3033d5699f2a-image-registry-private-configuration\") pod \"image-registry-86bdbc87b6-lrh8q\" (UID: \"d0f99899-5a9a-443c-9ca3-3033d5699f2a\") " pod="openshift-image-registry/image-registry-86bdbc87b6-lrh8q"
Jul 08 12:03:54.547136 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:54.547064    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d0f99899-5a9a-443c-9ca3-3033d5699f2a-installation-pull-secrets\") pod \"image-registry-86bdbc87b6-lrh8q\" (UID: \"d0f99899-5a9a-443c-9ca3-3033d5699f2a\") " pod="openshift-image-registry/image-registry-86bdbc87b6-lrh8q"
Jul 08 12:03:54.547136 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:54.547112    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d0f99899-5a9a-443c-9ca3-3033d5699f2a-trusted-ca\") pod \"image-registry-86bdbc87b6-lrh8q\" (UID: \"d0f99899-5a9a-443c-9ca3-3033d5699f2a\") " pod="openshift-image-registry/image-registry-86bdbc87b6-lrh8q"
Jul 08 12:03:54.547601 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:54.547580    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d0f99899-5a9a-443c-9ca3-3033d5699f2a-ca-trust-extracted\") pod \"image-registry-86bdbc87b6-lrh8q\" (UID: \"d0f99899-5a9a-443c-9ca3-3033d5699f2a\") " pod="openshift-image-registry/image-registry-86bdbc87b6-lrh8q"
Jul 08 12:03:54.548097 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:54.548077    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d0f99899-5a9a-443c-9ca3-3033d5699f2a-registry-certificates\") pod \"image-registry-86bdbc87b6-lrh8q\" (UID: \"d0f99899-5a9a-443c-9ca3-3033d5699f2a\") " pod="openshift-image-registry/image-registry-86bdbc87b6-lrh8q"
Jul 08 12:03:54.548306 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:54.548282    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d0f99899-5a9a-443c-9ca3-3033d5699f2a-trusted-ca\") pod \"image-registry-86bdbc87b6-lrh8q\" (UID: \"d0f99899-5a9a-443c-9ca3-3033d5699f2a\") " pod="openshift-image-registry/image-registry-86bdbc87b6-lrh8q"
Jul 08 12:03:54.549697 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:54.549674    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d0f99899-5a9a-443c-9ca3-3033d5699f2a-installation-pull-secrets\") pod \"image-registry-86bdbc87b6-lrh8q\" (UID: \"d0f99899-5a9a-443c-9ca3-3033d5699f2a\") " pod="openshift-image-registry/image-registry-86bdbc87b6-lrh8q"
Jul 08 12:03:54.549803 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:54.549707    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d0f99899-5a9a-443c-9ca3-3033d5699f2a-image-registry-private-configuration\") pod \"image-registry-86bdbc87b6-lrh8q\" (UID: \"d0f99899-5a9a-443c-9ca3-3033d5699f2a\") " pod="openshift-image-registry/image-registry-86bdbc87b6-lrh8q"
Jul 08 12:03:54.549851 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:54.549828    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d0f99899-5a9a-443c-9ca3-3033d5699f2a-registry-tls\") pod \"image-registry-86bdbc87b6-lrh8q\" (UID: \"d0f99899-5a9a-443c-9ca3-3033d5699f2a\") " pod="openshift-image-registry/image-registry-86bdbc87b6-lrh8q"
Jul 08 12:03:54.556309 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:54.556286    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d0f99899-5a9a-443c-9ca3-3033d5699f2a-bound-sa-token\") pod \"image-registry-86bdbc87b6-lrh8q\" (UID: \"d0f99899-5a9a-443c-9ca3-3033d5699f2a\") " pod="openshift-image-registry/image-registry-86bdbc87b6-lrh8q"
Jul 08 12:03:54.556454 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:54.556436    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wx8xx\" (UniqueName: \"kubernetes.io/projected/d0f99899-5a9a-443c-9ca3-3033d5699f2a-kube-api-access-wx8xx\") pod \"image-registry-86bdbc87b6-lrh8q\" (UID: \"d0f99899-5a9a-443c-9ca3-3033d5699f2a\") " pod="openshift-image-registry/image-registry-86bdbc87b6-lrh8q"
Jul 08 12:03:54.587101 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:54.587068    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86bdbc87b6-lrh8q"
Jul 08 12:03:54.719981 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:54.719901    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-86bdbc87b6-lrh8q"]
Jul 08 12:03:54.722980 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:03:54.722949    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0f99899_5a9a_443c_9ca3_3033d5699f2a.slice/crio-af591e9b659a95b16b3c40298ac02a2cf1602970de663d0c667664fc42bd0764 WatchSource:0}: Error finding container af591e9b659a95b16b3c40298ac02a2cf1602970de663d0c667664fc42bd0764: Status 404 returned error can't find the container with id af591e9b659a95b16b3c40298ac02a2cf1602970de663d0c667664fc42bd0764
Jul 08 12:03:55.188023 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:55.187990    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-86bdbc87b6-lrh8q" event={"ID":"d0f99899-5a9a-443c-9ca3-3033d5699f2a","Type":"ContainerStarted","Data":"28dedef5a5d1d371e2a352f0b672c29b0f158de49c4be157f42cc71b42078bee"}
Jul 08 12:03:55.188183 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:55.188045    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-86bdbc87b6-lrh8q" event={"ID":"d0f99899-5a9a-443c-9ca3-3033d5699f2a","Type":"ContainerStarted","Data":"af591e9b659a95b16b3c40298ac02a2cf1602970de663d0c667664fc42bd0764"}
Jul 08 12:03:55.188183 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:55.188081    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-86bdbc87b6-lrh8q"
Jul 08 12:03:55.213609 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:55.213549    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-86bdbc87b6-lrh8q" podStartSLOduration=1.213531317 podStartE2EDuration="1.213531317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 12:03:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 12:03:55.211711421 +0000 UTC m=+124.942086393" watchObservedRunningTime="2026-07-08 12:03:55.213531317 +0000 UTC m=+124.943906288"
Jul 08 12:03:59.591468 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:59.591432    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a64d26d8-d849-4f64-b43d-3a9133970f52-metrics-certs\") pod \"network-metrics-daemon-m5628\" (UID: \"a64d26d8-d849-4f64-b43d-3a9133970f52\") " pod="openshift-multus/network-metrics-daemon-m5628"
Jul 08 12:03:59.593872 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:59.593850    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a64d26d8-d849-4f64-b43d-3a9133970f52-metrics-certs\") pod \"network-metrics-daemon-m5628\" (UID: \"a64d26d8-d849-4f64-b43d-3a9133970f52\") " pod="openshift-multus/network-metrics-daemon-m5628"
Jul 08 12:03:59.633445 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:59.633415    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rdkjt\""
Jul 08 12:03:59.641330 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:59.641302    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m5628"
Jul 08 12:03:59.768175 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:03:59.768032    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-m5628"]
Jul 08 12:03:59.770594 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:03:59.770566    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda64d26d8_d849_4f64_b43d_3a9133970f52.slice/crio-163c8dae2e99db248ed7f8585562701cdc7c50c135d10eef1b422785d565a335 WatchSource:0}: Error finding container 163c8dae2e99db248ed7f8585562701cdc7c50c135d10eef1b422785d565a335: Status 404 returned error can't find the container with id 163c8dae2e99db248ed7f8585562701cdc7c50c135d10eef1b422785d565a335
Jul 08 12:04:00.202937 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:00.202897    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-m5628" event={"ID":"a64d26d8-d849-4f64-b43d-3a9133970f52","Type":"ContainerStarted","Data":"163c8dae2e99db248ed7f8585562701cdc7c50c135d10eef1b422785d565a335"}
Jul 08 12:04:02.211032 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:02.210991    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-m5628" event={"ID":"a64d26d8-d849-4f64-b43d-3a9133970f52","Type":"ContainerStarted","Data":"72bf7edb73ca6adc8afe2ae9721d8cc72228c4845ebc31b94292bc9c1569b514"}
Jul 08 12:04:02.211412 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:02.211055    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-m5628" event={"ID":"a64d26d8-d849-4f64-b43d-3a9133970f52","Type":"ContainerStarted","Data":"92681a744759f65cf058f081e686c2b33637bcfe538e4977b5217ee561f3bd8c"}
Jul 08 12:04:02.229787 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:02.229719    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-m5628" podStartSLOduration=129.607043139 podStartE2EDuration="2m11.229703657s" totalImagesPullingTime="1.622660518s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 12:01:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 12:04:02.228735887 +0000 UTC m=+131.959110857" watchObservedRunningTime="2026-07-08 12:04:02.229703657 +0000 UTC m=+131.960078623"
Jul 08 12:04:08.071374 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:08.071336    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-hf42b"]
Jul 08 12:04:08.073651 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:08.073631    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hf42b"
Jul 08 12:04:08.076028 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:08.076006    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-pnlbh\""
Jul 08 12:04:08.076148 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:08.076041    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 08 12:04:08.076148 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:08.076134    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 08 12:04:08.077070 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:08.077051    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 08 12:04:08.077162 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:08.077071    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 08 12:04:08.077162 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:08.077097    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 08 12:04:08.077162 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:08.077109    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 08 12:04:08.260471 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:08.260431    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/90f23b67-264d-4ec0-bac5-d87fcab5c19d-node-exporter-wtmp\") pod \"node-exporter-hf42b\" (UID: \"90f23b67-264d-4ec0-bac5-d87fcab5c19d\") " pod="openshift-monitoring/node-exporter-hf42b"
Jul 08 12:04:08.260665 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:08.260496    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/90f23b67-264d-4ec0-bac5-d87fcab5c19d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hf42b\" (UID: \"90f23b67-264d-4ec0-bac5-d87fcab5c19d\") " pod="openshift-monitoring/node-exporter-hf42b"
Jul 08 12:04:08.260665 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:08.260526    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk867\" (UniqueName: \"kubernetes.io/projected/90f23b67-264d-4ec0-bac5-d87fcab5c19d-kube-api-access-fk867\") pod \"node-exporter-hf42b\" (UID: \"90f23b67-264d-4ec0-bac5-d87fcab5c19d\") " pod="openshift-monitoring/node-exporter-hf42b"
Jul 08 12:04:08.260665 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:08.260550    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/90f23b67-264d-4ec0-bac5-d87fcab5c19d-root\") pod \"node-exporter-hf42b\" (UID: \"90f23b67-264d-4ec0-bac5-d87fcab5c19d\") " pod="openshift-monitoring/node-exporter-hf42b"
Jul 08 12:04:08.260665 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:08.260569    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/90f23b67-264d-4ec0-bac5-d87fcab5c19d-node-exporter-tls\") pod \"node-exporter-hf42b\" (UID: \"90f23b67-264d-4ec0-bac5-d87fcab5c19d\") " pod="openshift-monitoring/node-exporter-hf42b"
Jul 08 12:04:08.260665 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:08.260648    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/90f23b67-264d-4ec0-bac5-d87fcab5c19d-node-exporter-textfile\") pod \"node-exporter-hf42b\" (UID: \"90f23b67-264d-4ec0-bac5-d87fcab5c19d\") " pod="openshift-monitoring/node-exporter-hf42b"
Jul 08 12:04:08.260901 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:08.260693    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/90f23b67-264d-4ec0-bac5-d87fcab5c19d-sys\") pod \"node-exporter-hf42b\" (UID: \"90f23b67-264d-4ec0-bac5-d87fcab5c19d\") " pod="openshift-monitoring/node-exporter-hf42b"
Jul 08 12:04:08.260901 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:08.260712    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/90f23b67-264d-4ec0-bac5-d87fcab5c19d-metrics-client-ca\") pod \"node-exporter-hf42b\" (UID: \"90f23b67-264d-4ec0-bac5-d87fcab5c19d\") " pod="openshift-monitoring/node-exporter-hf42b"
Jul 08 12:04:08.260901 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:08.260741    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/90f23b67-264d-4ec0-bac5-d87fcab5c19d-node-exporter-accelerators-collector-config\") pod \"node-exporter-hf42b\" (UID: \"90f23b67-264d-4ec0-bac5-d87fcab5c19d\") " pod="openshift-monitoring/node-exporter-hf42b"
Jul 08 12:04:08.361618 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:08.361528    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/90f23b67-264d-4ec0-bac5-d87fcab5c19d-sys\") pod \"node-exporter-hf42b\" (UID: \"90f23b67-264d-4ec0-bac5-d87fcab5c19d\") " pod="openshift-monitoring/node-exporter-hf42b"
Jul 08 12:04:08.361618 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:08.361564    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/90f23b67-264d-4ec0-bac5-d87fcab5c19d-metrics-client-ca\") pod \"node-exporter-hf42b\" (UID: \"90f23b67-264d-4ec0-bac5-d87fcab5c19d\") " pod="openshift-monitoring/node-exporter-hf42b"
Jul 08 12:04:08.361618 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:08.361584    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/90f23b67-264d-4ec0-bac5-d87fcab5c19d-node-exporter-accelerators-collector-config\") pod \"node-exporter-hf42b\" (UID: \"90f23b67-264d-4ec0-bac5-d87fcab5c19d\") " pod="openshift-monitoring/node-exporter-hf42b"
Jul 08 12:04:08.361866 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:08.361647    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/90f23b67-264d-4ec0-bac5-d87fcab5c19d-sys\") pod \"node-exporter-hf42b\" (UID: \"90f23b67-264d-4ec0-bac5-d87fcab5c19d\") " pod="openshift-monitoring/node-exporter-hf42b"
Jul 08 12:04:08.361866 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:08.361713    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/90f23b67-264d-4ec0-bac5-d87fcab5c19d-node-exporter-wtmp\") pod \"node-exporter-hf42b\" (UID: \"90f23b67-264d-4ec0-bac5-d87fcab5c19d\") " pod="openshift-monitoring/node-exporter-hf42b"
Jul 08 12:04:08.361866 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:08.361758    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/90f23b67-264d-4ec0-bac5-d87fcab5c19d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hf42b\" (UID: \"90f23b67-264d-4ec0-bac5-d87fcab5c19d\") " pod="openshift-monitoring/node-exporter-hf42b"
Jul 08 12:04:08.361866 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:08.361820    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk867\" (UniqueName: \"kubernetes.io/projected/90f23b67-264d-4ec0-bac5-d87fcab5c19d-kube-api-access-fk867\") pod \"node-exporter-hf42b\" (UID: \"90f23b67-264d-4ec0-bac5-d87fcab5c19d\") " pod="openshift-monitoring/node-exporter-hf42b"
Jul 08 12:04:08.361866 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:08.361839    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/90f23b67-264d-4ec0-bac5-d87fcab5c19d-root\") pod \"node-exporter-hf42b\" (UID: \"90f23b67-264d-4ec0-bac5-d87fcab5c19d\") " pod="openshift-monitoring/node-exporter-hf42b"
Jul 08 12:04:08.361866 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:08.361860    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/90f23b67-264d-4ec0-bac5-d87fcab5c19d-node-exporter-tls\") pod \"node-exporter-hf42b\" (UID: \"90f23b67-264d-4ec0-bac5-d87fcab5c19d\") " pod="openshift-monitoring/node-exporter-hf42b"
Jul 08 12:04:08.362143 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:08.361896    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/90f23b67-264d-4ec0-bac5-d87fcab5c19d-node-exporter-wtmp\") pod \"node-exporter-hf42b\" (UID: \"90f23b67-264d-4ec0-bac5-d87fcab5c19d\") " pod="openshift-monitoring/node-exporter-hf42b"
Jul 08 12:04:08.362143 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:08.361993    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/90f23b67-264d-4ec0-bac5-d87fcab5c19d-node-exporter-textfile\") pod \"node-exporter-hf42b\" (UID: \"90f23b67-264d-4ec0-bac5-d87fcab5c19d\") " pod="openshift-monitoring/node-exporter-hf42b"
Jul 08 12:04:08.362246 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:08.362153    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/90f23b67-264d-4ec0-bac5-d87fcab5c19d-node-exporter-accelerators-collector-config\") pod \"node-exporter-hf42b\" (UID: \"90f23b67-264d-4ec0-bac5-d87fcab5c19d\") " pod="openshift-monitoring/node-exporter-hf42b"
Jul 08 12:04:08.362246 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:08.362238    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/90f23b67-264d-4ec0-bac5-d87fcab5c19d-metrics-client-ca\") pod \"node-exporter-hf42b\" (UID: \"90f23b67-264d-4ec0-bac5-d87fcab5c19d\") " pod="openshift-monitoring/node-exporter-hf42b"
Jul 08 12:04:08.362350 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:08.362329    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/90f23b67-264d-4ec0-bac5-d87fcab5c19d-node-exporter-textfile\") pod \"node-exporter-hf42b\" (UID: \"90f23b67-264d-4ec0-bac5-d87fcab5c19d\") " pod="openshift-monitoring/node-exporter-hf42b"
Jul 08 12:04:08.362402 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:08.362381    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/90f23b67-264d-4ec0-bac5-d87fcab5c19d-root\") pod \"node-exporter-hf42b\" (UID: \"90f23b67-264d-4ec0-bac5-d87fcab5c19d\") " pod="openshift-monitoring/node-exporter-hf42b"
Jul 08 12:04:08.364364 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:08.364344    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/90f23b67-264d-4ec0-bac5-d87fcab5c19d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hf42b\" (UID: \"90f23b67-264d-4ec0-bac5-d87fcab5c19d\") " pod="openshift-monitoring/node-exporter-hf42b"
Jul 08 12:04:08.364419 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:08.364398    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/90f23b67-264d-4ec0-bac5-d87fcab5c19d-node-exporter-tls\") pod \"node-exporter-hf42b\" (UID: \"90f23b67-264d-4ec0-bac5-d87fcab5c19d\") " pod="openshift-monitoring/node-exporter-hf42b"
Jul 08 12:04:08.373711 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:08.373688    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk867\" (UniqueName: \"kubernetes.io/projected/90f23b67-264d-4ec0-bac5-d87fcab5c19d-kube-api-access-fk867\") pod \"node-exporter-hf42b\" (UID: \"90f23b67-264d-4ec0-bac5-d87fcab5c19d\") " pod="openshift-monitoring/node-exporter-hf42b"
Jul 08 12:04:08.382697 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:08.382673    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hf42b"
Jul 08 12:04:08.395561 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:04:08.395530    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90f23b67_264d_4ec0_bac5_d87fcab5c19d.slice/crio-5fb00e7e3d9d23d15ee7879fbdaa3f62950af07d04a8221682092bef37f8834a WatchSource:0}: Error finding container 5fb00e7e3d9d23d15ee7879fbdaa3f62950af07d04a8221682092bef37f8834a: Status 404 returned error can't find the container with id 5fb00e7e3d9d23d15ee7879fbdaa3f62950af07d04a8221682092bef37f8834a
Jul 08 12:04:09.235431 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:09.235387    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hf42b" event={"ID":"90f23b67-264d-4ec0-bac5-d87fcab5c19d","Type":"ContainerStarted","Data":"5fb00e7e3d9d23d15ee7879fbdaa3f62950af07d04a8221682092bef37f8834a"}
Jul 08 12:04:09.368516 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:09.368481    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 08 12:04:09.371795 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:09.371759    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:04:09.374528 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:09.374500    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 08 12:04:09.374673 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:09.374562    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 08 12:04:09.374673 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:09.374666    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-tfj4m\""
Jul 08 12:04:09.374804 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:09.374723    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 08 12:04:09.374804 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:09.374748    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 08 12:04:09.375007 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:09.374991    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 08 12:04:09.375089 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:09.375013    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 08 12:04:09.375089 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:09.375053    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 08 12:04:09.375256 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:09.375239    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 08 12:04:09.375316 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:09.375299    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 08 12:04:09.395397 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:09.395363    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 08 12:04:09.470342 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:09.470303    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5473f735-ebad-4d77-ad8a-24e50f26d3b6-config-volume\") pod \"alertmanager-main-0\" (UID: \"5473f735-ebad-4d77-ad8a-24e50f26d3b6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:04:09.470342 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:09.470339    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5473f735-ebad-4d77-ad8a-24e50f26d3b6-config-out\") pod \"alertmanager-main-0\" (UID: \"5473f735-ebad-4d77-ad8a-24e50f26d3b6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:04:09.470587 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:09.470367    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5473f735-ebad-4d77-ad8a-24e50f26d3b6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5473f735-ebad-4d77-ad8a-24e50f26d3b6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:04:09.470587 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:09.470441    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5473f735-ebad-4d77-ad8a-24e50f26d3b6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5473f735-ebad-4d77-ad8a-24e50f26d3b6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:04:09.470587 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:09.470474    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5473f735-ebad-4d77-ad8a-24e50f26d3b6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5473f735-ebad-4d77-ad8a-24e50f26d3b6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:04:09.470587 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:09.470500    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5473f735-ebad-4d77-ad8a-24e50f26d3b6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5473f735-ebad-4d77-ad8a-24e50f26d3b6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:04:09.470587 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:09.470536    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq6lg\" (UniqueName: \"kubernetes.io/projected/5473f735-ebad-4d77-ad8a-24e50f26d3b6-kube-api-access-wq6lg\") pod \"alertmanager-main-0\" (UID: \"5473f735-ebad-4d77-ad8a-24e50f26d3b6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:04:09.470840 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:09.470609    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5473f735-ebad-4d77-ad8a-24e50f26d3b6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5473f735-ebad-4d77-ad8a-24e50f26d3b6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:04:09.470840 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:09.470650    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5473f735-ebad-4d77-ad8a-24e50f26d3b6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5473f735-ebad-4d77-ad8a-24e50f26d3b6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:04:09.470840 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:09.470666    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5473f735-ebad-4d77-ad8a-24e50f26d3b6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5473f735-ebad-4d77-ad8a-24e50f26d3b6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:04:09.470840 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:09.470706    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5473f735-ebad-4d77-ad8a-24e50f26d3b6-web-config\") pod \"alertmanager-main-0\" (UID: \"5473f735-ebad-4d77-ad8a-24e50f26d3b6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:04:09.470840 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:09.470753    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5473f735-ebad-4d77-ad8a-24e50f26d3b6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5473f735-ebad-4d77-ad8a-24e50f26d3b6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:04:09.470840 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:09.470819    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5473f735-ebad-4d77-ad8a-24e50f26d3b6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5473f735-ebad-4d77-ad8a-24e50f26d3b6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:04:09.571916 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:09.571883    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5473f735-ebad-4d77-ad8a-24e50f26d3b6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5473f735-ebad-4d77-ad8a-24e50f26d3b6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:04:09.572081 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:09.571941    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5473f735-ebad-4d77-ad8a-24e50f26d3b6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5473f735-ebad-4d77-ad8a-24e50f26d3b6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:04:09.572151 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:09.572087    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5473f735-ebad-4d77-ad8a-24e50f26d3b6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5473f735-ebad-4d77-ad8a-24e50f26d3b6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:04:09.572206 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:09.572184    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5473f735-ebad-4d77-ad8a-24e50f26d3b6-web-config\") pod \"alertmanager-main-0\" (UID: \"5473f735-ebad-4d77-ad8a-24e50f26d3b6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:04:09.572257 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:09.572212    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5473f735-ebad-4d77-ad8a-24e50f26d3b6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5473f735-ebad-4d77-ad8a-24e50f26d3b6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:04:09.572257 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:09.572241    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5473f735-ebad-4d77-ad8a-24e50f26d3b6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5473f735-ebad-4d77-ad8a-24e50f26d3b6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:04:09.572452 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:09.572409    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5473f735-ebad-4d77-ad8a-24e50f26d3b6-config-volume\") pod \"alertmanager-main-0\" (UID: \"5473f735-ebad-4d77-ad8a-24e50f26d3b6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:04:09.572527 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:09.572472    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5473f735-ebad-4d77-ad8a-24e50f26d3b6-config-out\") pod \"alertmanager-main-0\" (UID: \"5473f735-ebad-4d77-ad8a-24e50f26d3b6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:04:09.572527 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:09.572505    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5473f735-ebad-4d77-ad8a-24e50f26d3b6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5473f735-ebad-4d77-ad8a-24e50f26d3b6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:04:09.572625 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:09.572553    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5473f735-ebad-4d77-ad8a-24e50f26d3b6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5473f735-ebad-4d77-ad8a-24e50f26d3b6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:04:09.572625 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:09.572587    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5473f735-ebad-4d77-ad8a-24e50f26d3b6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5473f735-ebad-4d77-ad8a-24e50f26d3b6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:04:09.572725 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:09.572621    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5473f735-ebad-4d77-ad8a-24e50f26d3b6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5473f735-ebad-4d77-ad8a-24e50f26d3b6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:04:09.572725 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:09.572643    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq6lg\" (UniqueName: \"kubernetes.io/projected/5473f735-ebad-4d77-ad8a-24e50f26d3b6-kube-api-access-wq6lg\") pod \"alertmanager-main-0\" (UID: \"5473f735-ebad-4d77-ad8a-24e50f26d3b6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:04:09.572725 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:09.572706    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5473f735-ebad-4d77-ad8a-24e50f26d3b6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5473f735-ebad-4d77-ad8a-24e50f26d3b6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:04:09.573052 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:09.572990    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5473f735-ebad-4d77-ad8a-24e50f26d3b6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5473f735-ebad-4d77-ad8a-24e50f26d3b6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:04:09.575557 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:09.575118    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5473f735-ebad-4d77-ad8a-24e50f26d3b6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5473f735-ebad-4d77-ad8a-24e50f26d3b6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:04:09.576131 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:09.576087    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5473f735-ebad-4d77-ad8a-24e50f26d3b6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5473f735-ebad-4d77-ad8a-24e50f26d3b6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:04:09.576641 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:09.576588    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5473f735-ebad-4d77-ad8a-24e50f26d3b6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5473f735-ebad-4d77-ad8a-24e50f26d3b6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:04:09.577154 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:09.576873    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5473f735-ebad-4d77-ad8a-24e50f26d3b6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5473f735-ebad-4d77-ad8a-24e50f26d3b6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:04:09.577154 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:09.577049    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5473f735-ebad-4d77-ad8a-24e50f26d3b6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5473f735-ebad-4d77-ad8a-24e50f26d3b6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:04:09.577154 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:09.577124    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5473f735-ebad-4d77-ad8a-24e50f26d3b6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5473f735-ebad-4d77-ad8a-24e50f26d3b6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:04:09.577437 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:09.577420    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5473f735-ebad-4d77-ad8a-24e50f26d3b6-web-config\") pod \"alertmanager-main-0\" (UID: \"5473f735-ebad-4d77-ad8a-24e50f26d3b6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:04:09.577507 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:09.577458    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5473f735-ebad-4d77-ad8a-24e50f26d3b6-config-out\") pod \"alertmanager-main-0\" (UID: \"5473f735-ebad-4d77-ad8a-24e50f26d3b6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:04:09.577544 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:09.577516    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5473f735-ebad-4d77-ad8a-24e50f26d3b6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5473f735-ebad-4d77-ad8a-24e50f26d3b6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:04:09.577910 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:09.577894    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5473f735-ebad-4d77-ad8a-24e50f26d3b6-config-volume\") pod \"alertmanager-main-0\" (UID: \"5473f735-ebad-4d77-ad8a-24e50f26d3b6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:04:09.582540 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:09.582508    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq6lg\" (UniqueName: \"kubernetes.io/projected/5473f735-ebad-4d77-ad8a-24e50f26d3b6-kube-api-access-wq6lg\") pod \"alertmanager-main-0\" (UID: \"5473f735-ebad-4d77-ad8a-24e50f26d3b6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:04:09.684023 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:09.683991    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:04:09.829916 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:09.829880    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 08 12:04:09.832474 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:04:09.832441    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5473f735_ebad_4d77_ad8a_24e50f26d3b6.slice/crio-212e91714dee3f38249a5207e07e6dd2c7b5bcba10977360e66b5c10261b440c WatchSource:0}: Error finding container 212e91714dee3f38249a5207e07e6dd2c7b5bcba10977360e66b5c10261b440c: Status 404 returned error can't find the container with id 212e91714dee3f38249a5207e07e6dd2c7b5bcba10977360e66b5c10261b440c
Jul 08 12:04:10.239527 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:10.239493    2579 generic.go:356] "Generic (PLEG): container finished" podID="90f23b67-264d-4ec0-bac5-d87fcab5c19d" containerID="1f0e1a0adc20f0ed4774211a684827cef774a1e47d6690743b180534c638f121" exitCode=0
Jul 08 12:04:10.239961 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:10.239568    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hf42b" event={"ID":"90f23b67-264d-4ec0-bac5-d87fcab5c19d","Type":"ContainerDied","Data":"1f0e1a0adc20f0ed4774211a684827cef774a1e47d6690743b180534c638f121"}
Jul 08 12:04:10.240841 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:10.240816    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5473f735-ebad-4d77-ad8a-24e50f26d3b6","Type":"ContainerStarted","Data":"212e91714dee3f38249a5207e07e6dd2c7b5bcba10977360e66b5c10261b440c"}
Jul 08 12:04:11.246653 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:11.246552    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hf42b" event={"ID":"90f23b67-264d-4ec0-bac5-d87fcab5c19d","Type":"ContainerStarted","Data":"bf186c48f2b3ee820f207a05070c300e519e3948e28dc815a1eb48d516118be8"}
Jul 08 12:04:11.246653 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:11.246643    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hf42b" event={"ID":"90f23b67-264d-4ec0-bac5-d87fcab5c19d","Type":"ContainerStarted","Data":"21fa34a651ab285f6562a392b6c840bedcf9cbc5df2893668ffe122cca9b62ef"}
Jul 08 12:04:11.248148 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:11.248127    2579 generic.go:356] "Generic (PLEG): container finished" podID="5473f735-ebad-4d77-ad8a-24e50f26d3b6" containerID="4cc7418027887b5104fa0f0b7b5f2468ceaa12ebca4035741b6dd0e1edf575bb" exitCode=0
Jul 08 12:04:11.248211 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:11.248198    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5473f735-ebad-4d77-ad8a-24e50f26d3b6","Type":"ContainerDied","Data":"4cc7418027887b5104fa0f0b7b5f2468ceaa12ebca4035741b6dd0e1edf575bb"}
Jul 08 12:04:11.273711 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:11.273658    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-hf42b" podStartSLOduration=2.090164767 podStartE2EDuration="3.273642366s" totalImagesPullingTime="1.111657359s" totalInitContainerRuntime="71.82024ms" isStatefulPod=true podCreationTimestamp="2026-07-08 12:04:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 12:04:11.27237329 +0000 UTC m=+141.002748261" watchObservedRunningTime="2026-07-08 12:04:11.273642366 +0000 UTC m=+141.004017335"
Jul 08 12:04:13.258101 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:13.258067    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5473f735-ebad-4d77-ad8a-24e50f26d3b6","Type":"ContainerStarted","Data":"fa588d73449b98c73ad8b443a9a059a865ab7b79699f92fd1bb145830563963d"}
Jul 08 12:04:13.258378 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:13.258154    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5473f735-ebad-4d77-ad8a-24e50f26d3b6","Type":"ContainerStarted","Data":"114ac339492f55ef6e58eb09b8f02f2cf8d40fb51918a20399a2cc637d1f4bee"}
Jul 08 12:04:13.258378 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:13.258190    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5473f735-ebad-4d77-ad8a-24e50f26d3b6","Type":"ContainerStarted","Data":"286ad7bd142034cd3c11ff1f4d8a071f1ce40a5b412e68b79c11ec1b22e362eb"}
Jul 08 12:04:14.265078 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:14.265037    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5473f735-ebad-4d77-ad8a-24e50f26d3b6","Type":"ContainerStarted","Data":"5661028e227d5fccf7d06f1b763998343d48777c487b5b0b36a85c36832ddc2d"}
Jul 08 12:04:14.265533 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:14.265121    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5473f735-ebad-4d77-ad8a-24e50f26d3b6","Type":"ContainerStarted","Data":"d9c571b677bc48c60ef66a3b9cd31e356018c40f1b1a58b1d281bcad7e758270"}
Jul 08 12:04:14.265533 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:14.265158    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5473f735-ebad-4d77-ad8a-24e50f26d3b6","Type":"ContainerStarted","Data":"c07a6ea9140a766a0f6e2eba798e70ef016b205cfee8f9e7f024d4bff77cb3c1"}
Jul 08 12:04:14.297270 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:14.297209    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=1.558527451 podStartE2EDuration="5.297188966s" totalImagesPullingTime="3.718684151s" totalInitContainerRuntime="19.977364ms" isStatefulPod=false podCreationTimestamp="2026-07-08 12:04:09 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 12:04:14.296193419 +0000 UTC m=+144.026568391" watchObservedRunningTime="2026-07-08 12:04:14.297188966 +0000 UTC m=+144.027563936"
Jul 08 12:04:16.194852 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:16.194821    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-86bdbc87b6-lrh8q"
Jul 08 12:04:24.750171 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:24.750136    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-678df5d84b-v29jh"]
Jul 08 12:04:24.779081 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:24.779046    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-678df5d84b-v29jh"]
Jul 08 12:04:24.779222 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:24.779141    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-678df5d84b-v29jh"
Jul 08 12:04:24.784198 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:24.784131    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 08 12:04:24.784198 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:24.784168    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 08 12:04:24.785243 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:24.785219    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 08 12:04:24.786298 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:24.786249    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 08 12:04:24.786298 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:24.786255    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 08 12:04:24.786547 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:24.786499    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-km99f\""
Jul 08 12:04:24.786633 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:24.786599    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 08 12:04:24.786830 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:24.786809    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 08 12:04:24.789065 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:24.789046    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 08 12:04:24.893241 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:24.893211    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe815282-50d9-4ebd-a970-ca9c70a92dd9-trusted-ca-bundle\") pod \"console-678df5d84b-v29jh\" (UID: \"fe815282-50d9-4ebd-a970-ca9c70a92dd9\") " pod="openshift-console/console-678df5d84b-v29jh"
Jul 08 12:04:24.893241 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:24.893252    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fe815282-50d9-4ebd-a970-ca9c70a92dd9-oauth-serving-cert\") pod \"console-678df5d84b-v29jh\" (UID: \"fe815282-50d9-4ebd-a970-ca9c70a92dd9\") " pod="openshift-console/console-678df5d84b-v29jh"
Jul 08 12:04:24.893458 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:24.893275    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fe815282-50d9-4ebd-a970-ca9c70a92dd9-console-serving-cert\") pod \"console-678df5d84b-v29jh\" (UID: \"fe815282-50d9-4ebd-a970-ca9c70a92dd9\") " pod="openshift-console/console-678df5d84b-v29jh"
Jul 08 12:04:24.893458 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:24.893351    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fe815282-50d9-4ebd-a970-ca9c70a92dd9-console-oauth-config\") pod \"console-678df5d84b-v29jh\" (UID: \"fe815282-50d9-4ebd-a970-ca9c70a92dd9\") " pod="openshift-console/console-678df5d84b-v29jh"
Jul 08 12:04:24.893458 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:24.893391    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fe815282-50d9-4ebd-a970-ca9c70a92dd9-console-config\") pod \"console-678df5d84b-v29jh\" (UID: \"fe815282-50d9-4ebd-a970-ca9c70a92dd9\") " pod="openshift-console/console-678df5d84b-v29jh"
Jul 08 12:04:24.893458 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:24.893410    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fe815282-50d9-4ebd-a970-ca9c70a92dd9-service-ca\") pod \"console-678df5d84b-v29jh\" (UID: \"fe815282-50d9-4ebd-a970-ca9c70a92dd9\") " pod="openshift-console/console-678df5d84b-v29jh"
Jul 08 12:04:24.893458 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:24.893446    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgbnw\" (UniqueName: \"kubernetes.io/projected/fe815282-50d9-4ebd-a970-ca9c70a92dd9-kube-api-access-vgbnw\") pod \"console-678df5d84b-v29jh\" (UID: \"fe815282-50d9-4ebd-a970-ca9c70a92dd9\") " pod="openshift-console/console-678df5d84b-v29jh"
Jul 08 12:04:24.994524 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:24.994486    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe815282-50d9-4ebd-a970-ca9c70a92dd9-trusted-ca-bundle\") pod \"console-678df5d84b-v29jh\" (UID: \"fe815282-50d9-4ebd-a970-ca9c70a92dd9\") " pod="openshift-console/console-678df5d84b-v29jh"
Jul 08 12:04:24.994640 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:24.994535    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fe815282-50d9-4ebd-a970-ca9c70a92dd9-oauth-serving-cert\") pod \"console-678df5d84b-v29jh\" (UID: \"fe815282-50d9-4ebd-a970-ca9c70a92dd9\") " pod="openshift-console/console-678df5d84b-v29jh"
Jul 08 12:04:24.994640 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:24.994553    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fe815282-50d9-4ebd-a970-ca9c70a92dd9-console-serving-cert\") pod \"console-678df5d84b-v29jh\" (UID: \"fe815282-50d9-4ebd-a970-ca9c70a92dd9\") " pod="openshift-console/console-678df5d84b-v29jh"
Jul 08 12:04:24.994711 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:24.994631    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fe815282-50d9-4ebd-a970-ca9c70a92dd9-console-oauth-config\") pod \"console-678df5d84b-v29jh\" (UID: \"fe815282-50d9-4ebd-a970-ca9c70a92dd9\") " pod="openshift-console/console-678df5d84b-v29jh"
Jul 08 12:04:24.994711 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:24.994685    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fe815282-50d9-4ebd-a970-ca9c70a92dd9-console-config\") pod \"console-678df5d84b-v29jh\" (UID: \"fe815282-50d9-4ebd-a970-ca9c70a92dd9\") " pod="openshift-console/console-678df5d84b-v29jh"
Jul 08 12:04:24.994815 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:24.994709    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fe815282-50d9-4ebd-a970-ca9c70a92dd9-service-ca\") pod \"console-678df5d84b-v29jh\" (UID: \"fe815282-50d9-4ebd-a970-ca9c70a92dd9\") " pod="openshift-console/console-678df5d84b-v29jh"
Jul 08 12:04:24.994815 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:24.994762    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgbnw\" (UniqueName: \"kubernetes.io/projected/fe815282-50d9-4ebd-a970-ca9c70a92dd9-kube-api-access-vgbnw\") pod \"console-678df5d84b-v29jh\" (UID: \"fe815282-50d9-4ebd-a970-ca9c70a92dd9\") " pod="openshift-console/console-678df5d84b-v29jh"
Jul 08 12:04:24.995374 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:24.995352    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fe815282-50d9-4ebd-a970-ca9c70a92dd9-oauth-serving-cert\") pod \"console-678df5d84b-v29jh\" (UID: \"fe815282-50d9-4ebd-a970-ca9c70a92dd9\") " pod="openshift-console/console-678df5d84b-v29jh"
Jul 08 12:04:24.995529 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:24.995507    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fe815282-50d9-4ebd-a970-ca9c70a92dd9-console-config\") pod \"console-678df5d84b-v29jh\" (UID: \"fe815282-50d9-4ebd-a970-ca9c70a92dd9\") " pod="openshift-console/console-678df5d84b-v29jh"
Jul 08 12:04:24.995603 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:24.995507    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fe815282-50d9-4ebd-a970-ca9c70a92dd9-service-ca\") pod \"console-678df5d84b-v29jh\" (UID: \"fe815282-50d9-4ebd-a970-ca9c70a92dd9\") " pod="openshift-console/console-678df5d84b-v29jh"
Jul 08 12:04:24.995676 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:24.995656    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe815282-50d9-4ebd-a970-ca9c70a92dd9-trusted-ca-bundle\") pod \"console-678df5d84b-v29jh\" (UID: \"fe815282-50d9-4ebd-a970-ca9c70a92dd9\") " pod="openshift-console/console-678df5d84b-v29jh"
Jul 08 12:04:24.997687 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:24.997667    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fe815282-50d9-4ebd-a970-ca9c70a92dd9-console-serving-cert\") pod \"console-678df5d84b-v29jh\" (UID: \"fe815282-50d9-4ebd-a970-ca9c70a92dd9\") " pod="openshift-console/console-678df5d84b-v29jh"
Jul 08 12:04:24.997762 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:24.997713    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fe815282-50d9-4ebd-a970-ca9c70a92dd9-console-oauth-config\") pod \"console-678df5d84b-v29jh\" (UID: \"fe815282-50d9-4ebd-a970-ca9c70a92dd9\") " pod="openshift-console/console-678df5d84b-v29jh"
Jul 08 12:04:25.005414 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:25.005360    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgbnw\" (UniqueName: \"kubernetes.io/projected/fe815282-50d9-4ebd-a970-ca9c70a92dd9-kube-api-access-vgbnw\") pod \"console-678df5d84b-v29jh\" (UID: \"fe815282-50d9-4ebd-a970-ca9c70a92dd9\") " pod="openshift-console/console-678df5d84b-v29jh"
Jul 08 12:04:25.092289 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:25.092235    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-678df5d84b-v29jh"
Jul 08 12:04:25.224054 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:25.224032    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-678df5d84b-v29jh"]
Jul 08 12:04:25.226600 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:04:25.226572    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe815282_50d9_4ebd_a970_ca9c70a92dd9.slice/crio-878ebe5b0763e707d57b92a453df93479534c3955e942798d2f4ebe943931f40 WatchSource:0}: Error finding container 878ebe5b0763e707d57b92a453df93479534c3955e942798d2f4ebe943931f40: Status 404 returned error can't find the container with id 878ebe5b0763e707d57b92a453df93479534c3955e942798d2f4ebe943931f40
Jul 08 12:04:25.303331 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:25.303247    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-678df5d84b-v29jh" event={"ID":"fe815282-50d9-4ebd-a970-ca9c70a92dd9","Type":"ContainerStarted","Data":"878ebe5b0763e707d57b92a453df93479534c3955e942798d2f4ebe943931f40"}
Jul 08 12:04:27.201279 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:04:27.201237    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-jprx6" podUID="6fb2ecb5-0ba5-4b15-9654-76648d18ed3b"
Jul 08 12:04:27.310946 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:27.310911    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jprx6"
Jul 08 12:04:28.316153 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:28.316060    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-678df5d84b-v29jh" event={"ID":"fe815282-50d9-4ebd-a970-ca9c70a92dd9","Type":"ContainerStarted","Data":"f11c8e3249b376e1ab630f616ef6e7e298fc06bc679ec4ccaf84c31612aefe70"}
Jul 08 12:04:28.343063 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:28.343015    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-678df5d84b-v29jh" podStartSLOduration=1.5120105910000001 podStartE2EDuration="4.342998824s" totalImagesPullingTime="2.830988233s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 12:04:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 12:04:28.341848851 +0000 UTC m=+158.072223823" watchObservedRunningTime="2026-07-08 12:04:28.342998824 +0000 UTC m=+158.073373795"
Jul 08 12:04:32.053375 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:32.053340    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6fb2ecb5-0ba5-4b15-9654-76648d18ed3b-metrics-tls\") pod \"dns-default-jprx6\" (UID: \"6fb2ecb5-0ba5-4b15-9654-76648d18ed3b\") " pod="openshift-dns/dns-default-jprx6"
Jul 08 12:04:32.055696 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:32.055674    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6fb2ecb5-0ba5-4b15-9654-76648d18ed3b-metrics-tls\") pod \"dns-default-jprx6\" (UID: \"6fb2ecb5-0ba5-4b15-9654-76648d18ed3b\") " pod="openshift-dns/dns-default-jprx6"
Jul 08 12:04:32.114185 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:32.114152    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5pzr4\""
Jul 08 12:04:32.122079 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:32.122052    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jprx6"
Jul 08 12:04:32.252864 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:32.252838    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jprx6"]
Jul 08 12:04:32.255462 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:04:32.255422    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fb2ecb5_0ba5_4b15_9654_76648d18ed3b.slice/crio-28dffd592e92a155f6a64a7cbc0403d03b3fb75621a3225ca05ccf59237e5180 WatchSource:0}: Error finding container 28dffd592e92a155f6a64a7cbc0403d03b3fb75621a3225ca05ccf59237e5180: Status 404 returned error can't find the container with id 28dffd592e92a155f6a64a7cbc0403d03b3fb75621a3225ca05ccf59237e5180
Jul 08 12:04:32.331166 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:32.331081    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jprx6" event={"ID":"6fb2ecb5-0ba5-4b15-9654-76648d18ed3b","Type":"ContainerStarted","Data":"28dffd592e92a155f6a64a7cbc0403d03b3fb75621a3225ca05ccf59237e5180"}
Jul 08 12:04:35.092740 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:35.092692    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-678df5d84b-v29jh"
Jul 08 12:04:35.092740 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:35.092751    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-678df5d84b-v29jh"
Jul 08 12:04:35.097461 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:35.097437    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-678df5d84b-v29jh"
Jul 08 12:04:35.344958 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:35.344846    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jprx6" event={"ID":"6fb2ecb5-0ba5-4b15-9654-76648d18ed3b","Type":"ContainerStarted","Data":"8dac022dd08f75082cb8a44ec1d6b017c095b42c1a8d1e5df0a3ad000a68ecf2"}
Jul 08 12:04:35.344958 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:35.344907    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jprx6" event={"ID":"6fb2ecb5-0ba5-4b15-9654-76648d18ed3b","Type":"ContainerStarted","Data":"e460b56e360e19d044e8fef2c4b3f3244d899ed5b17385a0dae787954567eadc"}
Jul 08 12:04:35.348624 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:35.348604    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-678df5d84b-v29jh"
Jul 08 12:04:35.366020 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:35.365972    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-jprx6" podStartSLOduration=129.08723467 podStartE2EDuration="2m11.365957304s" totalImagesPullingTime="2.278722634s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 12:02:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 12:04:35.364196123 +0000 UTC m=+165.094571098" watchObservedRunningTime="2026-07-08 12:04:35.365957304 +0000 UTC m=+165.096332342"
Jul 08 12:04:36.348972 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:36.348942    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-jprx6"
Jul 08 12:04:38.357897 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:38.357859    2579 generic.go:356] "Generic (PLEG): container finished" podID="79addf06-f231-4993-8ae6-a9bec4194e1a" containerID="273d2f0ac5773b8f47d932ddd401e1ec9e206f93cf71e57b5ed99fe93792bfa6" exitCode=0
Jul 08 12:04:38.358338 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:38.357932    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-xzlcm" event={"ID":"79addf06-f231-4993-8ae6-a9bec4194e1a","Type":"ContainerDied","Data":"273d2f0ac5773b8f47d932ddd401e1ec9e206f93cf71e57b5ed99fe93792bfa6"}
Jul 08 12:04:38.358338 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:38.358318    2579 scope.go:117] "RemoveContainer" containerID="273d2f0ac5773b8f47d932ddd401e1ec9e206f93cf71e57b5ed99fe93792bfa6"
Jul 08 12:04:38.359429 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:38.359407    2579 generic.go:356] "Generic (PLEG): container finished" podID="ebb356ca-384f-4018-bf6d-9c9a1ee748bd" containerID="50bf9dd6f0102b82bb2fef0d20b344abce2b9ce87aa66dcb123f649a4adfbad0" exitCode=0
Jul 08 12:04:38.359550 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:38.359480    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-9bgrq" event={"ID":"ebb356ca-384f-4018-bf6d-9c9a1ee748bd","Type":"ContainerDied","Data":"50bf9dd6f0102b82bb2fef0d20b344abce2b9ce87aa66dcb123f649a4adfbad0"}
Jul 08 12:04:38.359805 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:38.359789    2579 scope.go:117] "RemoveContainer" containerID="50bf9dd6f0102b82bb2fef0d20b344abce2b9ce87aa66dcb123f649a4adfbad0"
Jul 08 12:04:39.366052 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:39.366015    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-9bgrq" event={"ID":"ebb356ca-384f-4018-bf6d-9c9a1ee748bd","Type":"ContainerStarted","Data":"062fcf6729a32a439e08bb294bd977d5f85e8a566e41759a13692abec42bb84a"}
Jul 08 12:04:39.367831 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:39.367798    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-xzlcm" event={"ID":"79addf06-f231-4993-8ae6-a9bec4194e1a","Type":"ContainerStarted","Data":"fa77d80ec9169b97b3c6b6b792f599a2ded76ced4673d603aef734e0a7c481e7"}
Jul 08 12:04:46.354620 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:04:46.354548    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-jprx6"
Jul 08 12:05:28.661209 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:28.661172    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 08 12:05:28.661725 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:28.661460    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5473f735-ebad-4d77-ad8a-24e50f26d3b6" containerName="alertmanager" containerID="cri-o://286ad7bd142034cd3c11ff1f4d8a071f1ce40a5b412e68b79c11ec1b22e362eb" gracePeriod=120
Jul 08 12:05:28.661725 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:28.661544    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5473f735-ebad-4d77-ad8a-24e50f26d3b6" containerName="kube-rbac-proxy-web" containerID="cri-o://fa588d73449b98c73ad8b443a9a059a865ab7b79699f92fd1bb145830563963d" gracePeriod=120
Jul 08 12:05:28.661725 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:28.661566    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5473f735-ebad-4d77-ad8a-24e50f26d3b6" containerName="kube-rbac-proxy" containerID="cri-o://c07a6ea9140a766a0f6e2eba798e70ef016b205cfee8f9e7f024d4bff77cb3c1" gracePeriod=120
Jul 08 12:05:28.661725 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:28.661612    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5473f735-ebad-4d77-ad8a-24e50f26d3b6" containerName="config-reloader" containerID="cri-o://114ac339492f55ef6e58eb09b8f02f2cf8d40fb51918a20399a2cc637d1f4bee" gracePeriod=120
Jul 08 12:05:28.661725 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:28.661534    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5473f735-ebad-4d77-ad8a-24e50f26d3b6" containerName="kube-rbac-proxy-metric" containerID="cri-o://d9c571b677bc48c60ef66a3b9cd31e356018c40f1b1a58b1d281bcad7e758270" gracePeriod=120
Jul 08 12:05:28.661725 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:28.661604    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5473f735-ebad-4d77-ad8a-24e50f26d3b6" containerName="prom-label-proxy" containerID="cri-o://5661028e227d5fccf7d06f1b763998343d48777c487b5b0b36a85c36832ddc2d" gracePeriod=120
Jul 08 12:05:29.556517 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:29.556481    2579 generic.go:356] "Generic (PLEG): container finished" podID="5473f735-ebad-4d77-ad8a-24e50f26d3b6" containerID="5661028e227d5fccf7d06f1b763998343d48777c487b5b0b36a85c36832ddc2d" exitCode=0
Jul 08 12:05:29.556517 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:29.556506    2579 generic.go:356] "Generic (PLEG): container finished" podID="5473f735-ebad-4d77-ad8a-24e50f26d3b6" containerID="c07a6ea9140a766a0f6e2eba798e70ef016b205cfee8f9e7f024d4bff77cb3c1" exitCode=0
Jul 08 12:05:29.556517 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:29.556514    2579 generic.go:356] "Generic (PLEG): container finished" podID="5473f735-ebad-4d77-ad8a-24e50f26d3b6" containerID="114ac339492f55ef6e58eb09b8f02f2cf8d40fb51918a20399a2cc637d1f4bee" exitCode=0
Jul 08 12:05:29.556517 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:29.556521    2579 generic.go:356] "Generic (PLEG): container finished" podID="5473f735-ebad-4d77-ad8a-24e50f26d3b6" containerID="286ad7bd142034cd3c11ff1f4d8a071f1ce40a5b412e68b79c11ec1b22e362eb" exitCode=0
Jul 08 12:05:29.556798 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:29.556548    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5473f735-ebad-4d77-ad8a-24e50f26d3b6","Type":"ContainerDied","Data":"5661028e227d5fccf7d06f1b763998343d48777c487b5b0b36a85c36832ddc2d"}
Jul 08 12:05:29.556798 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:29.556614    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5473f735-ebad-4d77-ad8a-24e50f26d3b6","Type":"ContainerDied","Data":"c07a6ea9140a766a0f6e2eba798e70ef016b205cfee8f9e7f024d4bff77cb3c1"}
Jul 08 12:05:29.556798 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:29.556640    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5473f735-ebad-4d77-ad8a-24e50f26d3b6","Type":"ContainerDied","Data":"114ac339492f55ef6e58eb09b8f02f2cf8d40fb51918a20399a2cc637d1f4bee"}
Jul 08 12:05:29.556798 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:29.556664    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5473f735-ebad-4d77-ad8a-24e50f26d3b6","Type":"ContainerDied","Data":"286ad7bd142034cd3c11ff1f4d8a071f1ce40a5b412e68b79c11ec1b22e362eb"}
Jul 08 12:05:29.907920 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:29.907892    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:05:30.014840 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.014804    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5473f735-ebad-4d77-ad8a-24e50f26d3b6-config-out\") pod \"5473f735-ebad-4d77-ad8a-24e50f26d3b6\" (UID: \"5473f735-ebad-4d77-ad8a-24e50f26d3b6\") "
Jul 08 12:05:30.014840 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.014839    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5473f735-ebad-4d77-ad8a-24e50f26d3b6-tls-assets\") pod \"5473f735-ebad-4d77-ad8a-24e50f26d3b6\" (UID: \"5473f735-ebad-4d77-ad8a-24e50f26d3b6\") "
Jul 08 12:05:30.015050 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.014873    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5473f735-ebad-4d77-ad8a-24e50f26d3b6-secret-alertmanager-kube-rbac-proxy-web\") pod \"5473f735-ebad-4d77-ad8a-24e50f26d3b6\" (UID: \"5473f735-ebad-4d77-ad8a-24e50f26d3b6\") "
Jul 08 12:05:30.015050 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.014914    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5473f735-ebad-4d77-ad8a-24e50f26d3b6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"5473f735-ebad-4d77-ad8a-24e50f26d3b6\" (UID: \"5473f735-ebad-4d77-ad8a-24e50f26d3b6\") "
Jul 08 12:05:30.015050 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.014939    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5473f735-ebad-4d77-ad8a-24e50f26d3b6-web-config\") pod \"5473f735-ebad-4d77-ad8a-24e50f26d3b6\" (UID: \"5473f735-ebad-4d77-ad8a-24e50f26d3b6\") "
Jul 08 12:05:30.015050 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.014981    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5473f735-ebad-4d77-ad8a-24e50f26d3b6-secret-alertmanager-kube-rbac-proxy\") pod \"5473f735-ebad-4d77-ad8a-24e50f26d3b6\" (UID: \"5473f735-ebad-4d77-ad8a-24e50f26d3b6\") "
Jul 08 12:05:30.015234 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.015115    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wq6lg\" (UniqueName: \"kubernetes.io/projected/5473f735-ebad-4d77-ad8a-24e50f26d3b6-kube-api-access-wq6lg\") pod \"5473f735-ebad-4d77-ad8a-24e50f26d3b6\" (UID: \"5473f735-ebad-4d77-ad8a-24e50f26d3b6\") "
Jul 08 12:05:30.015234 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.015164    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5473f735-ebad-4d77-ad8a-24e50f26d3b6-cluster-tls-config\") pod \"5473f735-ebad-4d77-ad8a-24e50f26d3b6\" (UID: \"5473f735-ebad-4d77-ad8a-24e50f26d3b6\") "
Jul 08 12:05:30.015234 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.015202    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5473f735-ebad-4d77-ad8a-24e50f26d3b6-alertmanager-trusted-ca-bundle\") pod \"5473f735-ebad-4d77-ad8a-24e50f26d3b6\" (UID: \"5473f735-ebad-4d77-ad8a-24e50f26d3b6\") "
Jul 08 12:05:30.015234 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.015229    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5473f735-ebad-4d77-ad8a-24e50f26d3b6-alertmanager-main-db\") pod \"5473f735-ebad-4d77-ad8a-24e50f26d3b6\" (UID: \"5473f735-ebad-4d77-ad8a-24e50f26d3b6\") "
Jul 08 12:05:30.015422 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.015292    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5473f735-ebad-4d77-ad8a-24e50f26d3b6-secret-alertmanager-main-tls\") pod \"5473f735-ebad-4d77-ad8a-24e50f26d3b6\" (UID: \"5473f735-ebad-4d77-ad8a-24e50f26d3b6\") "
Jul 08 12:05:30.015422 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.015321    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5473f735-ebad-4d77-ad8a-24e50f26d3b6-config-volume\") pod \"5473f735-ebad-4d77-ad8a-24e50f26d3b6\" (UID: \"5473f735-ebad-4d77-ad8a-24e50f26d3b6\") "
Jul 08 12:05:30.015422 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.015345    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5473f735-ebad-4d77-ad8a-24e50f26d3b6-metrics-client-ca\") pod \"5473f735-ebad-4d77-ad8a-24e50f26d3b6\" (UID: \"5473f735-ebad-4d77-ad8a-24e50f26d3b6\") "
Jul 08 12:05:30.015893 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.015868    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5473f735-ebad-4d77-ad8a-24e50f26d3b6-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "5473f735-ebad-4d77-ad8a-24e50f26d3b6" (UID: "5473f735-ebad-4d77-ad8a-24e50f26d3b6"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 12:05:30.016173 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.016031    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5473f735-ebad-4d77-ad8a-24e50f26d3b6-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "5473f735-ebad-4d77-ad8a-24e50f26d3b6" (UID: "5473f735-ebad-4d77-ad8a-24e50f26d3b6"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 12:05:30.016173 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.016164    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5473f735-ebad-4d77-ad8a-24e50f26d3b6-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "5473f735-ebad-4d77-ad8a-24e50f26d3b6" (UID: "5473f735-ebad-4d77-ad8a-24e50f26d3b6"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 12:05:30.018078 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.018032    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5473f735-ebad-4d77-ad8a-24e50f26d3b6-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "5473f735-ebad-4d77-ad8a-24e50f26d3b6" (UID: "5473f735-ebad-4d77-ad8a-24e50f26d3b6"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 12:05:30.018362 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.018335    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5473f735-ebad-4d77-ad8a-24e50f26d3b6-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "5473f735-ebad-4d77-ad8a-24e50f26d3b6" (UID: "5473f735-ebad-4d77-ad8a-24e50f26d3b6"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 12:05:30.018574 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.018530    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5473f735-ebad-4d77-ad8a-24e50f26d3b6-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "5473f735-ebad-4d77-ad8a-24e50f26d3b6" (UID: "5473f735-ebad-4d77-ad8a-24e50f26d3b6"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 12:05:30.018663 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.018581    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5473f735-ebad-4d77-ad8a-24e50f26d3b6-config-out" (OuterVolumeSpecName: "config-out") pod "5473f735-ebad-4d77-ad8a-24e50f26d3b6" (UID: "5473f735-ebad-4d77-ad8a-24e50f26d3b6"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 12:05:30.018904 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.018880    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5473f735-ebad-4d77-ad8a-24e50f26d3b6-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "5473f735-ebad-4d77-ad8a-24e50f26d3b6" (UID: "5473f735-ebad-4d77-ad8a-24e50f26d3b6"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 12:05:30.019012 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.018947    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5473f735-ebad-4d77-ad8a-24e50f26d3b6-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "5473f735-ebad-4d77-ad8a-24e50f26d3b6" (UID: "5473f735-ebad-4d77-ad8a-24e50f26d3b6"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 12:05:30.019449 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.019429    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5473f735-ebad-4d77-ad8a-24e50f26d3b6-config-volume" (OuterVolumeSpecName: "config-volume") pod "5473f735-ebad-4d77-ad8a-24e50f26d3b6" (UID: "5473f735-ebad-4d77-ad8a-24e50f26d3b6"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 12:05:30.019883 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.019861    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5473f735-ebad-4d77-ad8a-24e50f26d3b6-kube-api-access-wq6lg" (OuterVolumeSpecName: "kube-api-access-wq6lg") pod "5473f735-ebad-4d77-ad8a-24e50f26d3b6" (UID: "5473f735-ebad-4d77-ad8a-24e50f26d3b6"). InnerVolumeSpecName "kube-api-access-wq6lg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 12:05:30.022761 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.022661    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5473f735-ebad-4d77-ad8a-24e50f26d3b6-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "5473f735-ebad-4d77-ad8a-24e50f26d3b6" (UID: "5473f735-ebad-4d77-ad8a-24e50f26d3b6"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 12:05:30.028340 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.028317    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5473f735-ebad-4d77-ad8a-24e50f26d3b6-web-config" (OuterVolumeSpecName: "web-config") pod "5473f735-ebad-4d77-ad8a-24e50f26d3b6" (UID: "5473f735-ebad-4d77-ad8a-24e50f26d3b6"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 12:05:30.116186 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.116085    2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5473f735-ebad-4d77-ad8a-24e50f26d3b6-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-138-89.ec2.internal\" DevicePath \"\""
Jul 08 12:05:30.116186 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.116132    2579 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5473f735-ebad-4d77-ad8a-24e50f26d3b6-web-config\") on node \"ip-10-0-138-89.ec2.internal\" DevicePath \"\""
Jul 08 12:05:30.116186 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.116143    2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5473f735-ebad-4d77-ad8a-24e50f26d3b6-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-138-89.ec2.internal\" DevicePath \"\""
Jul 08 12:05:30.116186 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.116153    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wq6lg\" (UniqueName: \"kubernetes.io/projected/5473f735-ebad-4d77-ad8a-24e50f26d3b6-kube-api-access-wq6lg\") on node \"ip-10-0-138-89.ec2.internal\" DevicePath \"\""
Jul 08 12:05:30.116186 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.116163    2579 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5473f735-ebad-4d77-ad8a-24e50f26d3b6-cluster-tls-config\") on node \"ip-10-0-138-89.ec2.internal\" DevicePath \"\""
Jul 08 12:05:30.116186 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.116171    2579 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5473f735-ebad-4d77-ad8a-24e50f26d3b6-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-138-89.ec2.internal\" DevicePath \"\""
Jul 08 12:05:30.116186 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.116179    2579 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5473f735-ebad-4d77-ad8a-24e50f26d3b6-alertmanager-main-db\") on node \"ip-10-0-138-89.ec2.internal\" DevicePath \"\""
Jul 08 12:05:30.116186 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.116187    2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5473f735-ebad-4d77-ad8a-24e50f26d3b6-secret-alertmanager-main-tls\") on node \"ip-10-0-138-89.ec2.internal\" DevicePath \"\""
Jul 08 12:05:30.116186 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.116195    2579 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5473f735-ebad-4d77-ad8a-24e50f26d3b6-config-volume\") on node \"ip-10-0-138-89.ec2.internal\" DevicePath \"\""
Jul 08 12:05:30.116533 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.116203    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5473f735-ebad-4d77-ad8a-24e50f26d3b6-metrics-client-ca\") on node \"ip-10-0-138-89.ec2.internal\" DevicePath \"\""
Jul 08 12:05:30.116533 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.116211    2579 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5473f735-ebad-4d77-ad8a-24e50f26d3b6-config-out\") on node \"ip-10-0-138-89.ec2.internal\" DevicePath \"\""
Jul 08 12:05:30.116533 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.116218    2579 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5473f735-ebad-4d77-ad8a-24e50f26d3b6-tls-assets\") on node \"ip-10-0-138-89.ec2.internal\" DevicePath \"\""
Jul 08 12:05:30.116533 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.116226    2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5473f735-ebad-4d77-ad8a-24e50f26d3b6-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-138-89.ec2.internal\" DevicePath \"\""
Jul 08 12:05:30.562809 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.562753    2579 generic.go:356] "Generic (PLEG): container finished" podID="5473f735-ebad-4d77-ad8a-24e50f26d3b6" containerID="d9c571b677bc48c60ef66a3b9cd31e356018c40f1b1a58b1d281bcad7e758270" exitCode=0
Jul 08 12:05:30.562809 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.562802    2579 generic.go:356] "Generic (PLEG): container finished" podID="5473f735-ebad-4d77-ad8a-24e50f26d3b6" containerID="fa588d73449b98c73ad8b443a9a059a865ab7b79699f92fd1bb145830563963d" exitCode=0
Jul 08 12:05:30.563035 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.562843    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5473f735-ebad-4d77-ad8a-24e50f26d3b6","Type":"ContainerDied","Data":"d9c571b677bc48c60ef66a3b9cd31e356018c40f1b1a58b1d281bcad7e758270"}
Jul 08 12:05:30.563035 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.562904    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5473f735-ebad-4d77-ad8a-24e50f26d3b6","Type":"ContainerDied","Data":"fa588d73449b98c73ad8b443a9a059a865ab7b79699f92fd1bb145830563963d"}
Jul 08 12:05:30.563035 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.562847    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:05:30.563035 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.562940    2579 scope.go:117] "RemoveContainer" containerID="5661028e227d5fccf7d06f1b763998343d48777c487b5b0b36a85c36832ddc2d"
Jul 08 12:05:30.563035 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.562931    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5473f735-ebad-4d77-ad8a-24e50f26d3b6","Type":"ContainerDied","Data":"212e91714dee3f38249a5207e07e6dd2c7b5bcba10977360e66b5c10261b440c"}
Jul 08 12:05:30.571332 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.571313    2579 scope.go:117] "RemoveContainer" containerID="d9c571b677bc48c60ef66a3b9cd31e356018c40f1b1a58b1d281bcad7e758270"
Jul 08 12:05:30.578192 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.578175    2579 scope.go:117] "RemoveContainer" containerID="c07a6ea9140a766a0f6e2eba798e70ef016b205cfee8f9e7f024d4bff77cb3c1"
Jul 08 12:05:30.585039 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.585019    2579 scope.go:117] "RemoveContainer" containerID="fa588d73449b98c73ad8b443a9a059a865ab7b79699f92fd1bb145830563963d"
Jul 08 12:05:30.589186 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.589164    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 08 12:05:30.592303 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.592284    2579 scope.go:117] "RemoveContainer" containerID="114ac339492f55ef6e58eb09b8f02f2cf8d40fb51918a20399a2cc637d1f4bee"
Jul 08 12:05:30.597199 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.597177    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 08 12:05:30.599835 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.599814    2579 scope.go:117] "RemoveContainer" containerID="286ad7bd142034cd3c11ff1f4d8a071f1ce40a5b412e68b79c11ec1b22e362eb"
Jul 08 12:05:30.606659 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.606638    2579 scope.go:117] "RemoveContainer" containerID="4cc7418027887b5104fa0f0b7b5f2468ceaa12ebca4035741b6dd0e1edf575bb"
Jul 08 12:05:30.613435 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.613421    2579 scope.go:117] "RemoveContainer" containerID="5661028e227d5fccf7d06f1b763998343d48777c487b5b0b36a85c36832ddc2d"
Jul 08 12:05:30.613706 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:05:30.613687    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5661028e227d5fccf7d06f1b763998343d48777c487b5b0b36a85c36832ddc2d\": container with ID starting with 5661028e227d5fccf7d06f1b763998343d48777c487b5b0b36a85c36832ddc2d not found: ID does not exist" containerID="5661028e227d5fccf7d06f1b763998343d48777c487b5b0b36a85c36832ddc2d"
Jul 08 12:05:30.613765 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.613713    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5661028e227d5fccf7d06f1b763998343d48777c487b5b0b36a85c36832ddc2d"} err="failed to get container status \"5661028e227d5fccf7d06f1b763998343d48777c487b5b0b36a85c36832ddc2d\": rpc error: code = NotFound desc = could not find container \"5661028e227d5fccf7d06f1b763998343d48777c487b5b0b36a85c36832ddc2d\": container with ID starting with 5661028e227d5fccf7d06f1b763998343d48777c487b5b0b36a85c36832ddc2d not found: ID does not exist"
Jul 08 12:05:30.613765 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.613747    2579 scope.go:117] "RemoveContainer" containerID="d9c571b677bc48c60ef66a3b9cd31e356018c40f1b1a58b1d281bcad7e758270"
Jul 08 12:05:30.614067 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:05:30.614050    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9c571b677bc48c60ef66a3b9cd31e356018c40f1b1a58b1d281bcad7e758270\": container with ID starting with d9c571b677bc48c60ef66a3b9cd31e356018c40f1b1a58b1d281bcad7e758270 not found: ID does not exist" containerID="d9c571b677bc48c60ef66a3b9cd31e356018c40f1b1a58b1d281bcad7e758270"
Jul 08 12:05:30.614109 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.614075    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9c571b677bc48c60ef66a3b9cd31e356018c40f1b1a58b1d281bcad7e758270"} err="failed to get container status \"d9c571b677bc48c60ef66a3b9cd31e356018c40f1b1a58b1d281bcad7e758270\": rpc error: code = NotFound desc = could not find container \"d9c571b677bc48c60ef66a3b9cd31e356018c40f1b1a58b1d281bcad7e758270\": container with ID starting with d9c571b677bc48c60ef66a3b9cd31e356018c40f1b1a58b1d281bcad7e758270 not found: ID does not exist"
Jul 08 12:05:30.614109 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.614091    2579 scope.go:117] "RemoveContainer" containerID="c07a6ea9140a766a0f6e2eba798e70ef016b205cfee8f9e7f024d4bff77cb3c1"
Jul 08 12:05:30.614299 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:05:30.614284    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c07a6ea9140a766a0f6e2eba798e70ef016b205cfee8f9e7f024d4bff77cb3c1\": container with ID starting with c07a6ea9140a766a0f6e2eba798e70ef016b205cfee8f9e7f024d4bff77cb3c1 not found: ID does not exist" containerID="c07a6ea9140a766a0f6e2eba798e70ef016b205cfee8f9e7f024d4bff77cb3c1"
Jul 08 12:05:30.614340 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.614304    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c07a6ea9140a766a0f6e2eba798e70ef016b205cfee8f9e7f024d4bff77cb3c1"} err="failed to get container status \"c07a6ea9140a766a0f6e2eba798e70ef016b205cfee8f9e7f024d4bff77cb3c1\": rpc error: code = NotFound desc = could not find container \"c07a6ea9140a766a0f6e2eba798e70ef016b205cfee8f9e7f024d4bff77cb3c1\": container with ID starting with c07a6ea9140a766a0f6e2eba798e70ef016b205cfee8f9e7f024d4bff77cb3c1 not found: ID does not exist"
Jul 08 12:05:30.614340 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.614319    2579 scope.go:117] "RemoveContainer" containerID="fa588d73449b98c73ad8b443a9a059a865ab7b79699f92fd1bb145830563963d"
Jul 08 12:05:30.614545 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:05:30.614531    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa588d73449b98c73ad8b443a9a059a865ab7b79699f92fd1bb145830563963d\": container with ID starting with fa588d73449b98c73ad8b443a9a059a865ab7b79699f92fd1bb145830563963d not found: ID does not exist" containerID="fa588d73449b98c73ad8b443a9a059a865ab7b79699f92fd1bb145830563963d"
Jul 08 12:05:30.614595 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.614548    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa588d73449b98c73ad8b443a9a059a865ab7b79699f92fd1bb145830563963d"} err="failed to get container status \"fa588d73449b98c73ad8b443a9a059a865ab7b79699f92fd1bb145830563963d\": rpc error: code = NotFound desc = could not find container \"fa588d73449b98c73ad8b443a9a059a865ab7b79699f92fd1bb145830563963d\": container with ID starting with fa588d73449b98c73ad8b443a9a059a865ab7b79699f92fd1bb145830563963d not found: ID does not exist"
Jul 08 12:05:30.614595 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.614560    2579 scope.go:117] "RemoveContainer" containerID="114ac339492f55ef6e58eb09b8f02f2cf8d40fb51918a20399a2cc637d1f4bee"
Jul 08 12:05:30.614737 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:05:30.614718    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"114ac339492f55ef6e58eb09b8f02f2cf8d40fb51918a20399a2cc637d1f4bee\": container with ID starting with 114ac339492f55ef6e58eb09b8f02f2cf8d40fb51918a20399a2cc637d1f4bee not found: ID does not exist" containerID="114ac339492f55ef6e58eb09b8f02f2cf8d40fb51918a20399a2cc637d1f4bee"
Jul 08 12:05:30.614794 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.614741    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"114ac339492f55ef6e58eb09b8f02f2cf8d40fb51918a20399a2cc637d1f4bee"} err="failed to get container status \"114ac339492f55ef6e58eb09b8f02f2cf8d40fb51918a20399a2cc637d1f4bee\": rpc error: code = NotFound desc = could not find container \"114ac339492f55ef6e58eb09b8f02f2cf8d40fb51918a20399a2cc637d1f4bee\": container with ID starting with 114ac339492f55ef6e58eb09b8f02f2cf8d40fb51918a20399a2cc637d1f4bee not found: ID does not exist"
Jul 08 12:05:30.614794 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.614753    2579 scope.go:117] "RemoveContainer" containerID="286ad7bd142034cd3c11ff1f4d8a071f1ce40a5b412e68b79c11ec1b22e362eb"
Jul 08 12:05:30.614974 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:05:30.614958    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"286ad7bd142034cd3c11ff1f4d8a071f1ce40a5b412e68b79c11ec1b22e362eb\": container with ID starting with 286ad7bd142034cd3c11ff1f4d8a071f1ce40a5b412e68b79c11ec1b22e362eb not found: ID does not exist" containerID="286ad7bd142034cd3c11ff1f4d8a071f1ce40a5b412e68b79c11ec1b22e362eb"
Jul 08 12:05:30.615017 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.614977    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"286ad7bd142034cd3c11ff1f4d8a071f1ce40a5b412e68b79c11ec1b22e362eb"} err="failed to get container status \"286ad7bd142034cd3c11ff1f4d8a071f1ce40a5b412e68b79c11ec1b22e362eb\": rpc error: code = NotFound desc = could not find container \"286ad7bd142034cd3c11ff1f4d8a071f1ce40a5b412e68b79c11ec1b22e362eb\": container with ID starting with 286ad7bd142034cd3c11ff1f4d8a071f1ce40a5b412e68b79c11ec1b22e362eb not found: ID does not exist"
Jul 08 12:05:30.615017 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.614988    2579 scope.go:117] "RemoveContainer" containerID="4cc7418027887b5104fa0f0b7b5f2468ceaa12ebca4035741b6dd0e1edf575bb"
Jul 08 12:05:30.615175 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:05:30.615161    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cc7418027887b5104fa0f0b7b5f2468ceaa12ebca4035741b6dd0e1edf575bb\": container with ID starting with 4cc7418027887b5104fa0f0b7b5f2468ceaa12ebca4035741b6dd0e1edf575bb not found: ID does not exist" containerID="4cc7418027887b5104fa0f0b7b5f2468ceaa12ebca4035741b6dd0e1edf575bb"
Jul 08 12:05:30.615212 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.615178    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cc7418027887b5104fa0f0b7b5f2468ceaa12ebca4035741b6dd0e1edf575bb"} err="failed to get container status \"4cc7418027887b5104fa0f0b7b5f2468ceaa12ebca4035741b6dd0e1edf575bb\": rpc error: code = NotFound desc = could not find container \"4cc7418027887b5104fa0f0b7b5f2468ceaa12ebca4035741b6dd0e1edf575bb\": container with ID starting with 4cc7418027887b5104fa0f0b7b5f2468ceaa12ebca4035741b6dd0e1edf575bb not found: ID does not exist"
Jul 08 12:05:30.615212 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.615191    2579 scope.go:117] "RemoveContainer" containerID="5661028e227d5fccf7d06f1b763998343d48777c487b5b0b36a85c36832ddc2d"
Jul 08 12:05:30.615389 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.615373    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5661028e227d5fccf7d06f1b763998343d48777c487b5b0b36a85c36832ddc2d"} err="failed to get container status \"5661028e227d5fccf7d06f1b763998343d48777c487b5b0b36a85c36832ddc2d\": rpc error: code = NotFound desc = could not find container \"5661028e227d5fccf7d06f1b763998343d48777c487b5b0b36a85c36832ddc2d\": container with ID starting with 5661028e227d5fccf7d06f1b763998343d48777c487b5b0b36a85c36832ddc2d not found: ID does not exist"
Jul 08 12:05:30.615436 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.615389    2579 scope.go:117] "RemoveContainer" containerID="d9c571b677bc48c60ef66a3b9cd31e356018c40f1b1a58b1d281bcad7e758270"
Jul 08 12:05:30.615559 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.615545    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9c571b677bc48c60ef66a3b9cd31e356018c40f1b1a58b1d281bcad7e758270"} err="failed to get container status \"d9c571b677bc48c60ef66a3b9cd31e356018c40f1b1a58b1d281bcad7e758270\": rpc error: code = NotFound desc = could not find container \"d9c571b677bc48c60ef66a3b9cd31e356018c40f1b1a58b1d281bcad7e758270\": container with ID starting with d9c571b677bc48c60ef66a3b9cd31e356018c40f1b1a58b1d281bcad7e758270 not found: ID does not exist"
Jul 08 12:05:30.615598 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.615559    2579 scope.go:117] "RemoveContainer" containerID="c07a6ea9140a766a0f6e2eba798e70ef016b205cfee8f9e7f024d4bff77cb3c1"
Jul 08 12:05:30.615730 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.615715    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c07a6ea9140a766a0f6e2eba798e70ef016b205cfee8f9e7f024d4bff77cb3c1"} err="failed to get container status \"c07a6ea9140a766a0f6e2eba798e70ef016b205cfee8f9e7f024d4bff77cb3c1\": rpc error: code = NotFound desc = could not find container \"c07a6ea9140a766a0f6e2eba798e70ef016b205cfee8f9e7f024d4bff77cb3c1\": container with ID starting with c07a6ea9140a766a0f6e2eba798e70ef016b205cfee8f9e7f024d4bff77cb3c1 not found: ID does not exist"
Jul 08 12:05:30.615799 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.615730    2579 scope.go:117] "RemoveContainer" containerID="fa588d73449b98c73ad8b443a9a059a865ab7b79699f92fd1bb145830563963d"
Jul 08 12:05:30.615940 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.615923    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa588d73449b98c73ad8b443a9a059a865ab7b79699f92fd1bb145830563963d"} err="failed to get container status \"fa588d73449b98c73ad8b443a9a059a865ab7b79699f92fd1bb145830563963d\": rpc error: code = NotFound desc = could not find container \"fa588d73449b98c73ad8b443a9a059a865ab7b79699f92fd1bb145830563963d\": container with ID starting with fa588d73449b98c73ad8b443a9a059a865ab7b79699f92fd1bb145830563963d not found: ID does not exist"
Jul 08 12:05:30.615988 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.615941    2579 scope.go:117] "RemoveContainer" containerID="114ac339492f55ef6e58eb09b8f02f2cf8d40fb51918a20399a2cc637d1f4bee"
Jul 08 12:05:30.616146 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.616129    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"114ac339492f55ef6e58eb09b8f02f2cf8d40fb51918a20399a2cc637d1f4bee"} err="failed to get container status \"114ac339492f55ef6e58eb09b8f02f2cf8d40fb51918a20399a2cc637d1f4bee\": rpc error: code = NotFound desc = could not find container \"114ac339492f55ef6e58eb09b8f02f2cf8d40fb51918a20399a2cc637d1f4bee\": container with ID starting with 114ac339492f55ef6e58eb09b8f02f2cf8d40fb51918a20399a2cc637d1f4bee not found: ID does not exist"
Jul 08 12:05:30.616181 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.616147    2579 scope.go:117] "RemoveContainer" containerID="286ad7bd142034cd3c11ff1f4d8a071f1ce40a5b412e68b79c11ec1b22e362eb"
Jul 08 12:05:30.616315 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.616300    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"286ad7bd142034cd3c11ff1f4d8a071f1ce40a5b412e68b79c11ec1b22e362eb"} err="failed to get container status \"286ad7bd142034cd3c11ff1f4d8a071f1ce40a5b412e68b79c11ec1b22e362eb\": rpc error: code = NotFound desc = could not find container \"286ad7bd142034cd3c11ff1f4d8a071f1ce40a5b412e68b79c11ec1b22e362eb\": container with ID starting with 286ad7bd142034cd3c11ff1f4d8a071f1ce40a5b412e68b79c11ec1b22e362eb not found: ID does not exist"
Jul 08 12:05:30.616351 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.616315    2579 scope.go:117] "RemoveContainer" containerID="4cc7418027887b5104fa0f0b7b5f2468ceaa12ebca4035741b6dd0e1edf575bb"
Jul 08 12:05:30.616477 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.616462    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cc7418027887b5104fa0f0b7b5f2468ceaa12ebca4035741b6dd0e1edf575bb"} err="failed to get container status \"4cc7418027887b5104fa0f0b7b5f2468ceaa12ebca4035741b6dd0e1edf575bb\": rpc error: code = NotFound desc = could not find container \"4cc7418027887b5104fa0f0b7b5f2468ceaa12ebca4035741b6dd0e1edf575bb\": container with ID starting with 4cc7418027887b5104fa0f0b7b5f2468ceaa12ebca4035741b6dd0e1edf575bb not found: ID does not exist"
Jul 08 12:05:30.632948 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.632921    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 08 12:05:30.633155 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.633144    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5473f735-ebad-4d77-ad8a-24e50f26d3b6" containerName="alertmanager"
Jul 08 12:05:30.633195 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.633156    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5473f735-ebad-4d77-ad8a-24e50f26d3b6" containerName="alertmanager"
Jul 08 12:05:30.633195 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.633165    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5473f735-ebad-4d77-ad8a-24e50f26d3b6" containerName="config-reloader"
Jul 08 12:05:30.633195 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.633172    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5473f735-ebad-4d77-ad8a-24e50f26d3b6" containerName="config-reloader"
Jul 08 12:05:30.633195 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.633180    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5473f735-ebad-4d77-ad8a-24e50f26d3b6" containerName="prom-label-proxy"
Jul 08 12:05:30.633195 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.633186    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5473f735-ebad-4d77-ad8a-24e50f26d3b6" containerName="prom-label-proxy"
Jul 08 12:05:30.633195 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.633193    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5473f735-ebad-4d77-ad8a-24e50f26d3b6" containerName="kube-rbac-proxy-web"
Jul 08 12:05:30.633367 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.633198    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5473f735-ebad-4d77-ad8a-24e50f26d3b6" containerName="kube-rbac-proxy-web"
Jul 08 12:05:30.633367 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.633210    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5473f735-ebad-4d77-ad8a-24e50f26d3b6" containerName="kube-rbac-proxy"
Jul 08 12:05:30.633367 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.633215    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5473f735-ebad-4d77-ad8a-24e50f26d3b6" containerName="kube-rbac-proxy"
Jul 08 12:05:30.633367 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.633222    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5473f735-ebad-4d77-ad8a-24e50f26d3b6" containerName="init-config-reloader"
Jul 08 12:05:30.633367 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.633227    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5473f735-ebad-4d77-ad8a-24e50f26d3b6" containerName="init-config-reloader"
Jul 08 12:05:30.633367 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.633232    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5473f735-ebad-4d77-ad8a-24e50f26d3b6" containerName="kube-rbac-proxy-metric"
Jul 08 12:05:30.633367 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.633237    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5473f735-ebad-4d77-ad8a-24e50f26d3b6" containerName="kube-rbac-proxy-metric"
Jul 08 12:05:30.633367 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.633276    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5473f735-ebad-4d77-ad8a-24e50f26d3b6" containerName="kube-rbac-proxy-metric"
Jul 08 12:05:30.633367 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.633282    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5473f735-ebad-4d77-ad8a-24e50f26d3b6" containerName="prom-label-proxy"
Jul 08 12:05:30.633367 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.633288    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5473f735-ebad-4d77-ad8a-24e50f26d3b6" containerName="kube-rbac-proxy-web"
Jul 08 12:05:30.633367 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.633292    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5473f735-ebad-4d77-ad8a-24e50f26d3b6" containerName="alertmanager"
Jul 08 12:05:30.633367 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.633297    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5473f735-ebad-4d77-ad8a-24e50f26d3b6" containerName="kube-rbac-proxy"
Jul 08 12:05:30.633367 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.633302    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5473f735-ebad-4d77-ad8a-24e50f26d3b6" containerName="config-reloader"
Jul 08 12:05:30.636304 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.636288    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:05:30.638707 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.638684    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 08 12:05:30.638813 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.638689    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 08 12:05:30.638985 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.638967    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 08 12:05:30.639063 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.639037    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 08 12:05:30.639118 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.639070    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-tfj4m\""
Jul 08 12:05:30.639118 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.639111    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 08 12:05:30.639366 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.639351    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 08 12:05:30.639407 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.639393    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 08 12:05:30.639472 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.639457    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 08 12:05:30.646923 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.646899    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 08 12:05:30.653721 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.653693    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 08 12:05:30.715249 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.715214    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5473f735-ebad-4d77-ad8a-24e50f26d3b6" path="/var/lib/kubelet/pods/5473f735-ebad-4d77-ad8a-24e50f26d3b6/volumes"
Jul 08 12:05:30.720182 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.720156    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d7ac8123-78b8-4ca2-8ed7-e118f28a3248-config-volume\") pod \"alertmanager-main-0\" (UID: \"d7ac8123-78b8-4ca2-8ed7-e118f28a3248\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:05:30.720325 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.720196    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d7ac8123-78b8-4ca2-8ed7-e118f28a3248-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d7ac8123-78b8-4ca2-8ed7-e118f28a3248\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:05:30.720325 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.720224    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d7ac8123-78b8-4ca2-8ed7-e118f28a3248-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d7ac8123-78b8-4ca2-8ed7-e118f28a3248\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:05:30.720325 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.720258    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d7ac8123-78b8-4ca2-8ed7-e118f28a3248-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d7ac8123-78b8-4ca2-8ed7-e118f28a3248\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:05:30.720325 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.720279    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d7ac8123-78b8-4ca2-8ed7-e118f28a3248-config-out\") pod \"alertmanager-main-0\" (UID: \"d7ac8123-78b8-4ca2-8ed7-e118f28a3248\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:05:30.720325 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.720310    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d7ac8123-78b8-4ca2-8ed7-e118f28a3248-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d7ac8123-78b8-4ca2-8ed7-e118f28a3248\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:05:30.720540 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.720354    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7ac8123-78b8-4ca2-8ed7-e118f28a3248-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d7ac8123-78b8-4ca2-8ed7-e118f28a3248\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:05:30.720540 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.720406    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d7ac8123-78b8-4ca2-8ed7-e118f28a3248-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d7ac8123-78b8-4ca2-8ed7-e118f28a3248\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:05:30.720540 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.720428    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d7ac8123-78b8-4ca2-8ed7-e118f28a3248-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d7ac8123-78b8-4ca2-8ed7-e118f28a3248\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:05:30.720540 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.720453    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d7ac8123-78b8-4ca2-8ed7-e118f28a3248-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d7ac8123-78b8-4ca2-8ed7-e118f28a3248\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:05:30.720540 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.720468    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d7ac8123-78b8-4ca2-8ed7-e118f28a3248-web-config\") pod \"alertmanager-main-0\" (UID: \"d7ac8123-78b8-4ca2-8ed7-e118f28a3248\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:05:30.720540 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.720510    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rxfl\" (UniqueName: \"kubernetes.io/projected/d7ac8123-78b8-4ca2-8ed7-e118f28a3248-kube-api-access-8rxfl\") pod \"alertmanager-main-0\" (UID: \"d7ac8123-78b8-4ca2-8ed7-e118f28a3248\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:05:30.720540 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.720534    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d7ac8123-78b8-4ca2-8ed7-e118f28a3248-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d7ac8123-78b8-4ca2-8ed7-e118f28a3248\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:05:30.821817 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.821699    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d7ac8123-78b8-4ca2-8ed7-e118f28a3248-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d7ac8123-78b8-4ca2-8ed7-e118f28a3248\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:05:30.821817 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.821746    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d7ac8123-78b8-4ca2-8ed7-e118f28a3248-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d7ac8123-78b8-4ca2-8ed7-e118f28a3248\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:05:30.821817 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.821792    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d7ac8123-78b8-4ca2-8ed7-e118f28a3248-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d7ac8123-78b8-4ca2-8ed7-e118f28a3248\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:05:30.821817 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.821811    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d7ac8123-78b8-4ca2-8ed7-e118f28a3248-web-config\") pod \"alertmanager-main-0\" (UID: \"d7ac8123-78b8-4ca2-8ed7-e118f28a3248\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:05:30.822110 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.821835    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rxfl\" (UniqueName: \"kubernetes.io/projected/d7ac8123-78b8-4ca2-8ed7-e118f28a3248-kube-api-access-8rxfl\") pod \"alertmanager-main-0\" (UID: \"d7ac8123-78b8-4ca2-8ed7-e118f28a3248\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:05:30.822110 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.821853    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d7ac8123-78b8-4ca2-8ed7-e118f28a3248-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d7ac8123-78b8-4ca2-8ed7-e118f28a3248\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:05:30.822110 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.821875    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d7ac8123-78b8-4ca2-8ed7-e118f28a3248-config-volume\") pod \"alertmanager-main-0\" (UID: \"d7ac8123-78b8-4ca2-8ed7-e118f28a3248\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:05:30.822110 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.821892    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d7ac8123-78b8-4ca2-8ed7-e118f28a3248-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d7ac8123-78b8-4ca2-8ed7-e118f28a3248\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:05:30.822110 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.821914    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d7ac8123-78b8-4ca2-8ed7-e118f28a3248-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d7ac8123-78b8-4ca2-8ed7-e118f28a3248\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:05:30.822110 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.821944    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d7ac8123-78b8-4ca2-8ed7-e118f28a3248-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d7ac8123-78b8-4ca2-8ed7-e118f28a3248\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:05:30.822110 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.821964    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d7ac8123-78b8-4ca2-8ed7-e118f28a3248-config-out\") pod \"alertmanager-main-0\" (UID: \"d7ac8123-78b8-4ca2-8ed7-e118f28a3248\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:05:30.822110 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.821994    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d7ac8123-78b8-4ca2-8ed7-e118f28a3248-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d7ac8123-78b8-4ca2-8ed7-e118f28a3248\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:05:30.822110 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.822021    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7ac8123-78b8-4ca2-8ed7-e118f28a3248-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d7ac8123-78b8-4ca2-8ed7-e118f28a3248\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:05:30.822451 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.822333    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d7ac8123-78b8-4ca2-8ed7-e118f28a3248-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d7ac8123-78b8-4ca2-8ed7-e118f28a3248\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:05:30.822618 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.822585    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d7ac8123-78b8-4ca2-8ed7-e118f28a3248-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d7ac8123-78b8-4ca2-8ed7-e118f28a3248\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:05:30.822958 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.822863    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7ac8123-78b8-4ca2-8ed7-e118f28a3248-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d7ac8123-78b8-4ca2-8ed7-e118f28a3248\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:05:30.825274 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.825233    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d7ac8123-78b8-4ca2-8ed7-e118f28a3248-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d7ac8123-78b8-4ca2-8ed7-e118f28a3248\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:05:30.825729 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.825395    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d7ac8123-78b8-4ca2-8ed7-e118f28a3248-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d7ac8123-78b8-4ca2-8ed7-e118f28a3248\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:05:30.825729 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.825644    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d7ac8123-78b8-4ca2-8ed7-e118f28a3248-web-config\") pod \"alertmanager-main-0\" (UID: \"d7ac8123-78b8-4ca2-8ed7-e118f28a3248\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:05:30.825729 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.825688    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d7ac8123-78b8-4ca2-8ed7-e118f28a3248-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d7ac8123-78b8-4ca2-8ed7-e118f28a3248\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:05:30.825937 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.825765    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d7ac8123-78b8-4ca2-8ed7-e118f28a3248-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d7ac8123-78b8-4ca2-8ed7-e118f28a3248\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:05:30.825937 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.825845    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d7ac8123-78b8-4ca2-8ed7-e118f28a3248-config-out\") pod \"alertmanager-main-0\" (UID: \"d7ac8123-78b8-4ca2-8ed7-e118f28a3248\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:05:30.825937 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.825873    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d7ac8123-78b8-4ca2-8ed7-e118f28a3248-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d7ac8123-78b8-4ca2-8ed7-e118f28a3248\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:05:30.826028 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.825945    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d7ac8123-78b8-4ca2-8ed7-e118f28a3248-config-volume\") pod \"alertmanager-main-0\" (UID: \"d7ac8123-78b8-4ca2-8ed7-e118f28a3248\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:05:30.826902 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.826885    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d7ac8123-78b8-4ca2-8ed7-e118f28a3248-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d7ac8123-78b8-4ca2-8ed7-e118f28a3248\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:05:30.835098 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.835065    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rxfl\" (UniqueName: \"kubernetes.io/projected/d7ac8123-78b8-4ca2-8ed7-e118f28a3248-kube-api-access-8rxfl\") pod \"alertmanager-main-0\" (UID: \"d7ac8123-78b8-4ca2-8ed7-e118f28a3248\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:05:30.949763 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:30.949726    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 08 12:05:31.089845 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:31.089665    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 08 12:05:31.093175 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:05:31.093151    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7ac8123_78b8_4ca2_8ed7_e118f28a3248.slice/crio-9f761fe6d50ae57789429b4406e6f54736ca8d911bf0f27ad8484bdbfd1bb5d5 WatchSource:0}: Error finding container 9f761fe6d50ae57789429b4406e6f54736ca8d911bf0f27ad8484bdbfd1bb5d5: Status 404 returned error can't find the container with id 9f761fe6d50ae57789429b4406e6f54736ca8d911bf0f27ad8484bdbfd1bb5d5
Jul 08 12:05:31.571718 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:31.571684    2579 generic.go:356] "Generic (PLEG): container finished" podID="d7ac8123-78b8-4ca2-8ed7-e118f28a3248" containerID="c4af85c22f02f2a131c91069479ff8db09c6141e506977189c8cbb26d3b4fbfd" exitCode=0
Jul 08 12:05:31.571950 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:31.571730    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d7ac8123-78b8-4ca2-8ed7-e118f28a3248","Type":"ContainerDied","Data":"c4af85c22f02f2a131c91069479ff8db09c6141e506977189c8cbb26d3b4fbfd"}
Jul 08 12:05:31.571950 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:31.571820    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d7ac8123-78b8-4ca2-8ed7-e118f28a3248","Type":"ContainerStarted","Data":"9f761fe6d50ae57789429b4406e6f54736ca8d911bf0f27ad8484bdbfd1bb5d5"}
Jul 08 12:05:32.578615 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:32.578583    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d7ac8123-78b8-4ca2-8ed7-e118f28a3248","Type":"ContainerStarted","Data":"bd34671bf438b2e0fb6f378c43cdf940e5d35a6489fab01a9c6548421e9061e6"}
Jul 08 12:05:32.579019 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:32.578643    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d7ac8123-78b8-4ca2-8ed7-e118f28a3248","Type":"ContainerStarted","Data":"c8e0dc30a8d5f156c193353ffe9cda7236fff66387c07ae22295644782c81691"}
Jul 08 12:05:32.579019 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:32.578672    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d7ac8123-78b8-4ca2-8ed7-e118f28a3248","Type":"ContainerStarted","Data":"c687d4909d2545dcd18c1eb5b10e8dd99bdf9ef158ee29491396080bf26b13e2"}
Jul 08 12:05:32.579019 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:32.578698    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d7ac8123-78b8-4ca2-8ed7-e118f28a3248","Type":"ContainerStarted","Data":"841cbec734ba6025ab12e60df4bdeac878718e89000bd9454692dd402092613b"}
Jul 08 12:05:32.579019 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:32.578732    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d7ac8123-78b8-4ca2-8ed7-e118f28a3248","Type":"ContainerStarted","Data":"581dba6e68882a72bd5ef38a532c5bd2be9b43f3f12069390ee76fb5330e432c"}
Jul 08 12:05:32.579019 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:32.578758    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d7ac8123-78b8-4ca2-8ed7-e118f28a3248","Type":"ContainerStarted","Data":"a8fb7ccdd8131a0c1189bbc26825ba842fa176c7074979f176eaf2934d69cb35"}
Jul 08 12:05:32.610741 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:32.610686    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.595166962 podStartE2EDuration="2.610670236s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.503274ms" isStatefulPod=false podCreationTimestamp="2026-07-08 12:05:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 12:05:32.608512607 +0000 UTC m=+222.338887616" watchObservedRunningTime="2026-07-08 12:05:32.610670236 +0000 UTC m=+222.341045206"
Jul 08 12:05:42.135261 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:05:42.135214    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-678df5d84b-v29jh"]
Jul 08 12:06:07.161978 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:06:07.161927    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-678df5d84b-v29jh" podUID="fe815282-50d9-4ebd-a970-ca9c70a92dd9" containerName="console" containerID="cri-o://f11c8e3249b376e1ab630f616ef6e7e298fc06bc679ec4ccaf84c31612aefe70" gracePeriod=15
Jul 08 12:06:07.399740 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:06:07.399717    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-678df5d84b-v29jh"
Jul 08 12:06:07.523751 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:06:07.523716    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fe815282-50d9-4ebd-a970-ca9c70a92dd9-console-config\") pod \"fe815282-50d9-4ebd-a970-ca9c70a92dd9\" (UID: \"fe815282-50d9-4ebd-a970-ca9c70a92dd9\") "
Jul 08 12:06:07.523751 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:06:07.523751    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fe815282-50d9-4ebd-a970-ca9c70a92dd9-oauth-serving-cert\") pod \"fe815282-50d9-4ebd-a970-ca9c70a92dd9\" (UID: \"fe815282-50d9-4ebd-a970-ca9c70a92dd9\") "
Jul 08 12:06:07.524007 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:06:07.523833    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe815282-50d9-4ebd-a970-ca9c70a92dd9-trusted-ca-bundle\") pod \"fe815282-50d9-4ebd-a970-ca9c70a92dd9\" (UID: \"fe815282-50d9-4ebd-a970-ca9c70a92dd9\") "
Jul 08 12:06:07.524007 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:06:07.523877    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fe815282-50d9-4ebd-a970-ca9c70a92dd9-service-ca\") pod \"fe815282-50d9-4ebd-a970-ca9c70a92dd9\" (UID: \"fe815282-50d9-4ebd-a970-ca9c70a92dd9\") "
Jul 08 12:06:07.524084 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:06:07.524004    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fe815282-50d9-4ebd-a970-ca9c70a92dd9-console-oauth-config\") pod \"fe815282-50d9-4ebd-a970-ca9c70a92dd9\" (UID: \"fe815282-50d9-4ebd-a970-ca9c70a92dd9\") "
Jul 08 12:06:07.524084 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:06:07.524061    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fe815282-50d9-4ebd-a970-ca9c70a92dd9-console-serving-cert\") pod \"fe815282-50d9-4ebd-a970-ca9c70a92dd9\" (UID: \"fe815282-50d9-4ebd-a970-ca9c70a92dd9\") "
Jul 08 12:06:07.524176 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:06:07.524086    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vgbnw\" (UniqueName: \"kubernetes.io/projected/fe815282-50d9-4ebd-a970-ca9c70a92dd9-kube-api-access-vgbnw\") pod \"fe815282-50d9-4ebd-a970-ca9c70a92dd9\" (UID: \"fe815282-50d9-4ebd-a970-ca9c70a92dd9\") "
Jul 08 12:06:07.524251 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:06:07.524220    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe815282-50d9-4ebd-a970-ca9c70a92dd9-console-config" (OuterVolumeSpecName: "console-config") pod "fe815282-50d9-4ebd-a970-ca9c70a92dd9" (UID: "fe815282-50d9-4ebd-a970-ca9c70a92dd9"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 12:06:07.524377 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:06:07.524275    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe815282-50d9-4ebd-a970-ca9c70a92dd9-service-ca" (OuterVolumeSpecName: "service-ca") pod "fe815282-50d9-4ebd-a970-ca9c70a92dd9" (UID: "fe815282-50d9-4ebd-a970-ca9c70a92dd9"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 12:06:07.524426 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:06:07.524283    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe815282-50d9-4ebd-a970-ca9c70a92dd9-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "fe815282-50d9-4ebd-a970-ca9c70a92dd9" (UID: "fe815282-50d9-4ebd-a970-ca9c70a92dd9"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 12:06:07.524426 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:06:07.524305    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe815282-50d9-4ebd-a970-ca9c70a92dd9-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "fe815282-50d9-4ebd-a970-ca9c70a92dd9" (UID: "fe815282-50d9-4ebd-a970-ca9c70a92dd9"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 12:06:07.526251 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:06:07.526225    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe815282-50d9-4ebd-a970-ca9c70a92dd9-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "fe815282-50d9-4ebd-a970-ca9c70a92dd9" (UID: "fe815282-50d9-4ebd-a970-ca9c70a92dd9"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 12:06:07.526336 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:06:07.526313    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe815282-50d9-4ebd-a970-ca9c70a92dd9-kube-api-access-vgbnw" (OuterVolumeSpecName: "kube-api-access-vgbnw") pod "fe815282-50d9-4ebd-a970-ca9c70a92dd9" (UID: "fe815282-50d9-4ebd-a970-ca9c70a92dd9"). InnerVolumeSpecName "kube-api-access-vgbnw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 12:06:07.526381 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:06:07.526363    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe815282-50d9-4ebd-a970-ca9c70a92dd9-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "fe815282-50d9-4ebd-a970-ca9c70a92dd9" (UID: "fe815282-50d9-4ebd-a970-ca9c70a92dd9"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 12:06:07.625463 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:06:07.625410    2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fe815282-50d9-4ebd-a970-ca9c70a92dd9-service-ca\") on node \"ip-10-0-138-89.ec2.internal\" DevicePath \"\""
Jul 08 12:06:07.625463 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:06:07.625455    2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fe815282-50d9-4ebd-a970-ca9c70a92dd9-console-oauth-config\") on node \"ip-10-0-138-89.ec2.internal\" DevicePath \"\""
Jul 08 12:06:07.625463 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:06:07.625466    2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fe815282-50d9-4ebd-a970-ca9c70a92dd9-console-serving-cert\") on node \"ip-10-0-138-89.ec2.internal\" DevicePath \"\""
Jul 08 12:06:07.625463 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:06:07.625474    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgbnw\" (UniqueName: \"kubernetes.io/projected/fe815282-50d9-4ebd-a970-ca9c70a92dd9-kube-api-access-vgbnw\") on node \"ip-10-0-138-89.ec2.internal\" DevicePath \"\""
Jul 08 12:06:07.625463 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:06:07.625483    2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fe815282-50d9-4ebd-a970-ca9c70a92dd9-console-config\") on node \"ip-10-0-138-89.ec2.internal\" DevicePath \"\""
Jul 08 12:06:07.625755 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:06:07.625491    2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fe815282-50d9-4ebd-a970-ca9c70a92dd9-oauth-serving-cert\") on node \"ip-10-0-138-89.ec2.internal\" DevicePath \"\""
Jul 08 12:06:07.625755 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:06:07.625501    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe815282-50d9-4ebd-a970-ca9c70a92dd9-trusted-ca-bundle\") on node \"ip-10-0-138-89.ec2.internal\" DevicePath \"\""
Jul 08 12:06:07.708610 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:06:07.708575    2579 generic.go:356] "Generic (PLEG): container finished" podID="fe815282-50d9-4ebd-a970-ca9c70a92dd9" containerID="f11c8e3249b376e1ab630f616ef6e7e298fc06bc679ec4ccaf84c31612aefe70" exitCode=0
Jul 08 12:06:07.708795 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:06:07.708626    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-678df5d84b-v29jh"
Jul 08 12:06:07.708795 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:06:07.708666    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-678df5d84b-v29jh" event={"ID":"fe815282-50d9-4ebd-a970-ca9c70a92dd9","Type":"ContainerDied","Data":"f11c8e3249b376e1ab630f616ef6e7e298fc06bc679ec4ccaf84c31612aefe70"}
Jul 08 12:06:07.708795 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:06:07.708722    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-678df5d84b-v29jh" event={"ID":"fe815282-50d9-4ebd-a970-ca9c70a92dd9","Type":"ContainerDied","Data":"878ebe5b0763e707d57b92a453df93479534c3955e942798d2f4ebe943931f40"}
Jul 08 12:06:07.708795 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:06:07.708747    2579 scope.go:117] "RemoveContainer" containerID="f11c8e3249b376e1ab630f616ef6e7e298fc06bc679ec4ccaf84c31612aefe70"
Jul 08 12:06:07.717114 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:06:07.717097    2579 scope.go:117] "RemoveContainer" containerID="f11c8e3249b376e1ab630f616ef6e7e298fc06bc679ec4ccaf84c31612aefe70"
Jul 08 12:06:07.717365 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:06:07.717348    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f11c8e3249b376e1ab630f616ef6e7e298fc06bc679ec4ccaf84c31612aefe70\": container with ID starting with f11c8e3249b376e1ab630f616ef6e7e298fc06bc679ec4ccaf84c31612aefe70 not found: ID does not exist" containerID="f11c8e3249b376e1ab630f616ef6e7e298fc06bc679ec4ccaf84c31612aefe70"
Jul 08 12:06:07.717431 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:06:07.717378    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f11c8e3249b376e1ab630f616ef6e7e298fc06bc679ec4ccaf84c31612aefe70"} err="failed to get container status \"f11c8e3249b376e1ab630f616ef6e7e298fc06bc679ec4ccaf84c31612aefe70\": rpc error: code = NotFound desc = could not find container \"f11c8e3249b376e1ab630f616ef6e7e298fc06bc679ec4ccaf84c31612aefe70\": container with ID starting with f11c8e3249b376e1ab630f616ef6e7e298fc06bc679ec4ccaf84c31612aefe70 not found: ID does not exist"
Jul 08 12:06:07.733086 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:06:07.733062    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-678df5d84b-v29jh"]
Jul 08 12:06:07.738618 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:06:07.738596    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-678df5d84b-v29jh"]
Jul 08 12:06:08.714431 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:06:08.714398    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe815282-50d9-4ebd-a970-ca9c70a92dd9" path="/var/lib/kubelet/pods/fe815282-50d9-4ebd-a970-ca9c70a92dd9/volumes"
Jul 08 12:06:28.698175 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:06:28.698090    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d5877f5d6-lnnmk"]
Jul 08 12:06:28.698630 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:06:28.698354    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe815282-50d9-4ebd-a970-ca9c70a92dd9" containerName="console"
Jul 08 12:06:28.698630 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:06:28.698364    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe815282-50d9-4ebd-a970-ca9c70a92dd9" containerName="console"
Jul 08 12:06:28.698630 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:06:28.698410    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe815282-50d9-4ebd-a970-ca9c70a92dd9" containerName="console"
Jul 08 12:06:28.701184 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:06:28.701161    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d5877f5d6-lnnmk"
Jul 08 12:06:28.703973 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:06:28.703945    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 08 12:06:28.705263 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:06:28.705241    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 08 12:06:28.705352 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:06:28.705241    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 08 12:06:28.705352 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:06:28.705299    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 08 12:06:28.705440 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:06:28.705377    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-sh2cn\""
Jul 08 12:06:28.715725 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:06:28.715702    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d5877f5d6-lnnmk"]
Jul 08 12:06:28.780729 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:06:28.780697    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r5l5\" (UniqueName: \"kubernetes.io/projected/871eb266-204e-42c9-9932-f0de83905051-kube-api-access-7r5l5\") pod \"managed-serviceaccount-addon-agent-7d5877f5d6-lnnmk\" (UID: \"871eb266-204e-42c9-9932-f0de83905051\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d5877f5d6-lnnmk"
Jul 08 12:06:28.780944 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:06:28.780752    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/871eb266-204e-42c9-9932-f0de83905051-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7d5877f5d6-lnnmk\" (UID: \"871eb266-204e-42c9-9932-f0de83905051\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d5877f5d6-lnnmk"
Jul 08 12:06:28.840333 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:06:28.840298    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-598646d8f8-s9l7q"]
Jul 08 12:06:28.843257 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:06:28.843241    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-598646d8f8-s9l7q"
Jul 08 12:06:28.846080 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:06:28.846060    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 08 12:06:28.857788 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:06:28.857750    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-598646d8f8-s9l7q"]
Jul 08 12:06:28.882159 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:06:28.882129    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r5l5\" (UniqueName: \"kubernetes.io/projected/871eb266-204e-42c9-9932-f0de83905051-kube-api-access-7r5l5\") pod \"managed-serviceaccount-addon-agent-7d5877f5d6-lnnmk\" (UID: \"871eb266-204e-42c9-9932-f0de83905051\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d5877f5d6-lnnmk"
Jul 08 12:06:28.882356 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:06:28.882194    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/871eb266-204e-42c9-9932-f0de83905051-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7d5877f5d6-lnnmk\" (UID: \"871eb266-204e-42c9-9932-f0de83905051\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d5877f5d6-lnnmk"
Jul 08 12:06:28.882356 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:06:28.882238    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb9fc752-f931-4340-85bd-7827752eff29-tmp\") pod \"klusterlet-addon-workmgr-598646d8f8-s9l7q\" (UID: \"bb9fc752-f931-4340-85bd-7827752eff29\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-598646d8f8-s9l7q"
Jul 08 12:06:28.882356 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:06:28.882277    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgxzq\" (UniqueName: \"kubernetes.io/projected/bb9fc752-f931-4340-85bd-7827752eff29-kube-api-access-vgxzq\") pod \"klusterlet-addon-workmgr-598646d8f8-s9l7q\" (UID: \"bb9fc752-f931-4340-85bd-7827752eff29\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-598646d8f8-s9l7q"
Jul 08 12:06:28.882356 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:06:28.882335    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/bb9fc752-f931-4340-85bd-7827752eff29-klusterlet-config\") pod \"klusterlet-addon-workmgr-598646d8f8-s9l7q\" (UID: \"bb9fc752-f931-4340-85bd-7827752eff29\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-598646d8f8-s9l7q"
Jul 08 12:06:28.884814 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:06:28.884796    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/871eb266-204e-42c9-9932-f0de83905051-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7d5877f5d6-lnnmk\" (UID: \"871eb266-204e-42c9-9932-f0de83905051\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d5877f5d6-lnnmk"
Jul 08 12:06:28.891336 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:06:28.891312    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r5l5\" (UniqueName: \"kubernetes.io/projected/871eb266-204e-42c9-9932-f0de83905051-kube-api-access-7r5l5\") pod \"managed-serviceaccount-addon-agent-7d5877f5d6-lnnmk\" (UID: \"871eb266-204e-42c9-9932-f0de83905051\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d5877f5d6-lnnmk"
Jul 08 12:06:28.983199 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:06:28.983110    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb9fc752-f931-4340-85bd-7827752eff29-tmp\") pod \"klusterlet-addon-workmgr-598646d8f8-s9l7q\" (UID: \"bb9fc752-f931-4340-85bd-7827752eff29\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-598646d8f8-s9l7q"
Jul 08 12:06:28.983199 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:06:28.983161    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgxzq\" (UniqueName: \"kubernetes.io/projected/bb9fc752-f931-4340-85bd-7827752eff29-kube-api-access-vgxzq\") pod \"klusterlet-addon-workmgr-598646d8f8-s9l7q\" (UID: \"bb9fc752-f931-4340-85bd-7827752eff29\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-598646d8f8-s9l7q"
Jul 08 12:06:28.983199 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:06:28.983184    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/bb9fc752-f931-4340-85bd-7827752eff29-klusterlet-config\") pod \"klusterlet-addon-workmgr-598646d8f8-s9l7q\" (UID: \"bb9fc752-f931-4340-85bd-7827752eff29\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-598646d8f8-s9l7q"
Jul 08 12:06:28.983488 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:06:28.983471    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb9fc752-f931-4340-85bd-7827752eff29-tmp\") pod \"klusterlet-addon-workmgr-598646d8f8-s9l7q\" (UID: \"bb9fc752-f931-4340-85bd-7827752eff29\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-598646d8f8-s9l7q"
Jul 08 12:06:28.985659 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:06:28.985641    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/bb9fc752-f931-4340-85bd-7827752eff29-klusterlet-config\") pod \"klusterlet-addon-workmgr-598646d8f8-s9l7q\" (UID: \"bb9fc752-f931-4340-85bd-7827752eff29\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-598646d8f8-s9l7q"
Jul 08 12:06:28.994555 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:06:28.994529    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgxzq\" (UniqueName: \"kubernetes.io/projected/bb9fc752-f931-4340-85bd-7827752eff29-kube-api-access-vgxzq\") pod \"klusterlet-addon-workmgr-598646d8f8-s9l7q\" (UID: \"bb9fc752-f931-4340-85bd-7827752eff29\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-598646d8f8-s9l7q"
Jul 08 12:06:29.023465 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:06:29.023432    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d5877f5d6-lnnmk"
Jul 08 12:06:29.151942 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:06:29.151907    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-598646d8f8-s9l7q"
Jul 08 12:06:29.155334 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:06:29.155314    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d5877f5d6-lnnmk"]
Jul 08 12:06:29.159972 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:06:29.159943    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod871eb266_204e_42c9_9932_f0de83905051.slice/crio-9d794ab2be71c5e40204a2b246d6dd0a8f7332a693aeeef5f7d7bbcd869be9df WatchSource:0}: Error finding container 9d794ab2be71c5e40204a2b246d6dd0a8f7332a693aeeef5f7d7bbcd869be9df: Status 404 returned error can't find the container with id 9d794ab2be71c5e40204a2b246d6dd0a8f7332a693aeeef5f7d7bbcd869be9df
Jul 08 12:06:29.283075 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:06:29.283002    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-598646d8f8-s9l7q"]
Jul 08 12:06:29.285556 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:06:29.285525    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb9fc752_f931_4340_85bd_7827752eff29.slice/crio-b1b654e56a06c0865c588c9339deb4deb17353052a846e95185a98757492e994 WatchSource:0}: Error finding container b1b654e56a06c0865c588c9339deb4deb17353052a846e95185a98757492e994: Status 404 returned error can't find the container with id b1b654e56a06c0865c588c9339deb4deb17353052a846e95185a98757492e994
Jul 08 12:06:29.783644 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:06:29.783605    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-598646d8f8-s9l7q" event={"ID":"bb9fc752-f931-4340-85bd-7827752eff29","Type":"ContainerStarted","Data":"b1b654e56a06c0865c588c9339deb4deb17353052a846e95185a98757492e994"}
Jul 08 12:06:29.784520 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:06:29.784498    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d5877f5d6-lnnmk" event={"ID":"871eb266-204e-42c9-9932-f0de83905051","Type":"ContainerStarted","Data":"9d794ab2be71c5e40204a2b246d6dd0a8f7332a693aeeef5f7d7bbcd869be9df"}
Jul 08 12:06:32.799255 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:06:32.799214    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d5877f5d6-lnnmk" event={"ID":"871eb266-204e-42c9-9932-f0de83905051","Type":"ContainerStarted","Data":"c02be27870e642eec7eb4878e07238f92c21be76174c70fde95e0dd09595575c"}
Jul 08 12:06:32.820690 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:06:32.820637    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d5877f5d6-lnnmk" podStartSLOduration=1.74851465 podStartE2EDuration="4.8206205s" totalImagesPullingTime="3.07210585s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 12:06:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 12:06:32.819851679 +0000 UTC m=+282.550226652" watchObservedRunningTime="2026-07-08 12:06:32.8206205 +0000 UTC m=+282.550995471"
Jul 08 12:06:33.804192 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:06:33.804095    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-598646d8f8-s9l7q" event={"ID":"bb9fc752-f931-4340-85bd-7827752eff29","Type":"ContainerStarted","Data":"34277fe08b280461f1c9a63c7addbc4e8f18dde9949e491afa900aacf605fb6f"}
Jul 08 12:06:33.804578 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:06:33.804244    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-598646d8f8-s9l7q"
Jul 08 12:06:33.805909 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:06:33.805890    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-598646d8f8-s9l7q"
Jul 08 12:06:33.823755 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:06:33.823700    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-598646d8f8-s9l7q" podStartSLOduration=1.5755959480000001 podStartE2EDuration="5.823685122s" totalImagesPullingTime="4.248089174s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 12:06:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 12:06:33.822351184 +0000 UTC m=+283.552726155" watchObservedRunningTime="2026-07-08 12:06:33.823685122 +0000 UTC m=+283.554060092"
Jul 08 12:06:50.637875 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:06:50.637849    2579 kubelet.go:1648] "Image garbage collection succeeded"
Jul 08 12:07:05.031275 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:05.031240    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cnc2b"]
Jul 08 12:07:05.035338 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:05.035320    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cnc2b"
Jul 08 12:07:05.039883 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:05.039856    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-q248f\""
Jul 08 12:07:05.040024 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:05.039859    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 08 12:07:05.040024 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:05.039867    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 08 12:07:05.049272 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:05.049243    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cnc2b"]
Jul 08 12:07:05.176886 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:05.176845    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbzbw\" (UniqueName: \"kubernetes.io/projected/711211a9-ef16-47f7-b5f5-aa14eeafac9f-kube-api-access-wbzbw\") pod \"openshift-lws-operator-fd8ccff4c-cnc2b\" (UID: \"711211a9-ef16-47f7-b5f5-aa14eeafac9f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cnc2b"
Jul 08 12:07:05.177057 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:05.176909    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/711211a9-ef16-47f7-b5f5-aa14eeafac9f-tmp\") pod \"openshift-lws-operator-fd8ccff4c-cnc2b\" (UID: \"711211a9-ef16-47f7-b5f5-aa14eeafac9f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cnc2b"
Jul 08 12:07:05.277739 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:05.277703    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbzbw\" (UniqueName: \"kubernetes.io/projected/711211a9-ef16-47f7-b5f5-aa14eeafac9f-kube-api-access-wbzbw\") pod \"openshift-lws-operator-fd8ccff4c-cnc2b\" (UID: \"711211a9-ef16-47f7-b5f5-aa14eeafac9f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cnc2b"
Jul 08 12:07:05.277867 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:05.277792    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/711211a9-ef16-47f7-b5f5-aa14eeafac9f-tmp\") pod \"openshift-lws-operator-fd8ccff4c-cnc2b\" (UID: \"711211a9-ef16-47f7-b5f5-aa14eeafac9f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cnc2b"
Jul 08 12:07:05.278217 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:05.278196    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/711211a9-ef16-47f7-b5f5-aa14eeafac9f-tmp\") pod \"openshift-lws-operator-fd8ccff4c-cnc2b\" (UID: \"711211a9-ef16-47f7-b5f5-aa14eeafac9f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cnc2b"
Jul 08 12:07:05.288326 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:05.288250    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbzbw\" (UniqueName: \"kubernetes.io/projected/711211a9-ef16-47f7-b5f5-aa14eeafac9f-kube-api-access-wbzbw\") pod \"openshift-lws-operator-fd8ccff4c-cnc2b\" (UID: \"711211a9-ef16-47f7-b5f5-aa14eeafac9f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cnc2b"
Jul 08 12:07:05.344299 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:05.344253    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cnc2b"
Jul 08 12:07:05.479418 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:05.479372    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cnc2b"]
Jul 08 12:07:05.482327 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:07:05.482287    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod711211a9_ef16_47f7_b5f5_aa14eeafac9f.slice/crio-5c1a2b3dd6e25ec7a3d98dfe596ec3e2179291af31d6a15c8fed155c509eb557 WatchSource:0}: Error finding container 5c1a2b3dd6e25ec7a3d98dfe596ec3e2179291af31d6a15c8fed155c509eb557: Status 404 returned error can't find the container with id 5c1a2b3dd6e25ec7a3d98dfe596ec3e2179291af31d6a15c8fed155c509eb557
Jul 08 12:07:05.483898 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:05.483875    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 08 12:07:05.919391 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:05.919355    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cnc2b" event={"ID":"711211a9-ef16-47f7-b5f5-aa14eeafac9f","Type":"ContainerStarted","Data":"5c1a2b3dd6e25ec7a3d98dfe596ec3e2179291af31d6a15c8fed155c509eb557"}
Jul 08 12:07:08.933752 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:08.933716    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cnc2b" event={"ID":"711211a9-ef16-47f7-b5f5-aa14eeafac9f","Type":"ContainerStarted","Data":"24c0f34a5d76404edf43e1d37fbe4c7833985249453056b8290d443f0c616996"}
Jul 08 12:07:08.960518 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:08.960451    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cnc2b" podStartSLOduration=1.272837568 podStartE2EDuration="3.960430444s" totalImagesPullingTime="2.687592876s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 12:07:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 12:07:08.958928356 +0000 UTC m=+318.689303326" watchObservedRunningTime="2026-07-08 12:07:08.960430444 +0000 UTC m=+318.690805416"
Jul 08 12:07:27.872616 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:27.872563    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6db55b68b6-wrjvx"]
Jul 08 12:07:27.874732 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:27.874710    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6db55b68b6-wrjvx"
Jul 08 12:07:27.878562 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:27.878539    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 08 12:07:27.878707 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:27.878580    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 08 12:07:27.879006 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:27.878987    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-bp9fb\""
Jul 08 12:07:27.879006 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:27.878999    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 08 12:07:27.879252 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:27.879237    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 08 12:07:27.901392 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:27.901304    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6db55b68b6-wrjvx"]
Jul 08 12:07:27.955899 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:27.955867    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b083fd94-85f6-42f9-a56b-ed83d1e2c629-tmp\") pod \"opendatahub-operator-controller-manager-6db55b68b6-wrjvx\" (UID: \"b083fd94-85f6-42f9-a56b-ed83d1e2c629\") " pod="opendatahub/opendatahub-operator-controller-manager-6db55b68b6-wrjvx"
Jul 08 12:07:27.955899 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:27.955903    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jctd9\" (UniqueName: \"kubernetes.io/projected/b083fd94-85f6-42f9-a56b-ed83d1e2c629-kube-api-access-jctd9\") pod \"opendatahub-operator-controller-manager-6db55b68b6-wrjvx\" (UID: \"b083fd94-85f6-42f9-a56b-ed83d1e2c629\") " pod="opendatahub/opendatahub-operator-controller-manager-6db55b68b6-wrjvx"
Jul 08 12:07:27.956101 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:27.955930    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b083fd94-85f6-42f9-a56b-ed83d1e2c629-webhook-cert\") pod \"opendatahub-operator-controller-manager-6db55b68b6-wrjvx\" (UID: \"b083fd94-85f6-42f9-a56b-ed83d1e2c629\") " pod="opendatahub/opendatahub-operator-controller-manager-6db55b68b6-wrjvx"
Jul 08 12:07:27.956101 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:27.956001    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b083fd94-85f6-42f9-a56b-ed83d1e2c629-manifests\") pod \"opendatahub-operator-controller-manager-6db55b68b6-wrjvx\" (UID: \"b083fd94-85f6-42f9-a56b-ed83d1e2c629\") " pod="opendatahub/opendatahub-operator-controller-manager-6db55b68b6-wrjvx"
Jul 08 12:07:27.956101 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:27.956055    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b083fd94-85f6-42f9-a56b-ed83d1e2c629-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6db55b68b6-wrjvx\" (UID: \"b083fd94-85f6-42f9-a56b-ed83d1e2c629\") " pod="opendatahub/opendatahub-operator-controller-manager-6db55b68b6-wrjvx"
Jul 08 12:07:28.056645 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:28.056601    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b083fd94-85f6-42f9-a56b-ed83d1e2c629-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6db55b68b6-wrjvx\" (UID: \"b083fd94-85f6-42f9-a56b-ed83d1e2c629\") " pod="opendatahub/opendatahub-operator-controller-manager-6db55b68b6-wrjvx"
Jul 08 12:07:28.056881 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:28.056672    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b083fd94-85f6-42f9-a56b-ed83d1e2c629-tmp\") pod \"opendatahub-operator-controller-manager-6db55b68b6-wrjvx\" (UID: \"b083fd94-85f6-42f9-a56b-ed83d1e2c629\") " pod="opendatahub/opendatahub-operator-controller-manager-6db55b68b6-wrjvx"
Jul 08 12:07:28.056881 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:28.056702    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jctd9\" (UniqueName: \"kubernetes.io/projected/b083fd94-85f6-42f9-a56b-ed83d1e2c629-kube-api-access-jctd9\") pod \"opendatahub-operator-controller-manager-6db55b68b6-wrjvx\" (UID: \"b083fd94-85f6-42f9-a56b-ed83d1e2c629\") " pod="opendatahub/opendatahub-operator-controller-manager-6db55b68b6-wrjvx"
Jul 08 12:07:28.056881 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:28.056729    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b083fd94-85f6-42f9-a56b-ed83d1e2c629-webhook-cert\") pod \"opendatahub-operator-controller-manager-6db55b68b6-wrjvx\" (UID: \"b083fd94-85f6-42f9-a56b-ed83d1e2c629\") " pod="opendatahub/opendatahub-operator-controller-manager-6db55b68b6-wrjvx"
Jul 08 12:07:28.056881 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:28.056796    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b083fd94-85f6-42f9-a56b-ed83d1e2c629-manifests\") pod \"opendatahub-operator-controller-manager-6db55b68b6-wrjvx\" (UID: \"b083fd94-85f6-42f9-a56b-ed83d1e2c629\") " pod="opendatahub/opendatahub-operator-controller-manager-6db55b68b6-wrjvx"
Jul 08 12:07:28.057210 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:28.057188    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b083fd94-85f6-42f9-a56b-ed83d1e2c629-manifests\") pod \"opendatahub-operator-controller-manager-6db55b68b6-wrjvx\" (UID: \"b083fd94-85f6-42f9-a56b-ed83d1e2c629\") " pod="opendatahub/opendatahub-operator-controller-manager-6db55b68b6-wrjvx"
Jul 08 12:07:28.059333 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:28.059298    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b083fd94-85f6-42f9-a56b-ed83d1e2c629-tmp\") pod \"opendatahub-operator-controller-manager-6db55b68b6-wrjvx\" (UID: \"b083fd94-85f6-42f9-a56b-ed83d1e2c629\") " pod="opendatahub/opendatahub-operator-controller-manager-6db55b68b6-wrjvx"
Jul 08 12:07:28.059445 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:28.059378    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b083fd94-85f6-42f9-a56b-ed83d1e2c629-webhook-cert\") pod \"opendatahub-operator-controller-manager-6db55b68b6-wrjvx\" (UID: \"b083fd94-85f6-42f9-a56b-ed83d1e2c629\") " pod="opendatahub/opendatahub-operator-controller-manager-6db55b68b6-wrjvx"
Jul 08 12:07:28.059445 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:28.059394    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b083fd94-85f6-42f9-a56b-ed83d1e2c629-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6db55b68b6-wrjvx\" (UID: \"b083fd94-85f6-42f9-a56b-ed83d1e2c629\") " pod="opendatahub/opendatahub-operator-controller-manager-6db55b68b6-wrjvx"
Jul 08 12:07:28.071973 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:28.071945    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jctd9\" (UniqueName: \"kubernetes.io/projected/b083fd94-85f6-42f9-a56b-ed83d1e2c629-kube-api-access-jctd9\") pod \"opendatahub-operator-controller-manager-6db55b68b6-wrjvx\" (UID: \"b083fd94-85f6-42f9-a56b-ed83d1e2c629\") " pod="opendatahub/opendatahub-operator-controller-manager-6db55b68b6-wrjvx"
Jul 08 12:07:28.185408 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:28.185363    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6db55b68b6-wrjvx"
Jul 08 12:07:28.345038 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:28.345009    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6db55b68b6-wrjvx"]
Jul 08 12:07:28.347117 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:07:28.347088    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb083fd94_85f6_42f9_a56b_ed83d1e2c629.slice/crio-615b1c3e76c07e0ddb1e3f356d1c029da2188ca019e28b744603dd8d84b3e2c8 WatchSource:0}: Error finding container 615b1c3e76c07e0ddb1e3f356d1c029da2188ca019e28b744603dd8d84b3e2c8: Status 404 returned error can't find the container with id 615b1c3e76c07e0ddb1e3f356d1c029da2188ca019e28b744603dd8d84b3e2c8
Jul 08 12:07:29.009523 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:29.009448    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6db55b68b6-wrjvx" event={"ID":"b083fd94-85f6-42f9-a56b-ed83d1e2c629","Type":"ContainerStarted","Data":"615b1c3e76c07e0ddb1e3f356d1c029da2188ca019e28b744603dd8d84b3e2c8"}
Jul 08 12:07:32.024413 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:32.024376    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6db55b68b6-wrjvx" event={"ID":"b083fd94-85f6-42f9-a56b-ed83d1e2c629","Type":"ContainerStarted","Data":"5c6d98421b1f8c03eba9b67499eccb012751401cf250d14caabd3c1c2c188728"}
Jul 08 12:07:33.030637 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:33.030596    2579 generic.go:356] "Generic (PLEG): container finished" podID="b083fd94-85f6-42f9-a56b-ed83d1e2c629" containerID="5c6d98421b1f8c03eba9b67499eccb012751401cf250d14caabd3c1c2c188728" exitCode=0
Jul 08 12:07:33.031043 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:33.030686    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6db55b68b6-wrjvx" event={"ID":"b083fd94-85f6-42f9-a56b-ed83d1e2c629","Type":"ContainerDied","Data":"5c6d98421b1f8c03eba9b67499eccb012751401cf250d14caabd3c1c2c188728"}
Jul 08 12:07:34.036216 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:34.036183    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6db55b68b6-wrjvx" event={"ID":"b083fd94-85f6-42f9-a56b-ed83d1e2c629","Type":"ContainerStarted","Data":"9a74a3389e7ec84234b53a8cf429476183c697fd79ce88064e05e91660e32293"}
Jul 08 12:07:34.036696 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:34.036303    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6db55b68b6-wrjvx"
Jul 08 12:07:34.068072 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:34.068007    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6db55b68b6-wrjvx" podStartSLOduration=2.927862773 podStartE2EDuration="7.067988726s" totalImagesPullingTime="3.131316369s" totalInitContainerRuntime="1.008809584s" isStatefulPod=false podCreationTimestamp="2026-07-08 12:07:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 12:07:34.065312788 +0000 UTC m=+343.795687758" watchObservedRunningTime="2026-07-08 12:07:34.067988726 +0000 UTC m=+343.798363747"
Jul 08 12:07:45.042656 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:45.042580    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6db55b68b6-wrjvx"
Jul 08 12:07:48.643350 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:48.643320    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-765c5b48c6-kq8lm"]
Jul 08 12:07:48.647338 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:48.647314    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-765c5b48c6-kq8lm"
Jul 08 12:07:48.650877 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:48.650855    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-nkkbn\""
Jul 08 12:07:48.650995 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:48.650883    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 08 12:07:48.651455 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:48.651443    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 08 12:07:48.668665 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:48.668629    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-765c5b48c6-kq8lm"]
Jul 08 12:07:48.722495 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:48.722468    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fcb07119-dbca-41f5-98ee-bdcfef917bd3-tmp\") pod \"kube-auth-proxy-765c5b48c6-kq8lm\" (UID: \"fcb07119-dbca-41f5-98ee-bdcfef917bd3\") " pod="openshift-ingress/kube-auth-proxy-765c5b48c6-kq8lm"
Jul 08 12:07:48.722661 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:48.722513    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcb07119-dbca-41f5-98ee-bdcfef917bd3-tls-certs\") pod \"kube-auth-proxy-765c5b48c6-kq8lm\" (UID: \"fcb07119-dbca-41f5-98ee-bdcfef917bd3\") " pod="openshift-ingress/kube-auth-proxy-765c5b48c6-kq8lm"
Jul 08 12:07:48.722661 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:48.722539    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2spbw\" (UniqueName: \"kubernetes.io/projected/fcb07119-dbca-41f5-98ee-bdcfef917bd3-kube-api-access-2spbw\") pod \"kube-auth-proxy-765c5b48c6-kq8lm\" (UID: \"fcb07119-dbca-41f5-98ee-bdcfef917bd3\") " pod="openshift-ingress/kube-auth-proxy-765c5b48c6-kq8lm"
Jul 08 12:07:48.824060 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:48.824021    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fcb07119-dbca-41f5-98ee-bdcfef917bd3-tmp\") pod \"kube-auth-proxy-765c5b48c6-kq8lm\" (UID: \"fcb07119-dbca-41f5-98ee-bdcfef917bd3\") " pod="openshift-ingress/kube-auth-proxy-765c5b48c6-kq8lm"
Jul 08 12:07:48.824238 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:48.824070    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcb07119-dbca-41f5-98ee-bdcfef917bd3-tls-certs\") pod \"kube-auth-proxy-765c5b48c6-kq8lm\" (UID: \"fcb07119-dbca-41f5-98ee-bdcfef917bd3\") " pod="openshift-ingress/kube-auth-proxy-765c5b48c6-kq8lm"
Jul 08 12:07:48.824238 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:48.824091    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2spbw\" (UniqueName: \"kubernetes.io/projected/fcb07119-dbca-41f5-98ee-bdcfef917bd3-kube-api-access-2spbw\") pod \"kube-auth-proxy-765c5b48c6-kq8lm\" (UID: \"fcb07119-dbca-41f5-98ee-bdcfef917bd3\") " pod="openshift-ingress/kube-auth-proxy-765c5b48c6-kq8lm"
Jul 08 12:07:48.824238 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:07:48.824207    2579 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Jul 08 12:07:48.824347 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:07:48.824282    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcb07119-dbca-41f5-98ee-bdcfef917bd3-tls-certs podName:fcb07119-dbca-41f5-98ee-bdcfef917bd3 nodeName:}" failed. No retries permitted until 2026-07-08 12:07:49.324266395 +0000 UTC m=+359.054641348 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/fcb07119-dbca-41f5-98ee-bdcfef917bd3-tls-certs") pod "kube-auth-proxy-765c5b48c6-kq8lm" (UID: "fcb07119-dbca-41f5-98ee-bdcfef917bd3") : secret "kube-auth-proxy-tls" not found
Jul 08 12:07:48.826324 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:48.826300    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fcb07119-dbca-41f5-98ee-bdcfef917bd3-tmp\") pod \"kube-auth-proxy-765c5b48c6-kq8lm\" (UID: \"fcb07119-dbca-41f5-98ee-bdcfef917bd3\") " pod="openshift-ingress/kube-auth-proxy-765c5b48c6-kq8lm"
Jul 08 12:07:48.858455 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:48.858419    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2spbw\" (UniqueName: \"kubernetes.io/projected/fcb07119-dbca-41f5-98ee-bdcfef917bd3-kube-api-access-2spbw\") pod \"kube-auth-proxy-765c5b48c6-kq8lm\" (UID: \"fcb07119-dbca-41f5-98ee-bdcfef917bd3\") " pod="openshift-ingress/kube-auth-proxy-765c5b48c6-kq8lm"
Jul 08 12:07:49.327209 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:49.327166    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcb07119-dbca-41f5-98ee-bdcfef917bd3-tls-certs\") pod \"kube-auth-proxy-765c5b48c6-kq8lm\" (UID: \"fcb07119-dbca-41f5-98ee-bdcfef917bd3\") " pod="openshift-ingress/kube-auth-proxy-765c5b48c6-kq8lm"
Jul 08 12:07:49.329623 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:49.329592    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcb07119-dbca-41f5-98ee-bdcfef917bd3-tls-certs\") pod \"kube-auth-proxy-765c5b48c6-kq8lm\" (UID: \"fcb07119-dbca-41f5-98ee-bdcfef917bd3\") " pod="openshift-ingress/kube-auth-proxy-765c5b48c6-kq8lm"
Jul 08 12:07:49.557927 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:49.557881    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-765c5b48c6-kq8lm"
Jul 08 12:07:49.710072 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:49.710045    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-765c5b48c6-kq8lm"]
Jul 08 12:07:49.712732 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:07:49.712700    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcb07119_dbca_41f5_98ee_bdcfef917bd3.slice/crio-a02cba545bbb7a27309339028b9f7079ab6e7e04fd3b05e3d0df390683b573e8 WatchSource:0}: Error finding container a02cba545bbb7a27309339028b9f7079ab6e7e04fd3b05e3d0df390683b573e8: Status 404 returned error can't find the container with id a02cba545bbb7a27309339028b9f7079ab6e7e04fd3b05e3d0df390683b573e8
Jul 08 12:07:50.100740 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:50.100706    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-765c5b48c6-kq8lm" event={"ID":"fcb07119-dbca-41f5-98ee-bdcfef917bd3","Type":"ContainerStarted","Data":"a02cba545bbb7a27309339028b9f7079ab6e7e04fd3b05e3d0df390683b573e8"}
Jul 08 12:07:51.755051 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:51.755017    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-n8c8q"]
Jul 08 12:07:51.758962 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:51.758937    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-n8c8q"
Jul 08 12:07:51.762449 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:51.762418    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 08 12:07:51.762598 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:51.762526    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 08 12:07:51.762831 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:51.762816    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-f2z9h\""
Jul 08 12:07:51.763033 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:51.763020    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 08 12:07:51.763849 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:51.763832    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 08 12:07:51.783253 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:51.783215    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-n8c8q"]
Jul 08 12:07:51.847412 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:51.847365    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1ff2d17c-0150-46f1-bde0-43d0770da3f9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-n8c8q\" (UID: \"1ff2d17c-0150-46f1-bde0-43d0770da3f9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-n8c8q"
Jul 08 12:07:51.847626 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:51.847455    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ff2d17c-0150-46f1-bde0-43d0770da3f9-local-certs\") pod \"istiod-openshift-gateway-7574459656-n8c8q\" (UID: \"1ff2d17c-0150-46f1-bde0-43d0770da3f9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-n8c8q"
Jul 08 12:07:51.847626 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:51.847516    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1ff2d17c-0150-46f1-bde0-43d0770da3f9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-n8c8q\" (UID: \"1ff2d17c-0150-46f1-bde0-43d0770da3f9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-n8c8q"
Jul 08 12:07:51.847626 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:51.847553    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1ff2d17c-0150-46f1-bde0-43d0770da3f9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-n8c8q\" (UID: \"1ff2d17c-0150-46f1-bde0-43d0770da3f9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-n8c8q"
Jul 08 12:07:51.847626 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:51.847579    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1ff2d17c-0150-46f1-bde0-43d0770da3f9-cacerts\") pod \"istiod-openshift-gateway-7574459656-n8c8q\" (UID: \"1ff2d17c-0150-46f1-bde0-43d0770da3f9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-n8c8q"
Jul 08 12:07:51.847626 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:51.847613    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dkdf\" (UniqueName: \"kubernetes.io/projected/1ff2d17c-0150-46f1-bde0-43d0770da3f9-kube-api-access-7dkdf\") pod \"istiod-openshift-gateway-7574459656-n8c8q\" (UID: \"1ff2d17c-0150-46f1-bde0-43d0770da3f9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-n8c8q"
Jul 08 12:07:51.847894 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:51.847705    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ff2d17c-0150-46f1-bde0-43d0770da3f9-istio-token\") pod \"istiod-openshift-gateway-7574459656-n8c8q\" (UID: \"1ff2d17c-0150-46f1-bde0-43d0770da3f9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-n8c8q"
Jul 08 12:07:51.949140 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:51.949107    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ff2d17c-0150-46f1-bde0-43d0770da3f9-istio-token\") pod \"istiod-openshift-gateway-7574459656-n8c8q\" (UID: \"1ff2d17c-0150-46f1-bde0-43d0770da3f9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-n8c8q"
Jul 08 12:07:51.949140 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:51.949152    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1ff2d17c-0150-46f1-bde0-43d0770da3f9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-n8c8q\" (UID: \"1ff2d17c-0150-46f1-bde0-43d0770da3f9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-n8c8q"
Jul 08 12:07:51.949397 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:51.949170    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ff2d17c-0150-46f1-bde0-43d0770da3f9-local-certs\") pod \"istiod-openshift-gateway-7574459656-n8c8q\" (UID: \"1ff2d17c-0150-46f1-bde0-43d0770da3f9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-n8c8q"
Jul 08 12:07:51.949397 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:51.949212    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1ff2d17c-0150-46f1-bde0-43d0770da3f9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-n8c8q\" (UID: \"1ff2d17c-0150-46f1-bde0-43d0770da3f9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-n8c8q"
Jul 08 12:07:51.949397 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:51.949244    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1ff2d17c-0150-46f1-bde0-43d0770da3f9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-n8c8q\" (UID: \"1ff2d17c-0150-46f1-bde0-43d0770da3f9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-n8c8q"
Jul 08 12:07:51.949397 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:51.949269    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1ff2d17c-0150-46f1-bde0-43d0770da3f9-cacerts\") pod \"istiod-openshift-gateway-7574459656-n8c8q\" (UID: \"1ff2d17c-0150-46f1-bde0-43d0770da3f9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-n8c8q"
Jul 08 12:07:51.949397 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:51.949330    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dkdf\" (UniqueName: \"kubernetes.io/projected/1ff2d17c-0150-46f1-bde0-43d0770da3f9-kube-api-access-7dkdf\") pod \"istiod-openshift-gateway-7574459656-n8c8q\" (UID: \"1ff2d17c-0150-46f1-bde0-43d0770da3f9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-n8c8q"
Jul 08 12:07:51.949993 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:51.949965    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1ff2d17c-0150-46f1-bde0-43d0770da3f9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-n8c8q\" (UID: \"1ff2d17c-0150-46f1-bde0-43d0770da3f9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-n8c8q"
Jul 08 12:07:51.952285 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:51.952255    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1ff2d17c-0150-46f1-bde0-43d0770da3f9-cacerts\") pod \"istiod-openshift-gateway-7574459656-n8c8q\" (UID: \"1ff2d17c-0150-46f1-bde0-43d0770da3f9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-n8c8q"
Jul 08 12:07:51.952453 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:51.952298    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1ff2d17c-0150-46f1-bde0-43d0770da3f9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-n8c8q\" (UID: \"1ff2d17c-0150-46f1-bde0-43d0770da3f9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-n8c8q"
Jul 08 12:07:51.952453 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:51.952320    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ff2d17c-0150-46f1-bde0-43d0770da3f9-local-certs\") pod \"istiod-openshift-gateway-7574459656-n8c8q\" (UID: \"1ff2d17c-0150-46f1-bde0-43d0770da3f9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-n8c8q"
Jul 08 12:07:51.952595 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:51.952560    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1ff2d17c-0150-46f1-bde0-43d0770da3f9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-n8c8q\" (UID: \"1ff2d17c-0150-46f1-bde0-43d0770da3f9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-n8c8q"
Jul 08 12:07:51.966024 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:51.965993    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dkdf\" (UniqueName: \"kubernetes.io/projected/1ff2d17c-0150-46f1-bde0-43d0770da3f9-kube-api-access-7dkdf\") pod \"istiod-openshift-gateway-7574459656-n8c8q\" (UID: \"1ff2d17c-0150-46f1-bde0-43d0770da3f9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-n8c8q"
Jul 08 12:07:51.966382 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:51.966363    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ff2d17c-0150-46f1-bde0-43d0770da3f9-istio-token\") pod \"istiod-openshift-gateway-7574459656-n8c8q\" (UID: \"1ff2d17c-0150-46f1-bde0-43d0770da3f9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-n8c8q"
Jul 08 12:07:52.067572 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:52.067470    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-n8c8q"
Jul 08 12:07:52.237631 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:52.237597    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-n8c8q"]
Jul 08 12:07:52.718393 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:07:52.718364    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ff2d17c_0150_46f1_bde0_43d0770da3f9.slice/crio-596a2ecba7e764b23fc30a6be43a808cf8c5d3d68e2338e334fe9e33de1cdbdc WatchSource:0}: Error finding container 596a2ecba7e764b23fc30a6be43a808cf8c5d3d68e2338e334fe9e33de1cdbdc: Status 404 returned error can't find the container with id 596a2ecba7e764b23fc30a6be43a808cf8c5d3d68e2338e334fe9e33de1cdbdc
Jul 08 12:07:52.752893 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:52.752870    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 08 12:07:53.116847 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:53.116728    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-n8c8q" event={"ID":"1ff2d17c-0150-46f1-bde0-43d0770da3f9","Type":"ContainerStarted","Data":"596a2ecba7e764b23fc30a6be43a808cf8c5d3d68e2338e334fe9e33de1cdbdc"}
Jul 08 12:07:53.118158 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:53.118130    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-765c5b48c6-kq8lm" event={"ID":"fcb07119-dbca-41f5-98ee-bdcfef917bd3","Type":"ContainerStarted","Data":"29287ac44830d550db075edea28c7a82c4a73f21b789ed168d9773af296e030e"}
Jul 08 12:07:53.152433 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:53.152382    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-765c5b48c6-kq8lm" podStartSLOduration=2.116707715 podStartE2EDuration="5.15236537s" totalImagesPullingTime="3.035657655s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 12:07:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 12:07:53.150942021 +0000 UTC m=+362.881317035" watchObservedRunningTime="2026-07-08 12:07:53.15236537 +0000 UTC m=+362.882740340"
Jul 08 12:07:55.041917 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:55.041872    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 08 12:07:55.042308 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:55.041959    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 08 12:07:55.128322 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:55.128288    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-n8c8q" event={"ID":"1ff2d17c-0150-46f1-bde0-43d0770da3f9","Type":"ContainerStarted","Data":"036f245a4ee4fb0a80e1f8d98a49a67bef93f2a726250a993c2a4f22b4ad586f"}
Jul 08 12:07:55.128480 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:55.128382    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-n8c8q"
Jul 08 12:07:55.160985 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:55.160935    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-n8c8q" podStartSLOduration=1.839758458 podStartE2EDuration="4.16091877s" totalImagesPullingTime="2.321160312s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 12:07:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 12:07:55.159461154 +0000 UTC m=+364.889836125" watchObservedRunningTime="2026-07-08 12:07:55.16091877 +0000 UTC m=+364.891293740"
Jul 08 12:07:56.135164 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:56.135116    2579 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-n8c8q container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 08 12:07:56.135567 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:56.135174    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-n8c8q" podUID="1ff2d17c-0150-46f1-bde0-43d0770da3f9" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 12:07:59.134472 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:07:59.134443    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-n8c8q"
Jul 08 12:08:01.970127 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:01.970086    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-9qwx5"]
Jul 08 12:08:01.973707 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:01.973681    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-9qwx5"
Jul 08 12:08:01.978204 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:01.978183    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Jul 08 12:08:01.978204 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:01.978184    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-tttpg\""
Jul 08 12:08:01.981924 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:01.981901    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-2g5zq"]
Jul 08 12:08:01.984995 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:01.984975    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2g5zq"
Jul 08 12:08:01.988569 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:01.988547    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Jul 08 12:08:01.988679 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:01.988553    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-qqzth\""
Jul 08 12:08:01.991291 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:01.991203    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-9qwx5"]
Jul 08 12:08:01.997967 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:01.997936    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-2g5zq"]
Jul 08 12:08:02.041640 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:02.041603    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhstb\" (UniqueName: \"kubernetes.io/projected/19452e60-a504-485d-bf01-e0acffd5f18c-kube-api-access-zhstb\") pod \"kserve-controller-manager-d6f4b65b8-9qwx5\" (UID: \"19452e60-a504-485d-bf01-e0acffd5f18c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-9qwx5"
Jul 08 12:08:02.041847 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:02.041661    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2r5k\" (UniqueName: \"kubernetes.io/projected/8317baa3-d235-47f8-94c4-308c96c6fb2a-kube-api-access-p2r5k\") pod \"llmisvc-controller-manager-777c6bd7d5-2g5zq\" (UID: \"8317baa3-d235-47f8-94c4-308c96c6fb2a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2g5zq"
Jul 08 12:08:02.041847 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:02.041697    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/19452e60-a504-485d-bf01-e0acffd5f18c-cert\") pod \"kserve-controller-manager-d6f4b65b8-9qwx5\" (UID: \"19452e60-a504-485d-bf01-e0acffd5f18c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-9qwx5"
Jul 08 12:08:02.041847 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:02.041797    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8317baa3-d235-47f8-94c4-308c96c6fb2a-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-2g5zq\" (UID: \"8317baa3-d235-47f8-94c4-308c96c6fb2a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2g5zq"
Jul 08 12:08:02.143196 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:02.143154    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8317baa3-d235-47f8-94c4-308c96c6fb2a-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-2g5zq\" (UID: \"8317baa3-d235-47f8-94c4-308c96c6fb2a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2g5zq"
Jul 08 12:08:02.143400 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:02.143209    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhstb\" (UniqueName: \"kubernetes.io/projected/19452e60-a504-485d-bf01-e0acffd5f18c-kube-api-access-zhstb\") pod \"kserve-controller-manager-d6f4b65b8-9qwx5\" (UID: \"19452e60-a504-485d-bf01-e0acffd5f18c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-9qwx5"
Jul 08 12:08:02.143400 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:02.143243    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2r5k\" (UniqueName: \"kubernetes.io/projected/8317baa3-d235-47f8-94c4-308c96c6fb2a-kube-api-access-p2r5k\") pod \"llmisvc-controller-manager-777c6bd7d5-2g5zq\" (UID: \"8317baa3-d235-47f8-94c4-308c96c6fb2a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2g5zq"
Jul 08 12:08:02.143400 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:02.143257    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/19452e60-a504-485d-bf01-e0acffd5f18c-cert\") pod \"kserve-controller-manager-d6f4b65b8-9qwx5\" (UID: \"19452e60-a504-485d-bf01-e0acffd5f18c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-9qwx5"
Jul 08 12:08:02.143400 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:08:02.143317    2579 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Jul 08 12:08:02.143400 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:08:02.143386    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8317baa3-d235-47f8-94c4-308c96c6fb2a-cert podName:8317baa3-d235-47f8-94c4-308c96c6fb2a nodeName:}" failed. No retries permitted until 2026-07-08 12:08:02.643369413 +0000 UTC m=+372.373744363 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/8317baa3-d235-47f8-94c4-308c96c6fb2a-cert") pod "llmisvc-controller-manager-777c6bd7d5-2g5zq" (UID: "8317baa3-d235-47f8-94c4-308c96c6fb2a") : secret "llmisvc-webhook-server-cert" not found
Jul 08 12:08:02.145980 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:02.145952    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/19452e60-a504-485d-bf01-e0acffd5f18c-cert\") pod \"kserve-controller-manager-d6f4b65b8-9qwx5\" (UID: \"19452e60-a504-485d-bf01-e0acffd5f18c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-9qwx5"
Jul 08 12:08:02.163453 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:02.163416    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhstb\" (UniqueName: \"kubernetes.io/projected/19452e60-a504-485d-bf01-e0acffd5f18c-kube-api-access-zhstb\") pod \"kserve-controller-manager-d6f4b65b8-9qwx5\" (UID: \"19452e60-a504-485d-bf01-e0acffd5f18c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-9qwx5"
Jul 08 12:08:02.163645 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:02.163620    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2r5k\" (UniqueName: \"kubernetes.io/projected/8317baa3-d235-47f8-94c4-308c96c6fb2a-kube-api-access-p2r5k\") pod \"llmisvc-controller-manager-777c6bd7d5-2g5zq\" (UID: \"8317baa3-d235-47f8-94c4-308c96c6fb2a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2g5zq"
Jul 08 12:08:02.285025 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:02.284928    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-9qwx5"
Jul 08 12:08:02.453758 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:02.453722    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-9qwx5"]
Jul 08 12:08:02.456619 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:08:02.456591    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19452e60_a504_485d_bf01_e0acffd5f18c.slice/crio-eb4edccd7f5aa4edf066cd60f67c958ac212d93e35703add556ec7730c1a6f95 WatchSource:0}: Error finding container eb4edccd7f5aa4edf066cd60f67c958ac212d93e35703add556ec7730c1a6f95: Status 404 returned error can't find the container with id eb4edccd7f5aa4edf066cd60f67c958ac212d93e35703add556ec7730c1a6f95
Jul 08 12:08:02.648687 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:02.648595    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8317baa3-d235-47f8-94c4-308c96c6fb2a-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-2g5zq\" (UID: \"8317baa3-d235-47f8-94c4-308c96c6fb2a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2g5zq"
Jul 08 12:08:02.651205 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:02.651183    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8317baa3-d235-47f8-94c4-308c96c6fb2a-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-2g5zq\" (UID: \"8317baa3-d235-47f8-94c4-308c96c6fb2a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2g5zq"
Jul 08 12:08:02.895271 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:02.895230    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2g5zq"
Jul 08 12:08:03.041316 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:03.041274    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-2g5zq"]
Jul 08 12:08:03.045070 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:08:03.045040    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8317baa3_d235_47f8_94c4_308c96c6fb2a.slice/crio-a211b78772113a709063f72ba527ec42a57b048c5716295ed4c872d0bb53e054 WatchSource:0}: Error finding container a211b78772113a709063f72ba527ec42a57b048c5716295ed4c872d0bb53e054: Status 404 returned error can't find the container with id a211b78772113a709063f72ba527ec42a57b048c5716295ed4c872d0bb53e054
Jul 08 12:08:03.161088 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:03.161004    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2g5zq" event={"ID":"8317baa3-d235-47f8-94c4-308c96c6fb2a","Type":"ContainerStarted","Data":"a211b78772113a709063f72ba527ec42a57b048c5716295ed4c872d0bb53e054"}
Jul 08 12:08:03.162092 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:03.162071    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-9qwx5" event={"ID":"19452e60-a504-485d-bf01-e0acffd5f18c","Type":"ContainerStarted","Data":"eb4edccd7f5aa4edf066cd60f67c958ac212d93e35703add556ec7730c1a6f95"}
Jul 08 12:08:04.503127 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:04.503091    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-n8c8q"]
Jul 08 12:08:04.503618 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:04.503334    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-n8c8q" podUID="1ff2d17c-0150-46f1-bde0-43d0770da3f9" containerName="discovery" containerID="cri-o://036f245a4ee4fb0a80e1f8d98a49a67bef93f2a726250a993c2a4f22b4ad586f" gracePeriod=30
Jul 08 12:08:04.896264 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:04.896233    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-n8c8q"
Jul 08 12:08:04.971252 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:04.971217    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1ff2d17c-0150-46f1-bde0-43d0770da3f9-istio-kubeconfig\") pod \"1ff2d17c-0150-46f1-bde0-43d0770da3f9\" (UID: \"1ff2d17c-0150-46f1-bde0-43d0770da3f9\") "
Jul 08 12:08:04.971434 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:04.971278    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1ff2d17c-0150-46f1-bde0-43d0770da3f9-istio-csr-ca-configmap\") pod \"1ff2d17c-0150-46f1-bde0-43d0770da3f9\" (UID: \"1ff2d17c-0150-46f1-bde0-43d0770da3f9\") "
Jul 08 12:08:04.971434 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:04.971322    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7dkdf\" (UniqueName: \"kubernetes.io/projected/1ff2d17c-0150-46f1-bde0-43d0770da3f9-kube-api-access-7dkdf\") pod \"1ff2d17c-0150-46f1-bde0-43d0770da3f9\" (UID: \"1ff2d17c-0150-46f1-bde0-43d0770da3f9\") "
Jul 08 12:08:04.971434 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:04.971353    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1ff2d17c-0150-46f1-bde0-43d0770da3f9-cacerts\") pod \"1ff2d17c-0150-46f1-bde0-43d0770da3f9\" (UID: \"1ff2d17c-0150-46f1-bde0-43d0770da3f9\") "
Jul 08 12:08:04.971434 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:04.971426    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ff2d17c-0150-46f1-bde0-43d0770da3f9-istio-token\") pod \"1ff2d17c-0150-46f1-bde0-43d0770da3f9\" (UID: \"1ff2d17c-0150-46f1-bde0-43d0770da3f9\") "
Jul 08 12:08:04.971642 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:04.971457    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ff2d17c-0150-46f1-bde0-43d0770da3f9-local-certs\") pod \"1ff2d17c-0150-46f1-bde0-43d0770da3f9\" (UID: \"1ff2d17c-0150-46f1-bde0-43d0770da3f9\") "
Jul 08 12:08:04.971642 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:04.971480    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1ff2d17c-0150-46f1-bde0-43d0770da3f9-istio-csr-dns-cert\") pod \"1ff2d17c-0150-46f1-bde0-43d0770da3f9\" (UID: \"1ff2d17c-0150-46f1-bde0-43d0770da3f9\") "
Jul 08 12:08:04.972060 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:04.971743    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ff2d17c-0150-46f1-bde0-43d0770da3f9-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "1ff2d17c-0150-46f1-bde0-43d0770da3f9" (UID: "1ff2d17c-0150-46f1-bde0-43d0770da3f9"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 12:08:04.974994 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:04.974939    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ff2d17c-0150-46f1-bde0-43d0770da3f9-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "1ff2d17c-0150-46f1-bde0-43d0770da3f9" (UID: "1ff2d17c-0150-46f1-bde0-43d0770da3f9"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 12:08:04.974994 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:04.974963    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ff2d17c-0150-46f1-bde0-43d0770da3f9-local-certs" (OuterVolumeSpecName: "local-certs") pod "1ff2d17c-0150-46f1-bde0-43d0770da3f9" (UID: "1ff2d17c-0150-46f1-bde0-43d0770da3f9"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 12:08:04.975172 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:04.975050    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ff2d17c-0150-46f1-bde0-43d0770da3f9-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "1ff2d17c-0150-46f1-bde0-43d0770da3f9" (UID: "1ff2d17c-0150-46f1-bde0-43d0770da3f9"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 12:08:04.975172 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:04.975039    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ff2d17c-0150-46f1-bde0-43d0770da3f9-kube-api-access-7dkdf" (OuterVolumeSpecName: "kube-api-access-7dkdf") pod "1ff2d17c-0150-46f1-bde0-43d0770da3f9" (UID: "1ff2d17c-0150-46f1-bde0-43d0770da3f9"). InnerVolumeSpecName "kube-api-access-7dkdf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 12:08:04.975172 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:04.975140    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ff2d17c-0150-46f1-bde0-43d0770da3f9-istio-token" (OuterVolumeSpecName: "istio-token") pod "1ff2d17c-0150-46f1-bde0-43d0770da3f9" (UID: "1ff2d17c-0150-46f1-bde0-43d0770da3f9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 12:08:04.975683 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:04.975655    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ff2d17c-0150-46f1-bde0-43d0770da3f9-cacerts" (OuterVolumeSpecName: "cacerts") pod "1ff2d17c-0150-46f1-bde0-43d0770da3f9" (UID: "1ff2d17c-0150-46f1-bde0-43d0770da3f9"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 12:08:05.073206 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:05.073119    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ff2d17c-0150-46f1-bde0-43d0770da3f9-istio-token\") on node \"ip-10-0-138-89.ec2.internal\" DevicePath \"\""
Jul 08 12:08:05.073206 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:05.073146    2579 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ff2d17c-0150-46f1-bde0-43d0770da3f9-local-certs\") on node \"ip-10-0-138-89.ec2.internal\" DevicePath \"\""
Jul 08 12:08:05.073206 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:05.073159    2579 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1ff2d17c-0150-46f1-bde0-43d0770da3f9-istio-csr-dns-cert\") on node \"ip-10-0-138-89.ec2.internal\" DevicePath \"\""
Jul 08 12:08:05.073206 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:05.073173    2579 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1ff2d17c-0150-46f1-bde0-43d0770da3f9-istio-kubeconfig\") on node \"ip-10-0-138-89.ec2.internal\" DevicePath \"\""
Jul 08 12:08:05.073206 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:05.073186    2579 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1ff2d17c-0150-46f1-bde0-43d0770da3f9-istio-csr-ca-configmap\") on node \"ip-10-0-138-89.ec2.internal\" DevicePath \"\""
Jul 08 12:08:05.073206 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:05.073199    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dkdf\" (UniqueName: \"kubernetes.io/projected/1ff2d17c-0150-46f1-bde0-43d0770da3f9-kube-api-access-7dkdf\") on node \"ip-10-0-138-89.ec2.internal\" DevicePath \"\""
Jul 08 12:08:05.073206 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:05.073208    2579 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1ff2d17c-0150-46f1-bde0-43d0770da3f9-cacerts\") on node \"ip-10-0-138-89.ec2.internal\" DevicePath \"\""
Jul 08 12:08:05.172281 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:05.172248    2579 generic.go:356] "Generic (PLEG): container finished" podID="1ff2d17c-0150-46f1-bde0-43d0770da3f9" containerID="036f245a4ee4fb0a80e1f8d98a49a67bef93f2a726250a993c2a4f22b4ad586f" exitCode=0
Jul 08 12:08:05.172473 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:05.172310    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-n8c8q" event={"ID":"1ff2d17c-0150-46f1-bde0-43d0770da3f9","Type":"ContainerDied","Data":"036f245a4ee4fb0a80e1f8d98a49a67bef93f2a726250a993c2a4f22b4ad586f"}
Jul 08 12:08:05.172473 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:05.172356    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-n8c8q" event={"ID":"1ff2d17c-0150-46f1-bde0-43d0770da3f9","Type":"ContainerDied","Data":"596a2ecba7e764b23fc30a6be43a808cf8c5d3d68e2338e334fe9e33de1cdbdc"}
Jul 08 12:08:05.172473 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:05.172380    2579 scope.go:117] "RemoveContainer" containerID="036f245a4ee4fb0a80e1f8d98a49a67bef93f2a726250a993c2a4f22b4ad586f"
Jul 08 12:08:05.172473 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:05.172313    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-n8c8q"
Jul 08 12:08:05.226765 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:05.226705    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-n8c8q"]
Jul 08 12:08:05.232488 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:05.232449    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-n8c8q"]
Jul 08 12:08:05.617830 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:05.617668    2579 scope.go:117] "RemoveContainer" containerID="036f245a4ee4fb0a80e1f8d98a49a67bef93f2a726250a993c2a4f22b4ad586f"
Jul 08 12:08:05.618125 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:08:05.618090    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"036f245a4ee4fb0a80e1f8d98a49a67bef93f2a726250a993c2a4f22b4ad586f\": container with ID starting with 036f245a4ee4fb0a80e1f8d98a49a67bef93f2a726250a993c2a4f22b4ad586f not found: ID does not exist" containerID="036f245a4ee4fb0a80e1f8d98a49a67bef93f2a726250a993c2a4f22b4ad586f"
Jul 08 12:08:05.618166 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:05.618117    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"036f245a4ee4fb0a80e1f8d98a49a67bef93f2a726250a993c2a4f22b4ad586f"} err="failed to get container status \"036f245a4ee4fb0a80e1f8d98a49a67bef93f2a726250a993c2a4f22b4ad586f\": rpc error: code = NotFound desc = could not find container \"036f245a4ee4fb0a80e1f8d98a49a67bef93f2a726250a993c2a4f22b4ad586f\": container with ID starting with 036f245a4ee4fb0a80e1f8d98a49a67bef93f2a726250a993c2a4f22b4ad586f not found: ID does not exist"
Jul 08 12:08:06.179160 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:06.179127    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-9qwx5" event={"ID":"19452e60-a504-485d-bf01-e0acffd5f18c","Type":"ContainerStarted","Data":"acb2de6254dcf1e4dad0ed825cb6102a66698b24086a548859df827c6b52a893"}
Jul 08 12:08:06.179348 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:06.179265    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-9qwx5"
Jul 08 12:08:06.243297 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:06.243238    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-9qwx5" podStartSLOduration=1.993525862 podStartE2EDuration="5.243220679s" totalImagesPullingTime="3.249694817s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 12:08:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 12:08:06.24190552 +0000 UTC m=+375.972280492" watchObservedRunningTime="2026-07-08 12:08:06.243220679 +0000 UTC m=+375.973595650"
Jul 08 12:08:06.716341 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:06.716307    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ff2d17c-0150-46f1-bde0-43d0770da3f9" path="/var/lib/kubelet/pods/1ff2d17c-0150-46f1-bde0-43d0770da3f9/volumes"
Jul 08 12:08:07.184650 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:07.184618    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2g5zq" event={"ID":"8317baa3-d235-47f8-94c4-308c96c6fb2a","Type":"ContainerStarted","Data":"e0514fdb920a1365cda9ac184bebaafdf5cba794f9218f31e9d3e6eabbe4d33d"}
Jul 08 12:08:07.184949 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:07.184930    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2g5zq"
Jul 08 12:08:07.216487 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:07.216430    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2g5zq" podStartSLOduration=2.340720389 podStartE2EDuration="6.21641015s" totalImagesPullingTime="3.875689761s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 12:08:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 12:08:07.214493348 +0000 UTC m=+376.944868318" watchObservedRunningTime="2026-07-08 12:08:07.21641015 +0000 UTC m=+376.946785123"
Jul 08 12:08:37.190472 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:37.190444    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-9qwx5"
Jul 08 12:08:38.191526 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:38.191498    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2g5zq"
Jul 08 12:08:49.474654 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:49.474617    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2ks9q"]
Jul 08 12:08:49.475055 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:49.474923    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ff2d17c-0150-46f1-bde0-43d0770da3f9" containerName="discovery"
Jul 08 12:08:49.475055 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:49.474934    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ff2d17c-0150-46f1-bde0-43d0770da3f9" containerName="discovery"
Jul 08 12:08:49.475055 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:49.474999    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ff2d17c-0150-46f1-bde0-43d0770da3f9" containerName="discovery"
Jul 08 12:08:49.476579 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:49.476564    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-2ks9q"
Jul 08 12:08:49.479177 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:49.479151    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 08 12:08:49.479315 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:49.479193    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 08 12:08:49.480255 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:49.480237    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-gl9cc\""
Jul 08 12:08:49.486264 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:49.486240    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2ks9q"]
Jul 08 12:08:49.542842 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:49.542809    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsfr2\" (UniqueName: \"kubernetes.io/projected/73703229-9980-4fd8-9dc4-8ef55be09475-kube-api-access-tsfr2\") pod \"kuadrant-operator-catalog-2ks9q\" (UID: \"73703229-9980-4fd8-9dc4-8ef55be09475\") " pod="kuadrant-system/kuadrant-operator-catalog-2ks9q"
Jul 08 12:08:49.644187 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:49.644155    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsfr2\" (UniqueName: \"kubernetes.io/projected/73703229-9980-4fd8-9dc4-8ef55be09475-kube-api-access-tsfr2\") pod \"kuadrant-operator-catalog-2ks9q\" (UID: \"73703229-9980-4fd8-9dc4-8ef55be09475\") " pod="kuadrant-system/kuadrant-operator-catalog-2ks9q"
Jul 08 12:08:49.653495 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:49.653467    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsfr2\" (UniqueName: \"kubernetes.io/projected/73703229-9980-4fd8-9dc4-8ef55be09475-kube-api-access-tsfr2\") pod \"kuadrant-operator-catalog-2ks9q\" (UID: \"73703229-9980-4fd8-9dc4-8ef55be09475\") " pod="kuadrant-system/kuadrant-operator-catalog-2ks9q"
Jul 08 12:08:49.787109 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:49.787003    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-2ks9q"
Jul 08 12:08:49.825351 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:49.825309    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2ks9q"]
Jul 08 12:08:49.921539 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:49.921510    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2ks9q"]
Jul 08 12:08:49.924038 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:08:49.924011    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73703229_9980_4fd8_9dc4_8ef55be09475.slice/crio-c00f45fc2ab7feac9f9cf697174d6caab23e4c4175861786149ba095219965f9 WatchSource:0}: Error finding container c00f45fc2ab7feac9f9cf697174d6caab23e4c4175861786149ba095219965f9: Status 404 returned error can't find the container with id c00f45fc2ab7feac9f9cf697174d6caab23e4c4175861786149ba095219965f9
Jul 08 12:08:50.038375 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:50.038270    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-klrp9"]
Jul 08 12:08:50.041047 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:50.041028    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-klrp9"
Jul 08 12:08:50.050888 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:50.050864    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-klrp9"]
Jul 08 12:08:50.146984 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:50.146945    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb7gc\" (UniqueName: \"kubernetes.io/projected/06529644-27d8-4092-9f2f-4199a0132d2a-kube-api-access-qb7gc\") pod \"kuadrant-operator-catalog-klrp9\" (UID: \"06529644-27d8-4092-9f2f-4199a0132d2a\") " pod="kuadrant-system/kuadrant-operator-catalog-klrp9"
Jul 08 12:08:50.247597 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:50.247562    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb7gc\" (UniqueName: \"kubernetes.io/projected/06529644-27d8-4092-9f2f-4199a0132d2a-kube-api-access-qb7gc\") pod \"kuadrant-operator-catalog-klrp9\" (UID: \"06529644-27d8-4092-9f2f-4199a0132d2a\") " pod="kuadrant-system/kuadrant-operator-catalog-klrp9"
Jul 08 12:08:50.257511 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:50.257489    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb7gc\" (UniqueName: \"kubernetes.io/projected/06529644-27d8-4092-9f2f-4199a0132d2a-kube-api-access-qb7gc\") pod \"kuadrant-operator-catalog-klrp9\" (UID: \"06529644-27d8-4092-9f2f-4199a0132d2a\") " pod="kuadrant-system/kuadrant-operator-catalog-klrp9"
Jul 08 12:08:50.350370 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:50.350275    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-klrp9"
Jul 08 12:08:50.363280 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:50.363255    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-2ks9q" event={"ID":"73703229-9980-4fd8-9dc4-8ef55be09475","Type":"ContainerStarted","Data":"c00f45fc2ab7feac9f9cf697174d6caab23e4c4175861786149ba095219965f9"}
Jul 08 12:08:50.485511 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:50.485485    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-klrp9"]
Jul 08 12:08:51.368924 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:51.368891    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-klrp9" event={"ID":"06529644-27d8-4092-9f2f-4199a0132d2a","Type":"ContainerStarted","Data":"20b0e881ec22e0509b5f6921f5f8f7a620be1659416785e401d36db02de751a1"}
Jul 08 12:08:52.375842 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:52.375510    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-klrp9" event={"ID":"06529644-27d8-4092-9f2f-4199a0132d2a","Type":"ContainerStarted","Data":"8dd28ac558b51754f149769238206b318ff9701200cdb07642cfbae5216e453b"}
Jul 08 12:08:52.377999 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:52.377965    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-2ks9q" event={"ID":"73703229-9980-4fd8-9dc4-8ef55be09475","Type":"ContainerStarted","Data":"3457cc1c045c15a100e7cef0d0e0cc0cd1a4e9bdd75089d4b3a9cc50c62940dd"}
Jul 08 12:08:52.378104 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:52.378048    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-2ks9q" podUID="73703229-9980-4fd8-9dc4-8ef55be09475" containerName="registry-server" containerID="cri-o://3457cc1c045c15a100e7cef0d0e0cc0cd1a4e9bdd75089d4b3a9cc50c62940dd" gracePeriod=2
Jul 08 12:08:52.396311 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:52.396235    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-klrp9" podStartSLOduration=0.664762745 podStartE2EDuration="2.396219341s" totalImagesPullingTime="1.731456596s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 12:08:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 12:08:52.395053168 +0000 UTC m=+422.125428139" watchObservedRunningTime="2026-07-08 12:08:52.396219341 +0000 UTC m=+422.126594311"
Jul 08 12:08:52.417740 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:52.417690    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-2ks9q" podStartSLOduration=1.056902212 podStartE2EDuration="3.417671888s" totalImagesPullingTime="2.360769676s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 12:08:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 12:08:52.416158427 +0000 UTC m=+422.146533399" watchObservedRunningTime="2026-07-08 12:08:52.417671888 +0000 UTC m=+422.148046889"
Jul 08 12:08:52.619925 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:52.619900    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-2ks9q_73703229-9980-4fd8-9dc4-8ef55be09475/registry-server/0.log"
Jul 08 12:08:52.622307 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:52.622288    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-2ks9q"
Jul 08 12:08:52.668728 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:52.668695    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tsfr2\" (UniqueName: \"kubernetes.io/projected/73703229-9980-4fd8-9dc4-8ef55be09475-kube-api-access-tsfr2\") pod \"73703229-9980-4fd8-9dc4-8ef55be09475\" (UID: \"73703229-9980-4fd8-9dc4-8ef55be09475\") "
Jul 08 12:08:52.670986 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:52.670965    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73703229-9980-4fd8-9dc4-8ef55be09475-kube-api-access-tsfr2" (OuterVolumeSpecName: "kube-api-access-tsfr2") pod "73703229-9980-4fd8-9dc4-8ef55be09475" (UID: "73703229-9980-4fd8-9dc4-8ef55be09475"). InnerVolumeSpecName "kube-api-access-tsfr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 12:08:52.770189 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:52.770144    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsfr2\" (UniqueName: \"kubernetes.io/projected/73703229-9980-4fd8-9dc4-8ef55be09475-kube-api-access-tsfr2\") on node \"ip-10-0-138-89.ec2.internal\" DevicePath \"\""
Jul 08 12:08:53.383263 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:53.383236    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-2ks9q_73703229-9980-4fd8-9dc4-8ef55be09475/registry-server/0.log"
Jul 08 12:08:53.383671 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:53.383272    2579 generic.go:356] "Generic (PLEG): container finished" podID="73703229-9980-4fd8-9dc4-8ef55be09475" containerID="3457cc1c045c15a100e7cef0d0e0cc0cd1a4e9bdd75089d4b3a9cc50c62940dd" exitCode=2
Jul 08 12:08:53.383671 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:53.383309    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-2ks9q" event={"ID":"73703229-9980-4fd8-9dc4-8ef55be09475","Type":"ContainerDied","Data":"3457cc1c045c15a100e7cef0d0e0cc0cd1a4e9bdd75089d4b3a9cc50c62940dd"}
Jul 08 12:08:53.383671 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:53.383365    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-2ks9q" event={"ID":"73703229-9980-4fd8-9dc4-8ef55be09475","Type":"ContainerDied","Data":"c00f45fc2ab7feac9f9cf697174d6caab23e4c4175861786149ba095219965f9"}
Jul 08 12:08:53.383671 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:53.383321    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-2ks9q"
Jul 08 12:08:53.384328 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:53.383386    2579 scope.go:117] "RemoveContainer" containerID="3457cc1c045c15a100e7cef0d0e0cc0cd1a4e9bdd75089d4b3a9cc50c62940dd"
Jul 08 12:08:53.393785 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:53.393754    2579 scope.go:117] "RemoveContainer" containerID="3457cc1c045c15a100e7cef0d0e0cc0cd1a4e9bdd75089d4b3a9cc50c62940dd"
Jul 08 12:08:53.394127 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:08:53.394104    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3457cc1c045c15a100e7cef0d0e0cc0cd1a4e9bdd75089d4b3a9cc50c62940dd\": container with ID starting with 3457cc1c045c15a100e7cef0d0e0cc0cd1a4e9bdd75089d4b3a9cc50c62940dd not found: ID does not exist" containerID="3457cc1c045c15a100e7cef0d0e0cc0cd1a4e9bdd75089d4b3a9cc50c62940dd"
Jul 08 12:08:53.394200 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:53.394134    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3457cc1c045c15a100e7cef0d0e0cc0cd1a4e9bdd75089d4b3a9cc50c62940dd"} err="failed to get container status \"3457cc1c045c15a100e7cef0d0e0cc0cd1a4e9bdd75089d4b3a9cc50c62940dd\": rpc error: code = NotFound desc = could not find container \"3457cc1c045c15a100e7cef0d0e0cc0cd1a4e9bdd75089d4b3a9cc50c62940dd\": container with ID starting with 3457cc1c045c15a100e7cef0d0e0cc0cd1a4e9bdd75089d4b3a9cc50c62940dd not found: ID does not exist"
Jul 08 12:08:53.404861 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:53.404841    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2ks9q"]
Jul 08 12:08:53.408634 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:53.408613    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2ks9q"]
Jul 08 12:08:54.716962 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:08:54.716935    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73703229-9980-4fd8-9dc4-8ef55be09475" path="/var/lib/kubelet/pods/73703229-9980-4fd8-9dc4-8ef55be09475/volumes"
Jul 08 12:09:00.351241 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:00.351204    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-klrp9"
Jul 08 12:09:00.351739 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:00.351255    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-klrp9"
Jul 08 12:09:00.373919 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:00.373896    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-klrp9"
Jul 08 12:09:00.434491 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:00.434462    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-klrp9"
Jul 08 12:09:19.101379 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:19.101260    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-7xlrf"]
Jul 08 12:09:19.101857 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:19.101629    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73703229-9980-4fd8-9dc4-8ef55be09475" containerName="registry-server"
Jul 08 12:09:19.101857 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:19.101643    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="73703229-9980-4fd8-9dc4-8ef55be09475" containerName="registry-server"
Jul 08 12:09:19.101857 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:19.101698    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="73703229-9980-4fd8-9dc4-8ef55be09475" containerName="registry-server"
Jul 08 12:09:19.104573 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:19.104555    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7xlrf"
Jul 08 12:09:19.107496 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:19.107474    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-sllh8\""
Jul 08 12:09:19.116557 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:19.116537    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-7xlrf"]
Jul 08 12:09:19.201698 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:19.201670    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gfll\" (UniqueName: \"kubernetes.io/projected/eb3b9fcd-8703-4c3a-8709-f349ea0cc5e3-kube-api-access-5gfll\") pod \"authorino-operator-75cc8d5cb7-7xlrf\" (UID: \"eb3b9fcd-8703-4c3a-8709-f349ea0cc5e3\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-7xlrf"
Jul 08 12:09:19.302252 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:19.302214    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gfll\" (UniqueName: \"kubernetes.io/projected/eb3b9fcd-8703-4c3a-8709-f349ea0cc5e3-kube-api-access-5gfll\") pod \"authorino-operator-75cc8d5cb7-7xlrf\" (UID: \"eb3b9fcd-8703-4c3a-8709-f349ea0cc5e3\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-7xlrf"
Jul 08 12:09:19.311845 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:19.311822    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gfll\" (UniqueName: \"kubernetes.io/projected/eb3b9fcd-8703-4c3a-8709-f349ea0cc5e3-kube-api-access-5gfll\") pod \"authorino-operator-75cc8d5cb7-7xlrf\" (UID: \"eb3b9fcd-8703-4c3a-8709-f349ea0cc5e3\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-7xlrf"
Jul 08 12:09:19.413847 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:19.413749    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7xlrf"
Jul 08 12:09:19.550744 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:09:19.550711    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb3b9fcd_8703_4c3a_8709_f349ea0cc5e3.slice/crio-a25101c07b2853303289d2b90be2c1319dd909e77d84b1c7c4a781e1d4e6ae52 WatchSource:0}: Error finding container a25101c07b2853303289d2b90be2c1319dd909e77d84b1c7c4a781e1d4e6ae52: Status 404 returned error can't find the container with id a25101c07b2853303289d2b90be2c1319dd909e77d84b1c7c4a781e1d4e6ae52
Jul 08 12:09:19.554189 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:19.554164    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-7xlrf"]
Jul 08 12:09:20.499366 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:20.499326    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7xlrf" event={"ID":"eb3b9fcd-8703-4c3a-8709-f349ea0cc5e3","Type":"ContainerStarted","Data":"a25101c07b2853303289d2b90be2c1319dd909e77d84b1c7c4a781e1d4e6ae52"}
Jul 08 12:09:22.515023 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:22.514982    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7xlrf" event={"ID":"eb3b9fcd-8703-4c3a-8709-f349ea0cc5e3","Type":"ContainerStarted","Data":"c6cc6e1d9a093b10709f9e75d0bb178d28bfb18b39693485b17235119370f7ab"}
Jul 08 12:09:22.515404 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:22.515130    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7xlrf"
Jul 08 12:09:22.620210 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:22.620156    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7xlrf" podStartSLOduration=1.344284276 podStartE2EDuration="3.620137646s" totalImagesPullingTime="2.27585337s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 12:09:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 12:09:22.546543951 +0000 UTC m=+452.276918922" watchObservedRunningTime="2026-07-08 12:09:22.620137646 +0000 UTC m=+452.350512620"
Jul 08 12:09:22.620467 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:22.620448    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ls5vt"]
Jul 08 12:09:22.623891 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:22.623867    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ls5vt"
Jul 08 12:09:22.633166 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:22.633144    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-9x4hk\""
Jul 08 12:09:22.640126 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:22.640098    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ls5vt"]
Jul 08 12:09:22.732399 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:22.732364    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5d2t\" (UniqueName: \"kubernetes.io/projected/c64f287a-191f-46a1-aadc-f26cea7292d9-kube-api-access-b5d2t\") pod \"kuadrant-operator-controller-manager-8684c9ff49-ls5vt\" (UID: \"c64f287a-191f-46a1-aadc-f26cea7292d9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ls5vt"
Jul 08 12:09:22.732399 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:22.732400    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c64f287a-191f-46a1-aadc-f26cea7292d9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-ls5vt\" (UID: \"c64f287a-191f-46a1-aadc-f26cea7292d9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ls5vt"
Jul 08 12:09:22.833899 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:22.833804    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5d2t\" (UniqueName: \"kubernetes.io/projected/c64f287a-191f-46a1-aadc-f26cea7292d9-kube-api-access-b5d2t\") pod \"kuadrant-operator-controller-manager-8684c9ff49-ls5vt\" (UID: \"c64f287a-191f-46a1-aadc-f26cea7292d9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ls5vt"
Jul 08 12:09:22.833899 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:22.833844    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c64f287a-191f-46a1-aadc-f26cea7292d9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-ls5vt\" (UID: \"c64f287a-191f-46a1-aadc-f26cea7292d9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ls5vt"
Jul 08 12:09:22.834206 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:22.834187    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c64f287a-191f-46a1-aadc-f26cea7292d9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-ls5vt\" (UID: \"c64f287a-191f-46a1-aadc-f26cea7292d9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ls5vt"
Jul 08 12:09:22.846783 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:22.846734    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5d2t\" (UniqueName: \"kubernetes.io/projected/c64f287a-191f-46a1-aadc-f26cea7292d9-kube-api-access-b5d2t\") pod \"kuadrant-operator-controller-manager-8684c9ff49-ls5vt\" (UID: \"c64f287a-191f-46a1-aadc-f26cea7292d9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ls5vt"
Jul 08 12:09:22.933712 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:22.933667    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ls5vt"
Jul 08 12:09:23.075318 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:23.075223    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ls5vt"]
Jul 08 12:09:23.077605 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:09:23.077576    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc64f287a_191f_46a1_aadc_f26cea7292d9.slice/crio-6a84242fb7ddcbe52c43dcc772ec806244dff6a3062eed77bb0a7d6c6c3dd79b WatchSource:0}: Error finding container 6a84242fb7ddcbe52c43dcc772ec806244dff6a3062eed77bb0a7d6c6c3dd79b: Status 404 returned error can't find the container with id 6a84242fb7ddcbe52c43dcc772ec806244dff6a3062eed77bb0a7d6c6c3dd79b
Jul 08 12:09:23.520138 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:23.520099    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ls5vt" event={"ID":"c64f287a-191f-46a1-aadc-f26cea7292d9","Type":"ContainerStarted","Data":"6a84242fb7ddcbe52c43dcc772ec806244dff6a3062eed77bb0a7d6c6c3dd79b"}
Jul 08 12:09:24.597230 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:24.597194    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-64lwx"]
Jul 08 12:09:24.600322 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:24.600306    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-64lwx"
Jul 08 12:09:24.602864 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:24.602846    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-f5nd2\""
Jul 08 12:09:24.602960 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:24.602871    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 08 12:09:24.614268 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:24.614242    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-64lwx"]
Jul 08 12:09:24.649056 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:24.649027    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs2xf\" (UniqueName: \"kubernetes.io/projected/9340764e-4d7d-4140-9111-e58bedcfee6b-kube-api-access-xs2xf\") pod \"dns-operator-controller-manager-864dd78c4c-64lwx\" (UID: \"9340764e-4d7d-4140-9111-e58bedcfee6b\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-64lwx"
Jul 08 12:09:24.750316 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:24.750275    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs2xf\" (UniqueName: \"kubernetes.io/projected/9340764e-4d7d-4140-9111-e58bedcfee6b-kube-api-access-xs2xf\") pod \"dns-operator-controller-manager-864dd78c4c-64lwx\" (UID: \"9340764e-4d7d-4140-9111-e58bedcfee6b\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-64lwx"
Jul 08 12:09:24.766952 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:24.766925    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs2xf\" (UniqueName: \"kubernetes.io/projected/9340764e-4d7d-4140-9111-e58bedcfee6b-kube-api-access-xs2xf\") pod \"dns-operator-controller-manager-864dd78c4c-64lwx\" (UID: \"9340764e-4d7d-4140-9111-e58bedcfee6b\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-64lwx"
Jul 08 12:09:24.910838 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:24.910718    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-64lwx"
Jul 08 12:09:25.048869 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:25.048842    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-64lwx"]
Jul 08 12:09:25.052078 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:09:25.052053    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9340764e_4d7d_4140_9111_e58bedcfee6b.slice/crio-db555a76a370e0b5f3ba916b65144e6f54a4e921b64c2f7a53eb01c4a29828b8 WatchSource:0}: Error finding container db555a76a370e0b5f3ba916b65144e6f54a4e921b64c2f7a53eb01c4a29828b8: Status 404 returned error can't find the container with id db555a76a370e0b5f3ba916b65144e6f54a4e921b64c2f7a53eb01c4a29828b8
Jul 08 12:09:25.529713 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:25.529677    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-64lwx" event={"ID":"9340764e-4d7d-4140-9111-e58bedcfee6b","Type":"ContainerStarted","Data":"db555a76a370e0b5f3ba916b65144e6f54a4e921b64c2f7a53eb01c4a29828b8"}
Jul 08 12:09:27.296197 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:27.296156    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jtdbf"]
Jul 08 12:09:27.299997 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:27.299967    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jtdbf"
Jul 08 12:09:27.302945 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:27.302917    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-467cw\""
Jul 08 12:09:27.310869 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:27.310839    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jtdbf"]
Jul 08 12:09:27.370470 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:27.370430    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tchwp\" (UniqueName: \"kubernetes.io/projected/c0018713-4bd2-4dd6-b090-6cc1e012669e-kube-api-access-tchwp\") pod \"limitador-operator-controller-manager-859d7b78bd-jtdbf\" (UID: \"c0018713-4bd2-4dd6-b090-6cc1e012669e\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jtdbf"
Jul 08 12:09:27.471156 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:27.471073    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tchwp\" (UniqueName: \"kubernetes.io/projected/c0018713-4bd2-4dd6-b090-6cc1e012669e-kube-api-access-tchwp\") pod \"limitador-operator-controller-manager-859d7b78bd-jtdbf\" (UID: \"c0018713-4bd2-4dd6-b090-6cc1e012669e\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jtdbf"
Jul 08 12:09:27.482232 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:27.482198    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tchwp\" (UniqueName: \"kubernetes.io/projected/c0018713-4bd2-4dd6-b090-6cc1e012669e-kube-api-access-tchwp\") pod \"limitador-operator-controller-manager-859d7b78bd-jtdbf\" (UID: \"c0018713-4bd2-4dd6-b090-6cc1e012669e\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jtdbf"
Jul 08 12:09:27.541310 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:27.541273    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-64lwx" event={"ID":"9340764e-4d7d-4140-9111-e58bedcfee6b","Type":"ContainerStarted","Data":"25a4779bbf1297b76e66b38dc14d4d524f5f5476f198ef267f83062f64916335"}
Jul 08 12:09:27.541492 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:27.541418    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-64lwx"
Jul 08 12:09:27.566317 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:27.566264    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-64lwx" podStartSLOduration=1.440290595 podStartE2EDuration="3.566248025s" totalImagesPullingTime="2.12595743s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 12:09:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 12:09:27.565825121 +0000 UTC m=+457.296200090" watchObservedRunningTime="2026-07-08 12:09:27.566248025 +0000 UTC m=+457.296622995"
Jul 08 12:09:27.611431 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:27.611390    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jtdbf"
Jul 08 12:09:27.749896 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:27.749819    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jtdbf"]
Jul 08 12:09:27.753307 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:09:27.753277    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0018713_4bd2_4dd6_b090_6cc1e012669e.slice/crio-a028f056fe7917edeebeae5e463d07fd70350bb8169de0da6225d8f77a0c60cc WatchSource:0}: Error finding container a028f056fe7917edeebeae5e463d07fd70350bb8169de0da6225d8f77a0c60cc: Status 404 returned error can't find the container with id a028f056fe7917edeebeae5e463d07fd70350bb8169de0da6225d8f77a0c60cc
Jul 08 12:09:28.548465 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:28.548420    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jtdbf" event={"ID":"c0018713-4bd2-4dd6-b090-6cc1e012669e","Type":"ContainerStarted","Data":"a028f056fe7917edeebeae5e463d07fd70350bb8169de0da6225d8f77a0c60cc"}
Jul 08 12:09:30.066240 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:30.066158    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-78887568fd-gnvfk"]
Jul 08 12:09:30.069530 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:30.069506    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78887568fd-gnvfk"
Jul 08 12:09:30.072279 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:30.072258    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 08 12:09:30.072402 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:30.072264    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 08 12:09:30.072719 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:30.072698    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 08 12:09:30.072928 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:30.072916    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 08 12:09:30.073754 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:30.073734    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 08 12:09:30.074030 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:30.074015    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 08 12:09:30.074477 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:30.074462    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 08 12:09:30.074739 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:30.074723    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-km99f\""
Jul 08 12:09:30.077907 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:30.077887    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 08 12:09:30.079608 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:30.079585    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78887568fd-gnvfk"]
Jul 08 12:09:30.092337 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:30.092307    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6af39b1-04e6-48c4-b153-f83f77d1d28f-console-serving-cert\") pod \"console-78887568fd-gnvfk\" (UID: \"b6af39b1-04e6-48c4-b153-f83f77d1d28f\") " pod="openshift-console/console-78887568fd-gnvfk"
Jul 08 12:09:30.092337 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:30.092338    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b6af39b1-04e6-48c4-b153-f83f77d1d28f-console-config\") pod \"console-78887568fd-gnvfk\" (UID: \"b6af39b1-04e6-48c4-b153-f83f77d1d28f\") " pod="openshift-console/console-78887568fd-gnvfk"
Jul 08 12:09:30.092545 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:30.092404    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b6af39b1-04e6-48c4-b153-f83f77d1d28f-console-oauth-config\") pod \"console-78887568fd-gnvfk\" (UID: \"b6af39b1-04e6-48c4-b153-f83f77d1d28f\") " pod="openshift-console/console-78887568fd-gnvfk"
Jul 08 12:09:30.092545 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:30.092421    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b6af39b1-04e6-48c4-b153-f83f77d1d28f-service-ca\") pod \"console-78887568fd-gnvfk\" (UID: \"b6af39b1-04e6-48c4-b153-f83f77d1d28f\") " pod="openshift-console/console-78887568fd-gnvfk"
Jul 08 12:09:30.092545 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:30.092443    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6af39b1-04e6-48c4-b153-f83f77d1d28f-trusted-ca-bundle\") pod \"console-78887568fd-gnvfk\" (UID: \"b6af39b1-04e6-48c4-b153-f83f77d1d28f\") " pod="openshift-console/console-78887568fd-gnvfk"
Jul 08 12:09:30.092545 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:30.092465    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p2st\" (UniqueName: \"kubernetes.io/projected/b6af39b1-04e6-48c4-b153-f83f77d1d28f-kube-api-access-5p2st\") pod \"console-78887568fd-gnvfk\" (UID: \"b6af39b1-04e6-48c4-b153-f83f77d1d28f\") " pod="openshift-console/console-78887568fd-gnvfk"
Jul 08 12:09:30.092545 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:30.092526    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b6af39b1-04e6-48c4-b153-f83f77d1d28f-oauth-serving-cert\") pod \"console-78887568fd-gnvfk\" (UID: \"b6af39b1-04e6-48c4-b153-f83f77d1d28f\") " pod="openshift-console/console-78887568fd-gnvfk"
Jul 08 12:09:30.193373 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:30.193334    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b6af39b1-04e6-48c4-b153-f83f77d1d28f-console-oauth-config\") pod \"console-78887568fd-gnvfk\" (UID: \"b6af39b1-04e6-48c4-b153-f83f77d1d28f\") " pod="openshift-console/console-78887568fd-gnvfk"
Jul 08 12:09:30.193373 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:30.193376    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b6af39b1-04e6-48c4-b153-f83f77d1d28f-service-ca\") pod \"console-78887568fd-gnvfk\" (UID: \"b6af39b1-04e6-48c4-b153-f83f77d1d28f\") " pod="openshift-console/console-78887568fd-gnvfk"
Jul 08 12:09:30.193626 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:30.193398    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6af39b1-04e6-48c4-b153-f83f77d1d28f-trusted-ca-bundle\") pod \"console-78887568fd-gnvfk\" (UID: \"b6af39b1-04e6-48c4-b153-f83f77d1d28f\") " pod="openshift-console/console-78887568fd-gnvfk"
Jul 08 12:09:30.193626 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:30.193421    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p2st\" (UniqueName: \"kubernetes.io/projected/b6af39b1-04e6-48c4-b153-f83f77d1d28f-kube-api-access-5p2st\") pod \"console-78887568fd-gnvfk\" (UID: \"b6af39b1-04e6-48c4-b153-f83f77d1d28f\") " pod="openshift-console/console-78887568fd-gnvfk"
Jul 08 12:09:30.193626 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:30.193465    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b6af39b1-04e6-48c4-b153-f83f77d1d28f-oauth-serving-cert\") pod \"console-78887568fd-gnvfk\" (UID: \"b6af39b1-04e6-48c4-b153-f83f77d1d28f\") " pod="openshift-console/console-78887568fd-gnvfk"
Jul 08 12:09:30.193626 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:30.193511    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6af39b1-04e6-48c4-b153-f83f77d1d28f-console-serving-cert\") pod \"console-78887568fd-gnvfk\" (UID: \"b6af39b1-04e6-48c4-b153-f83f77d1d28f\") " pod="openshift-console/console-78887568fd-gnvfk"
Jul 08 12:09:30.193626 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:30.193536    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b6af39b1-04e6-48c4-b153-f83f77d1d28f-console-config\") pod \"console-78887568fd-gnvfk\" (UID: \"b6af39b1-04e6-48c4-b153-f83f77d1d28f\") " pod="openshift-console/console-78887568fd-gnvfk"
Jul 08 12:09:30.194308 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:30.194246    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b6af39b1-04e6-48c4-b153-f83f77d1d28f-console-config\") pod \"console-78887568fd-gnvfk\" (UID: \"b6af39b1-04e6-48c4-b153-f83f77d1d28f\") " pod="openshift-console/console-78887568fd-gnvfk"
Jul 08 12:09:30.194425 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:30.194411    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b6af39b1-04e6-48c4-b153-f83f77d1d28f-oauth-serving-cert\") pod \"console-78887568fd-gnvfk\" (UID: \"b6af39b1-04e6-48c4-b153-f83f77d1d28f\") " pod="openshift-console/console-78887568fd-gnvfk"
Jul 08 12:09:30.194649 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:30.194445    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b6af39b1-04e6-48c4-b153-f83f77d1d28f-service-ca\") pod \"console-78887568fd-gnvfk\" (UID: \"b6af39b1-04e6-48c4-b153-f83f77d1d28f\") " pod="openshift-console/console-78887568fd-gnvfk"
Jul 08 12:09:30.194649 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:30.194461    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6af39b1-04e6-48c4-b153-f83f77d1d28f-trusted-ca-bundle\") pod \"console-78887568fd-gnvfk\" (UID: \"b6af39b1-04e6-48c4-b153-f83f77d1d28f\") " pod="openshift-console/console-78887568fd-gnvfk"
Jul 08 12:09:30.196345 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:30.196312    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b6af39b1-04e6-48c4-b153-f83f77d1d28f-console-oauth-config\") pod \"console-78887568fd-gnvfk\" (UID: \"b6af39b1-04e6-48c4-b153-f83f77d1d28f\") " pod="openshift-console/console-78887568fd-gnvfk"
Jul 08 12:09:30.196540 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:30.196492    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6af39b1-04e6-48c4-b153-f83f77d1d28f-console-serving-cert\") pod \"console-78887568fd-gnvfk\" (UID: \"b6af39b1-04e6-48c4-b153-f83f77d1d28f\") " pod="openshift-console/console-78887568fd-gnvfk"
Jul 08 12:09:30.206515 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:30.206494    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p2st\" (UniqueName: \"kubernetes.io/projected/b6af39b1-04e6-48c4-b153-f83f77d1d28f-kube-api-access-5p2st\") pod \"console-78887568fd-gnvfk\" (UID: \"b6af39b1-04e6-48c4-b153-f83f77d1d28f\") " pod="openshift-console/console-78887568fd-gnvfk"
Jul 08 12:09:30.381255 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:30.381180    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78887568fd-gnvfk"
Jul 08 12:09:30.539706 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:30.539663    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78887568fd-gnvfk"]
Jul 08 12:09:30.561006 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:30.560961    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ls5vt" event={"ID":"c64f287a-191f-46a1-aadc-f26cea7292d9","Type":"ContainerStarted","Data":"bb09688dd903927e2e62a075c93760d92a79a08d63fa146dc34df921b972ccc3"}
Jul 08 12:09:30.561164 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:30.561072    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ls5vt"
Jul 08 12:09:30.585243 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:30.585184    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ls5vt" podStartSLOduration=2.125312611 podStartE2EDuration="8.58516632s" totalImagesPullingTime="6.459853709s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 12:09:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 12:09:30.583308818 +0000 UTC m=+460.313683805" watchObservedRunningTime="2026-07-08 12:09:30.58516632 +0000 UTC m=+460.315541293"
Jul 08 12:09:30.733205 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:09:30.733177    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6af39b1_04e6_48c4_b153_f83f77d1d28f.slice/crio-61c5f6217482ca7b058f7abf49548615288774addb9eff40a077494f1ab9bff7 WatchSource:0}: Error finding container 61c5f6217482ca7b058f7abf49548615288774addb9eff40a077494f1ab9bff7: Status 404 returned error can't find the container with id 61c5f6217482ca7b058f7abf49548615288774addb9eff40a077494f1ab9bff7
Jul 08 12:09:31.567254 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:31.567211    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jtdbf" event={"ID":"c0018713-4bd2-4dd6-b090-6cc1e012669e","Type":"ContainerStarted","Data":"79ffb6104d00536d8e4604b21ab732a5523ac536dfcfe2aad71e3a6b2d636960"}
Jul 08 12:09:31.567850 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:31.567318    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jtdbf"
Jul 08 12:09:31.568807 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:31.568755    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78887568fd-gnvfk" event={"ID":"b6af39b1-04e6-48c4-b153-f83f77d1d28f","Type":"ContainerStarted","Data":"5676b5c9c0b7fa84f8db04d5107f432d4ef926ebe96176b94bed5ff02512d928"}
Jul 08 12:09:31.568924 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:31.568821    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78887568fd-gnvfk" event={"ID":"b6af39b1-04e6-48c4-b153-f83f77d1d28f","Type":"ContainerStarted","Data":"61c5f6217482ca7b058f7abf49548615288774addb9eff40a077494f1ab9bff7"}
Jul 08 12:09:31.588967 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:31.588923    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jtdbf" podStartSLOduration=1.5529654050000001 podStartE2EDuration="4.588908392s" totalImagesPullingTime="3.035942987s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 12:09:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 12:09:31.587899831 +0000 UTC m=+461.318274801" watchObservedRunningTime="2026-07-08 12:09:31.588908392 +0000 UTC m=+461.319283363"
Jul 08 12:09:31.611576 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:31.611497    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-78887568fd-gnvfk" podStartSLOduration=1.611475783 podStartE2EDuration="1.611475783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 12:09:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 12:09:31.61069257 +0000 UTC m=+461.341067540" watchObservedRunningTime="2026-07-08 12:09:31.611475783 +0000 UTC m=+461.341850757"
Jul 08 12:09:32.600851 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:32.600813    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-vhwhh"]
Jul 08 12:09:32.604709 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:32.604689    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vhwhh"
Jul 08 12:09:32.608216 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:32.608196    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 08 12:09:32.608216 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:32.608207    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-2jxhl\""
Jul 08 12:09:32.608343 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:32.608268    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 08 12:09:32.613670 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:32.613648    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-vhwhh"]
Jul 08 12:09:32.716220 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:32.716182    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/6d96c715-e042-4e8f-9adf-de115944c84f-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-vhwhh\" (UID: \"6d96c715-e042-4e8f-9adf-de115944c84f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vhwhh"
Jul 08 12:09:32.716220 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:32.716219    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6d96c715-e042-4e8f-9adf-de115944c84f-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-vhwhh\" (UID: \"6d96c715-e042-4e8f-9adf-de115944c84f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vhwhh"
Jul 08 12:09:32.716435 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:32.716241    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxzsp\" (UniqueName: \"kubernetes.io/projected/6d96c715-e042-4e8f-9adf-de115944c84f-kube-api-access-sxzsp\") pod \"kuadrant-console-plugin-66b5774985-vhwhh\" (UID: \"6d96c715-e042-4e8f-9adf-de115944c84f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vhwhh"
Jul 08 12:09:32.817269 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:32.817233    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/6d96c715-e042-4e8f-9adf-de115944c84f-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-vhwhh\" (UID: \"6d96c715-e042-4e8f-9adf-de115944c84f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vhwhh"
Jul 08 12:09:32.817269 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:32.817272    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6d96c715-e042-4e8f-9adf-de115944c84f-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-vhwhh\" (UID: \"6d96c715-e042-4e8f-9adf-de115944c84f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vhwhh"
Jul 08 12:09:32.817522 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:32.817402    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxzsp\" (UniqueName: \"kubernetes.io/projected/6d96c715-e042-4e8f-9adf-de115944c84f-kube-api-access-sxzsp\") pod \"kuadrant-console-plugin-66b5774985-vhwhh\" (UID: \"6d96c715-e042-4e8f-9adf-de115944c84f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vhwhh"
Jul 08 12:09:32.817944 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:32.817928    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6d96c715-e042-4e8f-9adf-de115944c84f-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-vhwhh\" (UID: \"6d96c715-e042-4e8f-9adf-de115944c84f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vhwhh"
Jul 08 12:09:32.819737 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:32.819720    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/6d96c715-e042-4e8f-9adf-de115944c84f-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-vhwhh\" (UID: \"6d96c715-e042-4e8f-9adf-de115944c84f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vhwhh"
Jul 08 12:09:32.827854 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:32.827831    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxzsp\" (UniqueName: \"kubernetes.io/projected/6d96c715-e042-4e8f-9adf-de115944c84f-kube-api-access-sxzsp\") pod \"kuadrant-console-plugin-66b5774985-vhwhh\" (UID: \"6d96c715-e042-4e8f-9adf-de115944c84f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vhwhh"
Jul 08 12:09:32.914507 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:32.914426    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vhwhh"
Jul 08 12:09:33.050903 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:33.050878    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-vhwhh"]
Jul 08 12:09:33.053519 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:09:33.053482    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6d96c715_e042_4e8f_9adf_de115944c84f.slice/crio-d9dc89b6da40cf0d896d557466c25c0b6e57ced384b8ed437f104c77c5ca6d1d WatchSource:0}: Error finding container d9dc89b6da40cf0d896d557466c25c0b6e57ced384b8ed437f104c77c5ca6d1d: Status 404 returned error can't find the container with id d9dc89b6da40cf0d896d557466c25c0b6e57ced384b8ed437f104c77c5ca6d1d
Jul 08 12:09:33.522923 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:33.522893    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7xlrf"
Jul 08 12:09:33.579302 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:33.579265    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vhwhh" event={"ID":"6d96c715-e042-4e8f-9adf-de115944c84f","Type":"ContainerStarted","Data":"d9dc89b6da40cf0d896d557466c25c0b6e57ced384b8ed437f104c77c5ca6d1d"}
Jul 08 12:09:38.551844 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:38.551810    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-64lwx"
Jul 08 12:09:40.382080 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:40.382032    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-78887568fd-gnvfk"
Jul 08 12:09:40.382471 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:40.382140    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-78887568fd-gnvfk"
Jul 08 12:09:40.386832 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:40.386810    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-78887568fd-gnvfk"
Jul 08 12:09:40.617977 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:40.617952    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-78887568fd-gnvfk"
Jul 08 12:09:41.571229 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:41.571194    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ls5vt"
Jul 08 12:09:42.575984 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:42.575956    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jtdbf"
Jul 08 12:09:43.117627 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:43.117584    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6hpzh"]
Jul 08 12:09:43.121940 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:43.121913    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6hpzh"
Jul 08 12:09:43.135986 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:43.135957    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6hpzh"]
Jul 08 12:09:43.210748 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:43.210707    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c85xt\" (UniqueName: \"kubernetes.io/projected/072b825f-e79a-4945-bfe5-d0413e90ea22-kube-api-access-c85xt\") pod \"kuadrant-operator-controller-manager-55467d9f7-6hpzh\" (UID: \"072b825f-e79a-4945-bfe5-d0413e90ea22\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6hpzh"
Jul 08 12:09:43.210958 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:43.210881    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/072b825f-e79a-4945-bfe5-d0413e90ea22-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-6hpzh\" (UID: \"072b825f-e79a-4945-bfe5-d0413e90ea22\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6hpzh"
Jul 08 12:09:43.271482 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:43.271444    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6hpzh"]
Jul 08 12:09:43.271834 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:09:43.271811    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-c85xt], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6hpzh" podUID="072b825f-e79a-4945-bfe5-d0413e90ea22"
Jul 08 12:09:43.290793 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:43.290739    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6hpzh"]
Jul 08 12:09:43.310108 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:43.310074    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6hpzh"]
Jul 08 12:09:43.311642 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:43.311610    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/072b825f-e79a-4945-bfe5-d0413e90ea22-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-6hpzh\" (UID: \"072b825f-e79a-4945-bfe5-d0413e90ea22\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6hpzh"
Jul 08 12:09:43.311835 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:43.311713    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c85xt\" (UniqueName: \"kubernetes.io/projected/072b825f-e79a-4945-bfe5-d0413e90ea22-kube-api-access-c85xt\") pod \"kuadrant-operator-controller-manager-55467d9f7-6hpzh\" (UID: \"072b825f-e79a-4945-bfe5-d0413e90ea22\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6hpzh"
Jul 08 12:09:43.312063 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:43.312040    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/072b825f-e79a-4945-bfe5-d0413e90ea22-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-6hpzh\" (UID: \"072b825f-e79a-4945-bfe5-d0413e90ea22\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6hpzh"
Jul 08 12:09:43.317066 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:09:43.317047    2579 projected.go:196] Error preparing data for projected volume kube-api-access-c85xt for pod kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6hpzh: failed to fetch token: pod "kuadrant-operator-controller-manager-55467d9f7-6hpzh" not found
Jul 08 12:09:43.317172 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:09:43.317107    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/072b825f-e79a-4945-bfe5-d0413e90ea22-kube-api-access-c85xt podName:072b825f-e79a-4945-bfe5-d0413e90ea22 nodeName:}" failed. No retries permitted until 2026-07-08 12:09:43.817090845 +0000 UTC m=+473.547465795 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-c85xt" (UniqueName: "kubernetes.io/projected/072b825f-e79a-4945-bfe5-d0413e90ea22-kube-api-access-c85xt") pod "kuadrant-operator-controller-manager-55467d9f7-6hpzh" (UID: "072b825f-e79a-4945-bfe5-d0413e90ea22") : failed to fetch token: pod "kuadrant-operator-controller-manager-55467d9f7-6hpzh" not found
Jul 08 12:09:43.334057 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:43.334017    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ls5vt"]
Jul 08 12:09:43.334266 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:43.334241    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ls5vt" podUID="c64f287a-191f-46a1-aadc-f26cea7292d9" containerName="manager" containerID="cri-o://bb09688dd903927e2e62a075c93760d92a79a08d63fa146dc34df921b972ccc3" gracePeriod=2
Jul 08 12:09:43.352708 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:43.352683    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ls5vt"]
Jul 08 12:09:43.353465 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:09:43.353426    2579 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"c64f287a-191f-46a1-aadc-f26cea7292d9\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-ls5vt\": pods \"kuadrant-operator-controller-manager-8684c9ff49-ls5vt\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ls5vt"
Jul 08 12:09:43.369678 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:43.369601    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jtdbf"]
Jul 08 12:09:43.369850 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:43.369821    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jtdbf" podUID="c0018713-4bd2-4dd6-b090-6cc1e012669e" containerName="manager" containerID="cri-o://79ffb6104d00536d8e4604b21ab732a5523ac536dfcfe2aad71e3a6b2d636960" gracePeriod=2
Jul 08 12:09:43.371463 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:43.371441    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rdrdv"]
Jul 08 12:09:43.371817 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:43.371802    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c64f287a-191f-46a1-aadc-f26cea7292d9" containerName="manager"
Jul 08 12:09:43.371817 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:43.371817    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c64f287a-191f-46a1-aadc-f26cea7292d9" containerName="manager"
Jul 08 12:09:43.371953 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:43.371897    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c64f287a-191f-46a1-aadc-f26cea7292d9" containerName="manager"
Jul 08 12:09:43.375130 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:43.375111    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rdrdv"
Jul 08 12:09:43.381485 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:43.381460    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jtdbf"]
Jul 08 12:09:43.387129 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:09:43.387091    2579 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"c0018713-4bd2-4dd6-b090-6cc1e012669e\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-859d7b78bd-jtdbf\": pods \"limitador-operator-controller-manager-859d7b78bd-jtdbf\" not found" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jtdbf"
Jul 08 12:09:43.389338 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:09:43.389308    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-ls5vt\" is forbidden: User \"system:node:ip-10-0-138-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-89.ec2.internal' and this object" podUID="c64f287a-191f-46a1-aadc-f26cea7292d9" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ls5vt"
Jul 08 12:09:43.391923 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:09:43.391895    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-jtdbf\" is forbidden: User \"system:node:ip-10-0-138-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-89.ec2.internal' and this object" podUID="c0018713-4bd2-4dd6-b090-6cc1e012669e" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jtdbf"
Jul 08 12:09:43.394465 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:09:43.394437    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-ls5vt\" is forbidden: User \"system:node:ip-10-0-138-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-89.ec2.internal' and this object" podUID="c64f287a-191f-46a1-aadc-f26cea7292d9" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ls5vt"
Jul 08 12:09:43.401908 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:43.401824    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rdrdv"]
Jul 08 12:09:43.403745 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:43.403720    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hcwbd"]
Jul 08 12:09:43.404367 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:43.404344    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0018713-4bd2-4dd6-b090-6cc1e012669e" containerName="manager"
Jul 08 12:09:43.404483 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:43.404371    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0018713-4bd2-4dd6-b090-6cc1e012669e" containerName="manager"
Jul 08 12:09:43.404483 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:43.404450    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0018713-4bd2-4dd6-b090-6cc1e012669e" containerName="manager"
Jul 08 12:09:43.407482 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:43.407345    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hcwbd"
Jul 08 12:09:43.413035 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:43.413013    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxxqd\" (UniqueName: \"kubernetes.io/projected/4024b697-1dcc-4c34-b5cd-9b8ef427000a-kube-api-access-xxxqd\") pod \"kuadrant-operator-controller-manager-8684c9ff49-rdrdv\" (UID: \"4024b697-1dcc-4c34-b5cd-9b8ef427000a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rdrdv"
Jul 08 12:09:43.413162 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:43.413140    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4024b697-1dcc-4c34-b5cd-9b8ef427000a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-rdrdv\" (UID: \"4024b697-1dcc-4c34-b5cd-9b8ef427000a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rdrdv"
Jul 08 12:09:43.424077 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:43.424050    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hcwbd"]
Jul 08 12:09:43.434961 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:09:43.434920    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-jtdbf\" is forbidden: User \"system:node:ip-10-0-138-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-89.ec2.internal' and this object" podUID="c0018713-4bd2-4dd6-b090-6cc1e012669e" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jtdbf"
Jul 08 12:09:43.439157 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:09:43.439121    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-ls5vt\" is forbidden: User \"system:node:ip-10-0-138-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-89.ec2.internal' and this object" podUID="c64f287a-191f-46a1-aadc-f26cea7292d9" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ls5vt"
Jul 08 12:09:43.513907 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:43.513704    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxxqd\" (UniqueName: \"kubernetes.io/projected/4024b697-1dcc-4c34-b5cd-9b8ef427000a-kube-api-access-xxxqd\") pod \"kuadrant-operator-controller-manager-8684c9ff49-rdrdv\" (UID: \"4024b697-1dcc-4c34-b5cd-9b8ef427000a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rdrdv"
Jul 08 12:09:43.513907 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:43.513799    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wth8m\" (UniqueName: \"kubernetes.io/projected/46272467-ba68-4376-bd7e-ffed4281cd12-kube-api-access-wth8m\") pod \"limitador-operator-controller-manager-859d7b78bd-hcwbd\" (UID: \"46272467-ba68-4376-bd7e-ffed4281cd12\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hcwbd"
Jul 08 12:09:43.514166 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:43.513960    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4024b697-1dcc-4c34-b5cd-9b8ef427000a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-rdrdv\" (UID: \"4024b697-1dcc-4c34-b5cd-9b8ef427000a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rdrdv"
Jul 08 12:09:43.514505 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:43.514458    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4024b697-1dcc-4c34-b5cd-9b8ef427000a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-rdrdv\" (UID: \"4024b697-1dcc-4c34-b5cd-9b8ef427000a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rdrdv"
Jul 08 12:09:43.525471 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:43.525437    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxxqd\" (UniqueName: \"kubernetes.io/projected/4024b697-1dcc-4c34-b5cd-9b8ef427000a-kube-api-access-xxxqd\") pod \"kuadrant-operator-controller-manager-8684c9ff49-rdrdv\" (UID: \"4024b697-1dcc-4c34-b5cd-9b8ef427000a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rdrdv"
Jul 08 12:09:43.614531 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:43.614496    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wth8m\" (UniqueName: \"kubernetes.io/projected/46272467-ba68-4376-bd7e-ffed4281cd12-kube-api-access-wth8m\") pod \"limitador-operator-controller-manager-859d7b78bd-hcwbd\" (UID: \"46272467-ba68-4376-bd7e-ffed4281cd12\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hcwbd"
Jul 08 12:09:43.634211 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:43.634175    2579 generic.go:356] "Generic (PLEG): container finished" podID="c0018713-4bd2-4dd6-b090-6cc1e012669e" containerID="79ffb6104d00536d8e4604b21ab732a5523ac536dfcfe2aad71e3a6b2d636960" exitCode=0
Jul 08 12:09:43.634351 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:43.634295    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a028f056fe7917edeebeae5e463d07fd70350bb8169de0da6225d8f77a0c60cc"
Jul 08 12:09:43.635937 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:43.635915    2579 generic.go:356] "Generic (PLEG): container finished" podID="c64f287a-191f-46a1-aadc-f26cea7292d9" containerID="bb09688dd903927e2e62a075c93760d92a79a08d63fa146dc34df921b972ccc3" exitCode=0
Jul 08 12:09:43.636112 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:43.635984    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6hpzh"
Jul 08 12:09:43.636112 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:43.636005    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a84242fb7ddcbe52c43dcc772ec806244dff6a3062eed77bb0a7d6c6c3dd79b"
Jul 08 12:09:43.641481 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:09:43.641453    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-ls5vt\" is forbidden: User \"system:node:ip-10-0-138-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-89.ec2.internal' and this object" podUID="c64f287a-191f-46a1-aadc-f26cea7292d9" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ls5vt"
Jul 08 12:09:43.643629 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:43.643609    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ls5vt"
Jul 08 12:09:43.645508 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:09:43.645485    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-6hpzh\" is forbidden: User \"system:node:ip-10-0-138-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-89.ec2.internal' and this object" podUID="072b825f-e79a-4945-bfe5-d0413e90ea22" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6hpzh"
Jul 08 12:09:43.646262 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:43.646243    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wth8m\" (UniqueName: \"kubernetes.io/projected/46272467-ba68-4376-bd7e-ffed4281cd12-kube-api-access-wth8m\") pod \"limitador-operator-controller-manager-859d7b78bd-hcwbd\" (UID: \"46272467-ba68-4376-bd7e-ffed4281cd12\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hcwbd"
Jul 08 12:09:43.647150 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:43.647131    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jtdbf"
Jul 08 12:09:43.647945 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:09:43.647920    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-jtdbf\" is forbidden: User \"system:node:ip-10-0-138-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-89.ec2.internal' and this object" podUID="c0018713-4bd2-4dd6-b090-6cc1e012669e" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jtdbf"
Jul 08 12:09:43.650152 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:43.650137    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6hpzh"
Jul 08 12:09:43.651689 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:09:43.651668    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-jtdbf\" is forbidden: User \"system:node:ip-10-0-138-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-89.ec2.internal' and this object" podUID="c0018713-4bd2-4dd6-b090-6cc1e012669e" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jtdbf"
Jul 08 12:09:43.654318 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:09:43.654297    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-ls5vt\" is forbidden: User \"system:node:ip-10-0-138-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-89.ec2.internal' and this object" podUID="c64f287a-191f-46a1-aadc-f26cea7292d9" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ls5vt"
Jul 08 12:09:43.656531 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:09:43.656512    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-6hpzh\" is forbidden: User \"system:node:ip-10-0-138-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-89.ec2.internal' and this object" podUID="072b825f-e79a-4945-bfe5-d0413e90ea22" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6hpzh"
Jul 08 12:09:43.659446 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:09:43.659429    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-jtdbf\" is forbidden: User \"system:node:ip-10-0-138-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-89.ec2.internal' and this object" podUID="c0018713-4bd2-4dd6-b090-6cc1e012669e" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jtdbf"
Jul 08 12:09:43.663369 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:09:43.663353    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-ls5vt\" is forbidden: User \"system:node:ip-10-0-138-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-89.ec2.internal' and this object" podUID="c64f287a-191f-46a1-aadc-f26cea7292d9" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ls5vt"
Jul 08 12:09:43.666874 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:09:43.666856    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-6hpzh\" is forbidden: User \"system:node:ip-10-0-138-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-89.ec2.internal' and this object" podUID="072b825f-e79a-4945-bfe5-d0413e90ea22" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6hpzh"
Jul 08 12:09:43.715506 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:43.715478    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tchwp\" (UniqueName: \"kubernetes.io/projected/c0018713-4bd2-4dd6-b090-6cc1e012669e-kube-api-access-tchwp\") pod \"c0018713-4bd2-4dd6-b090-6cc1e012669e\" (UID: \"c0018713-4bd2-4dd6-b090-6cc1e012669e\") "
Jul 08 12:09:43.715691 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:43.715529    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b5d2t\" (UniqueName: \"kubernetes.io/projected/c64f287a-191f-46a1-aadc-f26cea7292d9-kube-api-access-b5d2t\") pod \"c64f287a-191f-46a1-aadc-f26cea7292d9\" (UID: \"c64f287a-191f-46a1-aadc-f26cea7292d9\") "
Jul 08 12:09:43.715831 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:43.715746    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/072b825f-e79a-4945-bfe5-d0413e90ea22-extensions-socket-volume\") pod \"072b825f-e79a-4945-bfe5-d0413e90ea22\" (UID: \"072b825f-e79a-4945-bfe5-d0413e90ea22\") "
Jul 08 12:09:43.715934 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:43.715845    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c64f287a-191f-46a1-aadc-f26cea7292d9-extensions-socket-volume\") pod \"c64f287a-191f-46a1-aadc-f26cea7292d9\" (UID: \"c64f287a-191f-46a1-aadc-f26cea7292d9\") "
Jul 08 12:09:43.716071 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:43.716045    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/072b825f-e79a-4945-bfe5-d0413e90ea22-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "072b825f-e79a-4945-bfe5-d0413e90ea22" (UID: "072b825f-e79a-4945-bfe5-d0413e90ea22"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 12:09:43.716174 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:43.716156    2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/072b825f-e79a-4945-bfe5-d0413e90ea22-extensions-socket-volume\") on node \"ip-10-0-138-89.ec2.internal\" DevicePath \"\""
Jul 08 12:09:43.716244 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:43.716179    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c85xt\" (UniqueName: \"kubernetes.io/projected/072b825f-e79a-4945-bfe5-d0413e90ea22-kube-api-access-c85xt\") on node \"ip-10-0-138-89.ec2.internal\" DevicePath \"\""
Jul 08 12:09:43.716539 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:43.716497    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c64f287a-191f-46a1-aadc-f26cea7292d9-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "c64f287a-191f-46a1-aadc-f26cea7292d9" (UID: "c64f287a-191f-46a1-aadc-f26cea7292d9"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 12:09:43.718001 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:43.717976    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0018713-4bd2-4dd6-b090-6cc1e012669e-kube-api-access-tchwp" (OuterVolumeSpecName: "kube-api-access-tchwp") pod "c0018713-4bd2-4dd6-b090-6cc1e012669e" (UID: "c0018713-4bd2-4dd6-b090-6cc1e012669e"). InnerVolumeSpecName "kube-api-access-tchwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 12:09:43.718409 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:43.718386    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c64f287a-191f-46a1-aadc-f26cea7292d9-kube-api-access-b5d2t" (OuterVolumeSpecName: "kube-api-access-b5d2t") pod "c64f287a-191f-46a1-aadc-f26cea7292d9" (UID: "c64f287a-191f-46a1-aadc-f26cea7292d9"). InnerVolumeSpecName "kube-api-access-b5d2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 12:09:43.759608 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:43.759570    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rdrdv"
Jul 08 12:09:43.766737 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:43.766713    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hcwbd"
Jul 08 12:09:43.817857 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:43.817786    2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c64f287a-191f-46a1-aadc-f26cea7292d9-extensions-socket-volume\") on node \"ip-10-0-138-89.ec2.internal\" DevicePath \"\""
Jul 08 12:09:43.817857 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:43.817820    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tchwp\" (UniqueName: \"kubernetes.io/projected/c0018713-4bd2-4dd6-b090-6cc1e012669e-kube-api-access-tchwp\") on node \"ip-10-0-138-89.ec2.internal\" DevicePath \"\""
Jul 08 12:09:43.817857 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:43.817833    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5d2t\" (UniqueName: \"kubernetes.io/projected/c64f287a-191f-46a1-aadc-f26cea7292d9-kube-api-access-b5d2t\") on node \"ip-10-0-138-89.ec2.internal\" DevicePath \"\""
Jul 08 12:09:43.925306 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:43.925285    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rdrdv"]
Jul 08 12:09:43.927735 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:09:43.927711    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4024b697_1dcc_4c34_b5cd_9b8ef427000a.slice/crio-8ac5c84b741c889bf54a19d4f001eed60dee4af3b26efbf9b61cdf182ff810e0 WatchSource:0}: Error finding container 8ac5c84b741c889bf54a19d4f001eed60dee4af3b26efbf9b61cdf182ff810e0: Status 404 returned error can't find the container with id 8ac5c84b741c889bf54a19d4f001eed60dee4af3b26efbf9b61cdf182ff810e0
Jul 08 12:09:43.936486 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:43.936464    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hcwbd"]
Jul 08 12:09:43.938558 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:09:43.938537    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46272467_ba68_4376_bd7e_ffed4281cd12.slice/crio-160d15098d93e1bda19fd03838f249c15b79fe6d70362551d465329d6b204206 WatchSource:0}: Error finding container 160d15098d93e1bda19fd03838f249c15b79fe6d70362551d465329d6b204206: Status 404 returned error can't find the container with id 160d15098d93e1bda19fd03838f249c15b79fe6d70362551d465329d6b204206
Jul 08 12:09:44.644414 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:44.643887    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hcwbd" event={"ID":"46272467-ba68-4376-bd7e-ffed4281cd12","Type":"ContainerStarted","Data":"00ce2b5fc77846ad48f3e119803f7d654c451e9f8b8e7db72f173d618b89fb6c"}
Jul 08 12:09:44.644414 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:44.644053    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hcwbd"
Jul 08 12:09:44.644414 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:44.644094    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hcwbd" event={"ID":"46272467-ba68-4376-bd7e-ffed4281cd12","Type":"ContainerStarted","Data":"160d15098d93e1bda19fd03838f249c15b79fe6d70362551d465329d6b204206"}
Jul 08 12:09:44.646100 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:44.646059    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rdrdv" event={"ID":"4024b697-1dcc-4c34-b5cd-9b8ef427000a","Type":"ContainerStarted","Data":"749386db5fa55e6cafa358d9b591ffaa2a1b8a1e665a0559a5b555e135bd93fc"}
Jul 08 12:09:44.646216 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:44.646149    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rdrdv" event={"ID":"4024b697-1dcc-4c34-b5cd-9b8ef427000a","Type":"ContainerStarted","Data":"8ac5c84b741c889bf54a19d4f001eed60dee4af3b26efbf9b61cdf182ff810e0"}
Jul 08 12:09:44.646349 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:44.646220    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rdrdv"
Jul 08 12:09:44.646349 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:44.646306    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6hpzh"
Jul 08 12:09:44.646470 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:44.646423    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ls5vt"
Jul 08 12:09:44.646669 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:44.646651    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jtdbf"
Jul 08 12:09:44.648136 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:09:44.648111    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-jtdbf\" is forbidden: User \"system:node:ip-10-0-138-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-89.ec2.internal' and this object" podUID="c0018713-4bd2-4dd6-b090-6cc1e012669e" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jtdbf"
Jul 08 12:09:44.653658 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:09:44.653630    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-ls5vt\" is forbidden: User \"system:node:ip-10-0-138-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-89.ec2.internal' and this object" podUID="c64f287a-191f-46a1-aadc-f26cea7292d9" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ls5vt"
Jul 08 12:09:44.679898 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:44.679842    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hcwbd" podStartSLOduration=1.679826599 podStartE2EDuration="1.679826599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 12:09:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 12:09:44.677151731 +0000 UTC m=+474.407526702" watchObservedRunningTime="2026-07-08 12:09:44.679826599 +0000 UTC m=+474.410201569"
Jul 08 12:09:44.680466 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:09:44.680441    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-6hpzh\" is forbidden: User \"system:node:ip-10-0-138-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-89.ec2.internal' and this object" podUID="072b825f-e79a-4945-bfe5-d0413e90ea22" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6hpzh"
Jul 08 12:09:44.683480 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:09:44.683446    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-jtdbf\" is forbidden: User \"system:node:ip-10-0-138-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-89.ec2.internal' and this object" podUID="c0018713-4bd2-4dd6-b090-6cc1e012669e" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jtdbf"
Jul 08 12:09:44.691021 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:09:44.690984    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-ls5vt\" is forbidden: User \"system:node:ip-10-0-138-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-89.ec2.internal' and this object" podUID="c64f287a-191f-46a1-aadc-f26cea7292d9" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ls5vt"
Jul 08 12:09:44.693178 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:09:44.693152    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-6hpzh\" is forbidden: User \"system:node:ip-10-0-138-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-89.ec2.internal' and this object" podUID="072b825f-e79a-4945-bfe5-d0413e90ea22" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6hpzh"
Jul 08 12:09:44.718846 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:44.718811    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="072b825f-e79a-4945-bfe5-d0413e90ea22" path="/var/lib/kubelet/pods/072b825f-e79a-4945-bfe5-d0413e90ea22/volumes"
Jul 08 12:09:44.719131 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:44.719116    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0018713-4bd2-4dd6-b090-6cc1e012669e" path="/var/lib/kubelet/pods/c0018713-4bd2-4dd6-b090-6cc1e012669e/volumes"
Jul 08 12:09:44.719603 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:44.719585    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c64f287a-191f-46a1-aadc-f26cea7292d9" path="/var/lib/kubelet/pods/c64f287a-191f-46a1-aadc-f26cea7292d9/volumes"
Jul 08 12:09:44.730913 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:44.730863    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rdrdv" podStartSLOduration=1.730841946 podStartE2EDuration="1.730841946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 12:09:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 12:09:44.729654888 +0000 UTC m=+474.460029861" watchObservedRunningTime="2026-07-08 12:09:44.730841946 +0000 UTC m=+474.461216920"
Jul 08 12:09:55.653518 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:55.653471    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rdrdv"
Jul 08 12:09:55.653967 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:55.653646    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hcwbd"
Jul 08 12:09:58.722563 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:58.722533    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vhwhh" event={"ID":"6d96c715-e042-4e8f-9adf-de115944c84f","Type":"ContainerStarted","Data":"9f10bd7c949722cd1a42b6a6ed82f8d5ef220a770fc84f0f449cd4286c1d33c6"}
Jul 08 12:09:58.744603 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:09:58.744552    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vhwhh" podStartSLOduration=1.6241373650000002 podStartE2EDuration="26.74453789s" totalImagesPullingTime="25.120400525s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 12:09:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 12:09:58.74264999 +0000 UTC m=+488.473024962" watchObservedRunningTime="2026-07-08 12:09:58.74453789 +0000 UTC m=+488.474912860"
Jul 08 12:10:02.936139 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:02.936105    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rdrdv"]
Jul 08 12:10:02.936674 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:02.936340    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rdrdv" podUID="4024b697-1dcc-4c34-b5cd-9b8ef427000a" containerName="manager" containerID="cri-o://749386db5fa55e6cafa358d9b591ffaa2a1b8a1e665a0559a5b555e135bd93fc" gracePeriod=10
Jul 08 12:10:03.179538 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:03.179512    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rdrdv"
Jul 08 12:10:03.294828 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:03.294709    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4024b697-1dcc-4c34-b5cd-9b8ef427000a-extensions-socket-volume\") pod \"4024b697-1dcc-4c34-b5cd-9b8ef427000a\" (UID: \"4024b697-1dcc-4c34-b5cd-9b8ef427000a\") "
Jul 08 12:10:03.295010 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:03.294863    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xxxqd\" (UniqueName: \"kubernetes.io/projected/4024b697-1dcc-4c34-b5cd-9b8ef427000a-kube-api-access-xxxqd\") pod \"4024b697-1dcc-4c34-b5cd-9b8ef427000a\" (UID: \"4024b697-1dcc-4c34-b5cd-9b8ef427000a\") "
Jul 08 12:10:03.295108 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:03.295083    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4024b697-1dcc-4c34-b5cd-9b8ef427000a-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "4024b697-1dcc-4c34-b5cd-9b8ef427000a" (UID: "4024b697-1dcc-4c34-b5cd-9b8ef427000a"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 12:10:03.297012 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:03.296983    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4024b697-1dcc-4c34-b5cd-9b8ef427000a-kube-api-access-xxxqd" (OuterVolumeSpecName: "kube-api-access-xxxqd") pod "4024b697-1dcc-4c34-b5cd-9b8ef427000a" (UID: "4024b697-1dcc-4c34-b5cd-9b8ef427000a"). InnerVolumeSpecName "kube-api-access-xxxqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 12:10:03.395682 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:03.395646    2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4024b697-1dcc-4c34-b5cd-9b8ef427000a-extensions-socket-volume\") on node \"ip-10-0-138-89.ec2.internal\" DevicePath \"\""
Jul 08 12:10:03.395682 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:03.395674    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxxqd\" (UniqueName: \"kubernetes.io/projected/4024b697-1dcc-4c34-b5cd-9b8ef427000a-kube-api-access-xxxqd\") on node \"ip-10-0-138-89.ec2.internal\" DevicePath \"\""
Jul 08 12:10:03.749851 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:03.749814    2579 generic.go:356] "Generic (PLEG): container finished" podID="4024b697-1dcc-4c34-b5cd-9b8ef427000a" containerID="749386db5fa55e6cafa358d9b591ffaa2a1b8a1e665a0559a5b555e135bd93fc" exitCode=0
Jul 08 12:10:03.750037 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:03.749866    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rdrdv"
Jul 08 12:10:03.750037 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:03.749898    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rdrdv" event={"ID":"4024b697-1dcc-4c34-b5cd-9b8ef427000a","Type":"ContainerDied","Data":"749386db5fa55e6cafa358d9b591ffaa2a1b8a1e665a0559a5b555e135bd93fc"}
Jul 08 12:10:03.750037 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:03.749952    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rdrdv" event={"ID":"4024b697-1dcc-4c34-b5cd-9b8ef427000a","Type":"ContainerDied","Data":"8ac5c84b741c889bf54a19d4f001eed60dee4af3b26efbf9b61cdf182ff810e0"}
Jul 08 12:10:03.750037 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:03.749974    2579 scope.go:117] "RemoveContainer" containerID="749386db5fa55e6cafa358d9b591ffaa2a1b8a1e665a0559a5b555e135bd93fc"
Jul 08 12:10:03.764162 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:03.764145    2579 scope.go:117] "RemoveContainer" containerID="749386db5fa55e6cafa358d9b591ffaa2a1b8a1e665a0559a5b555e135bd93fc"
Jul 08 12:10:03.764436 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:10:03.764418    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"749386db5fa55e6cafa358d9b591ffaa2a1b8a1e665a0559a5b555e135bd93fc\": container with ID starting with 749386db5fa55e6cafa358d9b591ffaa2a1b8a1e665a0559a5b555e135bd93fc not found: ID does not exist" containerID="749386db5fa55e6cafa358d9b591ffaa2a1b8a1e665a0559a5b555e135bd93fc"
Jul 08 12:10:03.764485 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:03.764443    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"749386db5fa55e6cafa358d9b591ffaa2a1b8a1e665a0559a5b555e135bd93fc"} err="failed to get container status \"749386db5fa55e6cafa358d9b591ffaa2a1b8a1e665a0559a5b555e135bd93fc\": rpc error: code = NotFound desc = could not find container \"749386db5fa55e6cafa358d9b591ffaa2a1b8a1e665a0559a5b555e135bd93fc\": container with ID starting with 749386db5fa55e6cafa358d9b591ffaa2a1b8a1e665a0559a5b555e135bd93fc not found: ID does not exist"
Jul 08 12:10:03.783652 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:03.783626    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rdrdv"]
Jul 08 12:10:03.791646 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:03.791623    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rdrdv"]
Jul 08 12:10:04.716537 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:04.716507    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4024b697-1dcc-4c34-b5cd-9b8ef427000a" path="/var/lib/kubelet/pods/4024b697-1dcc-4c34-b5cd-9b8ef427000a/volumes"
Jul 08 12:10:21.207914 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:21.207878    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-4jltv"]
Jul 08 12:10:21.208381 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:21.208202    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4024b697-1dcc-4c34-b5cd-9b8ef427000a" containerName="manager"
Jul 08 12:10:21.208381 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:21.208212    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4024b697-1dcc-4c34-b5cd-9b8ef427000a" containerName="manager"
Jul 08 12:10:21.208381 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:21.208276    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4024b697-1dcc-4c34-b5cd-9b8ef427000a" containerName="manager"
Jul 08 12:10:21.263977 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:21.263936    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-4jltv"]
Jul 08 12:10:21.264160 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:21.264063    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-4jltv"
Jul 08 12:10:21.266634 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:21.266612    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 08 12:10:21.309544 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:21.309513    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-4jltv"]
Jul 08 12:10:21.344076 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:21.344043    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1b9e8946-60e1-459d-9b23-9d7f75290e87-config-file\") pod \"limitador-limitador-64c94fd64f-4jltv\" (UID: \"1b9e8946-60e1-459d-9b23-9d7f75290e87\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-4jltv"
Jul 08 12:10:21.344244 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:21.344093    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qsdg\" (UniqueName: \"kubernetes.io/projected/1b9e8946-60e1-459d-9b23-9d7f75290e87-kube-api-access-8qsdg\") pod \"limitador-limitador-64c94fd64f-4jltv\" (UID: \"1b9e8946-60e1-459d-9b23-9d7f75290e87\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-4jltv"
Jul 08 12:10:21.445141 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:21.445106    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1b9e8946-60e1-459d-9b23-9d7f75290e87-config-file\") pod \"limitador-limitador-64c94fd64f-4jltv\" (UID: \"1b9e8946-60e1-459d-9b23-9d7f75290e87\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-4jltv"
Jul 08 12:10:21.445391 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:21.445163    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qsdg\" (UniqueName: \"kubernetes.io/projected/1b9e8946-60e1-459d-9b23-9d7f75290e87-kube-api-access-8qsdg\") pod \"limitador-limitador-64c94fd64f-4jltv\" (UID: \"1b9e8946-60e1-459d-9b23-9d7f75290e87\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-4jltv"
Jul 08 12:10:21.445721 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:21.445699    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1b9e8946-60e1-459d-9b23-9d7f75290e87-config-file\") pod \"limitador-limitador-64c94fd64f-4jltv\" (UID: \"1b9e8946-60e1-459d-9b23-9d7f75290e87\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-4jltv"
Jul 08 12:10:21.454959 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:21.454937    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qsdg\" (UniqueName: \"kubernetes.io/projected/1b9e8946-60e1-459d-9b23-9d7f75290e87-kube-api-access-8qsdg\") pod \"limitador-limitador-64c94fd64f-4jltv\" (UID: \"1b9e8946-60e1-459d-9b23-9d7f75290e87\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-4jltv"
Jul 08 12:10:21.576982 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:21.576886    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-4jltv"
Jul 08 12:10:21.689661 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:21.689296    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:10:21.729295 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:21.729264    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:10:21.729437 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:21.729322    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:10:21.729437 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:21.729346    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-4jltv"]
Jul 08 12:10:21.729437 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:21.729410    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-gkcph"
Jul 08 12:10:21.732083 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:10:21.732054    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b9e8946_60e1_459d_9b23_9d7f75290e87.slice/crio-a65978e8ef0eb91d04d63b7fe0bbedce3c5db82d36550569f4620f4fa4e644e5 WatchSource:0}: Error finding container a65978e8ef0eb91d04d63b7fe0bbedce3c5db82d36550569f4620f4fa4e644e5: Status 404 returned error can't find the container with id a65978e8ef0eb91d04d63b7fe0bbedce3c5db82d36550569f4620f4fa4e644e5
Jul 08 12:10:21.747465 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:21.747439    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqc8x\" (UniqueName: \"kubernetes.io/projected/13eeecbc-6a72-4f7b-8d26-99e73d820a30-kube-api-access-wqc8x\") pod \"limitador-limitador-54d6d988d5-gkcph\" (UID: \"13eeecbc-6a72-4f7b-8d26-99e73d820a30\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-gkcph"
Jul 08 12:10:21.747575 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:21.747492    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/13eeecbc-6a72-4f7b-8d26-99e73d820a30-config-file\") pod \"limitador-limitador-54d6d988d5-gkcph\" (UID: \"13eeecbc-6a72-4f7b-8d26-99e73d820a30\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-gkcph"
Jul 08 12:10:21.838429 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:21.838349    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-4jltv" event={"ID":"1b9e8946-60e1-459d-9b23-9d7f75290e87","Type":"ContainerStarted","Data":"a65978e8ef0eb91d04d63b7fe0bbedce3c5db82d36550569f4620f4fa4e644e5"}
Jul 08 12:10:21.848741 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:21.848714    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/13eeecbc-6a72-4f7b-8d26-99e73d820a30-config-file\") pod \"limitador-limitador-54d6d988d5-gkcph\" (UID: \"13eeecbc-6a72-4f7b-8d26-99e73d820a30\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-gkcph"
Jul 08 12:10:21.848897 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:21.848787    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqc8x\" (UniqueName: \"kubernetes.io/projected/13eeecbc-6a72-4f7b-8d26-99e73d820a30-kube-api-access-wqc8x\") pod \"limitador-limitador-54d6d988d5-gkcph\" (UID: \"13eeecbc-6a72-4f7b-8d26-99e73d820a30\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-gkcph"
Jul 08 12:10:21.849396 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:21.849375    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/13eeecbc-6a72-4f7b-8d26-99e73d820a30-config-file\") pod \"limitador-limitador-54d6d988d5-gkcph\" (UID: \"13eeecbc-6a72-4f7b-8d26-99e73d820a30\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-gkcph"
Jul 08 12:10:21.867291 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:21.867262    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqc8x\" (UniqueName: \"kubernetes.io/projected/13eeecbc-6a72-4f7b-8d26-99e73d820a30-kube-api-access-wqc8x\") pod \"limitador-limitador-54d6d988d5-gkcph\" (UID: \"13eeecbc-6a72-4f7b-8d26-99e73d820a30\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-gkcph"
Jul 08 12:10:22.048487 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:22.048447    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-gkcph"
Jul 08 12:10:22.204045 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:22.204018    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:10:22.204632 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:10:22.204611    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13eeecbc_6a72_4f7b_8d26_99e73d820a30.slice/crio-226cb38a1c5d45d48bc6bec8cc2a09a619faec78b4f24bee4af69fca379ede68 WatchSource:0}: Error finding container 226cb38a1c5d45d48bc6bec8cc2a09a619faec78b4f24bee4af69fca379ede68: Status 404 returned error can't find the container with id 226cb38a1c5d45d48bc6bec8cc2a09a619faec78b4f24bee4af69fca379ede68
Jul 08 12:10:22.845686 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:22.845649    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-gkcph" event={"ID":"13eeecbc-6a72-4f7b-8d26-99e73d820a30","Type":"ContainerStarted","Data":"226cb38a1c5d45d48bc6bec8cc2a09a619faec78b4f24bee4af69fca379ede68"}
Jul 08 12:10:25.864556 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:25.864455    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-4jltv" event={"ID":"1b9e8946-60e1-459d-9b23-9d7f75290e87","Type":"ContainerStarted","Data":"d8cc54cf4f4db09858b9e4f3417b37a11155274590a7481c954896029095c920"}
Jul 08 12:10:25.864969 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:25.864621    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-4jltv"
Jul 08 12:10:25.866040 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:25.866021    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-gkcph" event={"ID":"13eeecbc-6a72-4f7b-8d26-99e73d820a30","Type":"ContainerStarted","Data":"cb0464d062f26adc396f7cc7958c84208cd2c5f51db3e8327bb47f5d786b215d"}
Jul 08 12:10:25.866132 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:25.866073    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-gkcph"
Jul 08 12:10:25.883717 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:25.883658    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-4jltv" podStartSLOduration=1.018512026 podStartE2EDuration="4.883640408s" totalImagesPullingTime="3.865128382s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 12:10:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 12:10:25.882555461 +0000 UTC m=+515.612930432" watchObservedRunningTime="2026-07-08 12:10:25.883640408 +0000 UTC m=+515.614015381"
Jul 08 12:10:25.901167 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:25.901117    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-gkcph" podStartSLOduration=1.5015210940000001 podStartE2EDuration="4.901102241s" totalImagesPullingTime="3.399581147s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 12:10:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 12:10:25.899981465 +0000 UTC m=+515.630356437" watchObservedRunningTime="2026-07-08 12:10:25.901102241 +0000 UTC m=+515.631477211"
Jul 08 12:10:36.872689 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:36.872653    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-4jltv"
Jul 08 12:10:36.873160 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:36.872836    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-gkcph"
Jul 08 12:10:36.951752 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:36.951710    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-4jltv"]
Jul 08 12:10:36.951978 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:36.951953    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-4jltv" podUID="1b9e8946-60e1-459d-9b23-9d7f75290e87" containerName="limitador" containerID="cri-o://d8cc54cf4f4db09858b9e4f3417b37a11155274590a7481c954896029095c920" gracePeriod=30
Jul 08 12:10:37.499076 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:37.499053    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-4jltv"
Jul 08 12:10:37.589087 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:37.588989    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1b9e8946-60e1-459d-9b23-9d7f75290e87-config-file\") pod \"1b9e8946-60e1-459d-9b23-9d7f75290e87\" (UID: \"1b9e8946-60e1-459d-9b23-9d7f75290e87\") "
Jul 08 12:10:37.589087 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:37.589032    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8qsdg\" (UniqueName: \"kubernetes.io/projected/1b9e8946-60e1-459d-9b23-9d7f75290e87-kube-api-access-8qsdg\") pod \"1b9e8946-60e1-459d-9b23-9d7f75290e87\" (UID: \"1b9e8946-60e1-459d-9b23-9d7f75290e87\") "
Jul 08 12:10:37.589375 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:37.589350    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b9e8946-60e1-459d-9b23-9d7f75290e87-config-file" (OuterVolumeSpecName: "config-file") pod "1b9e8946-60e1-459d-9b23-9d7f75290e87" (UID: "1b9e8946-60e1-459d-9b23-9d7f75290e87"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 12:10:37.591161 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:37.591143    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b9e8946-60e1-459d-9b23-9d7f75290e87-kube-api-access-8qsdg" (OuterVolumeSpecName: "kube-api-access-8qsdg") pod "1b9e8946-60e1-459d-9b23-9d7f75290e87" (UID: "1b9e8946-60e1-459d-9b23-9d7f75290e87"). InnerVolumeSpecName "kube-api-access-8qsdg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 12:10:37.689818 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:37.689745    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qsdg\" (UniqueName: \"kubernetes.io/projected/1b9e8946-60e1-459d-9b23-9d7f75290e87-kube-api-access-8qsdg\") on node \"ip-10-0-138-89.ec2.internal\" DevicePath \"\""
Jul 08 12:10:37.689818 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:37.689813    2579 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1b9e8946-60e1-459d-9b23-9d7f75290e87-config-file\") on node \"ip-10-0-138-89.ec2.internal\" DevicePath \"\""
Jul 08 12:10:37.928346 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:37.928314    2579 generic.go:356] "Generic (PLEG): container finished" podID="1b9e8946-60e1-459d-9b23-9d7f75290e87" containerID="d8cc54cf4f4db09858b9e4f3417b37a11155274590a7481c954896029095c920" exitCode=0
Jul 08 12:10:37.928827 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:37.928359    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-4jltv"
Jul 08 12:10:37.928827 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:37.928397    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-4jltv" event={"ID":"1b9e8946-60e1-459d-9b23-9d7f75290e87","Type":"ContainerDied","Data":"d8cc54cf4f4db09858b9e4f3417b37a11155274590a7481c954896029095c920"}
Jul 08 12:10:37.928827 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:37.928458    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-4jltv" event={"ID":"1b9e8946-60e1-459d-9b23-9d7f75290e87","Type":"ContainerDied","Data":"a65978e8ef0eb91d04d63b7fe0bbedce3c5db82d36550569f4620f4fa4e644e5"}
Jul 08 12:10:37.928827 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:37.928495    2579 scope.go:117] "RemoveContainer" containerID="d8cc54cf4f4db09858b9e4f3417b37a11155274590a7481c954896029095c920"
Jul 08 12:10:37.939299 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:37.939248    2579 scope.go:117] "RemoveContainer" containerID="d8cc54cf4f4db09858b9e4f3417b37a11155274590a7481c954896029095c920"
Jul 08 12:10:37.939695 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:10:37.939673    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8cc54cf4f4db09858b9e4f3417b37a11155274590a7481c954896029095c920\": container with ID starting with d8cc54cf4f4db09858b9e4f3417b37a11155274590a7481c954896029095c920 not found: ID does not exist" containerID="d8cc54cf4f4db09858b9e4f3417b37a11155274590a7481c954896029095c920"
Jul 08 12:10:37.939797 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:37.939710    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8cc54cf4f4db09858b9e4f3417b37a11155274590a7481c954896029095c920"} err="failed to get container status \"d8cc54cf4f4db09858b9e4f3417b37a11155274590a7481c954896029095c920\": rpc error: code = NotFound desc = could not find container \"d8cc54cf4f4db09858b9e4f3417b37a11155274590a7481c954896029095c920\": container with ID starting with d8cc54cf4f4db09858b9e4f3417b37a11155274590a7481c954896029095c920 not found: ID does not exist"
Jul 08 12:10:37.958108 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:37.958085    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-4jltv"]
Jul 08 12:10:37.960245 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:37.960225    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-4jltv"]
Jul 08 12:10:38.716003 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:38.715971    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b9e8946-60e1-459d-9b23-9d7f75290e87" path="/var/lib/kubelet/pods/1b9e8946-60e1-459d-9b23-9d7f75290e87/volumes"
Jul 08 12:10:42.830296 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:42.830214    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-phqr7"]
Jul 08 12:10:42.830766 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:42.830562    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b9e8946-60e1-459d-9b23-9d7f75290e87" containerName="limitador"
Jul 08 12:10:42.830766 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:42.830572    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b9e8946-60e1-459d-9b23-9d7f75290e87" containerName="limitador"
Jul 08 12:10:42.830766 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:42.830618    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b9e8946-60e1-459d-9b23-9d7f75290e87" containerName="limitador"
Jul 08 12:10:42.836009 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:42.835989    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-phqr7"
Jul 08 12:10:42.838842 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:42.838822    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 08 12:10:42.838947 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:42.838822    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Jul 08 12:10:42.839213 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:42.839201    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 08 12:10:42.839905 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:42.839890    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-hmzbn\""
Jul 08 12:10:42.856472 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:42.856444    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-phqr7"]
Jul 08 12:10:42.936082 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:42.936046    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/22a80d6e-25a2-4eb4-b647-895dae1d9ee9-data\") pod \"postgres-69fb4594cf-phqr7\" (UID: \"22a80d6e-25a2-4eb4-b647-895dae1d9ee9\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-phqr7"
Jul 08 12:10:42.936267 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:42.936122    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lscgz\" (UniqueName: \"kubernetes.io/projected/22a80d6e-25a2-4eb4-b647-895dae1d9ee9-kube-api-access-lscgz\") pod \"postgres-69fb4594cf-phqr7\" (UID: \"22a80d6e-25a2-4eb4-b647-895dae1d9ee9\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-phqr7"
Jul 08 12:10:43.036981 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:43.036945    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/22a80d6e-25a2-4eb4-b647-895dae1d9ee9-data\") pod \"postgres-69fb4594cf-phqr7\" (UID: \"22a80d6e-25a2-4eb4-b647-895dae1d9ee9\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-phqr7"
Jul 08 12:10:43.037169 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:43.037019    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lscgz\" (UniqueName: \"kubernetes.io/projected/22a80d6e-25a2-4eb4-b647-895dae1d9ee9-kube-api-access-lscgz\") pod \"postgres-69fb4594cf-phqr7\" (UID: \"22a80d6e-25a2-4eb4-b647-895dae1d9ee9\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-phqr7"
Jul 08 12:10:43.037350 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:43.037330    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/22a80d6e-25a2-4eb4-b647-895dae1d9ee9-data\") pod \"postgres-69fb4594cf-phqr7\" (UID: \"22a80d6e-25a2-4eb4-b647-895dae1d9ee9\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-phqr7"
Jul 08 12:10:43.047141 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:43.047106    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lscgz\" (UniqueName: \"kubernetes.io/projected/22a80d6e-25a2-4eb4-b647-895dae1d9ee9-kube-api-access-lscgz\") pod \"postgres-69fb4594cf-phqr7\" (UID: \"22a80d6e-25a2-4eb4-b647-895dae1d9ee9\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-phqr7"
Jul 08 12:10:43.146106 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:43.146007    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-phqr7"
Jul 08 12:10:43.278291 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:43.278268    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-phqr7"]
Jul 08 12:10:43.280062 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:10:43.280031    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22a80d6e_25a2_4eb4_b647_895dae1d9ee9.slice/crio-fbc6673eb1661ebe9e50cbca13a1db957c2b526e65fe89c2579ebb879e8cac2e WatchSource:0}: Error finding container fbc6673eb1661ebe9e50cbca13a1db957c2b526e65fe89c2579ebb879e8cac2e: Status 404 returned error can't find the container with id fbc6673eb1661ebe9e50cbca13a1db957c2b526e65fe89c2579ebb879e8cac2e
Jul 08 12:10:43.957813 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:43.957764    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-phqr7" event={"ID":"22a80d6e-25a2-4eb4-b647-895dae1d9ee9","Type":"ContainerStarted","Data":"fbc6673eb1661ebe9e50cbca13a1db957c2b526e65fe89c2579ebb879e8cac2e"}
Jul 08 12:10:48.983851 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:48.983803    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-phqr7" event={"ID":"22a80d6e-25a2-4eb4-b647-895dae1d9ee9","Type":"ContainerStarted","Data":"324cb1ea6737a11ee2596336c30f246691d76c10efc63922b9521cef54a45a61"}
Jul 08 12:10:48.984253 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:48.983879    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-phqr7"
Jul 08 12:10:49.005313 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:49.005263    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-phqr7" podStartSLOduration=2.31332202 podStartE2EDuration="7.005248878s" totalImagesPullingTime="4.691926858s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 12:10:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 12:10:49.00402991 +0000 UTC m=+538.734404880" watchObservedRunningTime="2026-07-08 12:10:49.005248878 +0000 UTC m=+538.735623849"
Jul 08 12:10:55.017932 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:10:55.017903    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-phqr7"
Jul 08 12:11:02.192684 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:02.192649    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7bbd6859f5-9ccqc"]
Jul 08 12:11:02.213147 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:02.213119    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7bbd6859f5-9ccqc"]
Jul 08 12:11:02.213326 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:02.213246    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bbd6859f5-9ccqc"
Jul 08 12:11:02.216107 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:02.216079    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-25m4d\""
Jul 08 12:11:02.216107 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:02.216092    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 08 12:11:02.217128 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:02.217100    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 08 12:11:02.313567 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:02.313526    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9557t\" (UniqueName: \"kubernetes.io/projected/12f93142-6df7-4888-9504-72347e2bed1e-kube-api-access-9557t\") pod \"maas-controller-7bbd6859f5-9ccqc\" (UID: \"12f93142-6df7-4888-9504-72347e2bed1e\") " pod="opendatahub/maas-controller-7bbd6859f5-9ccqc"
Jul 08 12:11:02.313752 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:02.313578    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/12f93142-6df7-4888-9504-72347e2bed1e-webhook-cert\") pod \"maas-controller-7bbd6859f5-9ccqc\" (UID: \"12f93142-6df7-4888-9504-72347e2bed1e\") " pod="opendatahub/maas-controller-7bbd6859f5-9ccqc"
Jul 08 12:11:02.390046 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:02.390017    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-858769f6f8-vgwvn"]
Jul 08 12:11:02.392329 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:02.392313    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-858769f6f8-vgwvn"
Jul 08 12:11:02.403508 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:02.403484    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-858769f6f8-vgwvn"]
Jul 08 12:11:02.414112 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:02.414082    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9557t\" (UniqueName: \"kubernetes.io/projected/12f93142-6df7-4888-9504-72347e2bed1e-kube-api-access-9557t\") pod \"maas-controller-7bbd6859f5-9ccqc\" (UID: \"12f93142-6df7-4888-9504-72347e2bed1e\") " pod="opendatahub/maas-controller-7bbd6859f5-9ccqc"
Jul 08 12:11:02.414226 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:02.414125    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/12f93142-6df7-4888-9504-72347e2bed1e-webhook-cert\") pod \"maas-controller-7bbd6859f5-9ccqc\" (UID: \"12f93142-6df7-4888-9504-72347e2bed1e\") " pod="opendatahub/maas-controller-7bbd6859f5-9ccqc"
Jul 08 12:11:02.416718 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:02.416696    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/12f93142-6df7-4888-9504-72347e2bed1e-webhook-cert\") pod \"maas-controller-7bbd6859f5-9ccqc\" (UID: \"12f93142-6df7-4888-9504-72347e2bed1e\") " pod="opendatahub/maas-controller-7bbd6859f5-9ccqc"
Jul 08 12:11:02.426665 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:02.426644    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9557t\" (UniqueName: \"kubernetes.io/projected/12f93142-6df7-4888-9504-72347e2bed1e-kube-api-access-9557t\") pod \"maas-controller-7bbd6859f5-9ccqc\" (UID: \"12f93142-6df7-4888-9504-72347e2bed1e\") " pod="opendatahub/maas-controller-7bbd6859f5-9ccqc"
Jul 08 12:11:02.515055 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:02.514971    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw9dl\" (UniqueName: \"kubernetes.io/projected/72b4701f-eb6c-47e6-abfd-cfbe250778ab-kube-api-access-dw9dl\") pod \"maas-controller-858769f6f8-vgwvn\" (UID: \"72b4701f-eb6c-47e6-abfd-cfbe250778ab\") " pod="opendatahub/maas-controller-858769f6f8-vgwvn"
Jul 08 12:11:02.515211 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:02.515067    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/72b4701f-eb6c-47e6-abfd-cfbe250778ab-webhook-cert\") pod \"maas-controller-858769f6f8-vgwvn\" (UID: \"72b4701f-eb6c-47e6-abfd-cfbe250778ab\") " pod="opendatahub/maas-controller-858769f6f8-vgwvn"
Jul 08 12:11:02.524852 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:02.524829    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bbd6859f5-9ccqc"
Jul 08 12:11:02.616129 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:02.616099    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw9dl\" (UniqueName: \"kubernetes.io/projected/72b4701f-eb6c-47e6-abfd-cfbe250778ab-kube-api-access-dw9dl\") pod \"maas-controller-858769f6f8-vgwvn\" (UID: \"72b4701f-eb6c-47e6-abfd-cfbe250778ab\") " pod="opendatahub/maas-controller-858769f6f8-vgwvn"
Jul 08 12:11:02.616300 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:02.616190    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/72b4701f-eb6c-47e6-abfd-cfbe250778ab-webhook-cert\") pod \"maas-controller-858769f6f8-vgwvn\" (UID: \"72b4701f-eb6c-47e6-abfd-cfbe250778ab\") " pod="opendatahub/maas-controller-858769f6f8-vgwvn"
Jul 08 12:11:02.619304 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:02.619275    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/72b4701f-eb6c-47e6-abfd-cfbe250778ab-webhook-cert\") pod \"maas-controller-858769f6f8-vgwvn\" (UID: \"72b4701f-eb6c-47e6-abfd-cfbe250778ab\") " pod="opendatahub/maas-controller-858769f6f8-vgwvn"
Jul 08 12:11:02.629008 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:02.628536    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw9dl\" (UniqueName: \"kubernetes.io/projected/72b4701f-eb6c-47e6-abfd-cfbe250778ab-kube-api-access-dw9dl\") pod \"maas-controller-858769f6f8-vgwvn\" (UID: \"72b4701f-eb6c-47e6-abfd-cfbe250778ab\") " pod="opendatahub/maas-controller-858769f6f8-vgwvn"
Jul 08 12:11:02.661648 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:02.661625    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7bbd6859f5-9ccqc"]
Jul 08 12:11:02.663897 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:11:02.663865    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12f93142_6df7_4888_9504_72347e2bed1e.slice/crio-6d504baccf9c8960d2f14b4ff6cc3d95ba597d3e4711f08ba5849d0b154f5fe6 WatchSource:0}: Error finding container 6d504baccf9c8960d2f14b4ff6cc3d95ba597d3e4711f08ba5849d0b154f5fe6: Status 404 returned error can't find the container with id 6d504baccf9c8960d2f14b4ff6cc3d95ba597d3e4711f08ba5849d0b154f5fe6
Jul 08 12:11:02.702360 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:02.702328    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-858769f6f8-vgwvn"
Jul 08 12:11:02.866531 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:02.866496    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-858769f6f8-vgwvn"]
Jul 08 12:11:02.869688 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:11:02.869659    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72b4701f_eb6c_47e6_abfd_cfbe250778ab.slice/crio-c1d3643dc8082493096908f6ef5059a017080d24fc483c7fdd757ac77218081a WatchSource:0}: Error finding container c1d3643dc8082493096908f6ef5059a017080d24fc483c7fdd757ac77218081a: Status 404 returned error can't find the container with id c1d3643dc8082493096908f6ef5059a017080d24fc483c7fdd757ac77218081a
Jul 08 12:11:03.050810 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:03.050699    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bbd6859f5-9ccqc" event={"ID":"12f93142-6df7-4888-9504-72347e2bed1e","Type":"ContainerStarted","Data":"6d504baccf9c8960d2f14b4ff6cc3d95ba597d3e4711f08ba5849d0b154f5fe6"}
Jul 08 12:11:03.051742 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:03.051723    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-858769f6f8-vgwvn" event={"ID":"72b4701f-eb6c-47e6-abfd-cfbe250778ab","Type":"ContainerStarted","Data":"c1d3643dc8082493096908f6ef5059a017080d24fc483c7fdd757ac77218081a"}
Jul 08 12:11:06.071854 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:06.071811    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bbd6859f5-9ccqc" event={"ID":"12f93142-6df7-4888-9504-72347e2bed1e","Type":"ContainerStarted","Data":"195af9f354459d589d4433062562a78e184c395a07c034989d4b6ca5282ba2e6"}
Jul 08 12:11:06.072364 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:06.071955    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7bbd6859f5-9ccqc"
Jul 08 12:11:06.073653 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:06.073629    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-858769f6f8-vgwvn" event={"ID":"72b4701f-eb6c-47e6-abfd-cfbe250778ab","Type":"ContainerStarted","Data":"f7671ee1098beff1a21d9efd6e412599197bb72766a3040514249b9d6a0ce64f"}
Jul 08 12:11:06.073797 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:06.073678    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-858769f6f8-vgwvn"
Jul 08 12:11:06.092714 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:06.092665    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7bbd6859f5-9ccqc" podStartSLOduration=1.458335991 podStartE2EDuration="4.092650957s" totalImagesPullingTime="2.634314966s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 12:11:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 12:11:06.090909939 +0000 UTC m=+555.821284923" watchObservedRunningTime="2026-07-08 12:11:06.092650957 +0000 UTC m=+555.823025928"
Jul 08 12:11:06.115876 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:06.115825    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-858769f6f8-vgwvn" podStartSLOduration=1.680230027 podStartE2EDuration="4.115805244s" totalImagesPullingTime="2.435575217s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 12:11:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 12:11:06.113269524 +0000 UTC m=+555.843644493" watchObservedRunningTime="2026-07-08 12:11:06.115805244 +0000 UTC m=+555.846180217"
Jul 08 12:11:17.085572 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:17.085540    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7bbd6859f5-9ccqc"
Jul 08 12:11:17.086045 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:17.085612    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-858769f6f8-vgwvn"
Jul 08 12:11:17.162416 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:17.162385    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7bbd6859f5-9ccqc"]
Jul 08 12:11:17.162730 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:17.162698    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-7bbd6859f5-9ccqc" podUID="12f93142-6df7-4888-9504-72347e2bed1e" containerName="manager" containerID="cri-o://195af9f354459d589d4433062562a78e184c395a07c034989d4b6ca5282ba2e6" gracePeriod=30
Jul 08 12:11:17.413017 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:17.412993    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bbd6859f5-9ccqc"
Jul 08 12:11:17.448510 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:17.448480    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9557t\" (UniqueName: \"kubernetes.io/projected/12f93142-6df7-4888-9504-72347e2bed1e-kube-api-access-9557t\") pod \"12f93142-6df7-4888-9504-72347e2bed1e\" (UID: \"12f93142-6df7-4888-9504-72347e2bed1e\") "
Jul 08 12:11:17.448682 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:17.448519    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/12f93142-6df7-4888-9504-72347e2bed1e-webhook-cert\") pod \"12f93142-6df7-4888-9504-72347e2bed1e\" (UID: \"12f93142-6df7-4888-9504-72347e2bed1e\") "
Jul 08 12:11:17.450931 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:17.450900    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12f93142-6df7-4888-9504-72347e2bed1e-kube-api-access-9557t" (OuterVolumeSpecName: "kube-api-access-9557t") pod "12f93142-6df7-4888-9504-72347e2bed1e" (UID: "12f93142-6df7-4888-9504-72347e2bed1e"). InnerVolumeSpecName "kube-api-access-9557t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 12:11:17.450931 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:17.450910    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12f93142-6df7-4888-9504-72347e2bed1e-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "12f93142-6df7-4888-9504-72347e2bed1e" (UID: "12f93142-6df7-4888-9504-72347e2bed1e"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 12:11:17.549799 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:17.549730    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9557t\" (UniqueName: \"kubernetes.io/projected/12f93142-6df7-4888-9504-72347e2bed1e-kube-api-access-9557t\") on node \"ip-10-0-138-89.ec2.internal\" DevicePath \"\""
Jul 08 12:11:17.549799 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:17.549763    2579 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/12f93142-6df7-4888-9504-72347e2bed1e-webhook-cert\") on node \"ip-10-0-138-89.ec2.internal\" DevicePath \"\""
Jul 08 12:11:18.129135 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:18.129105    2579 generic.go:356] "Generic (PLEG): container finished" podID="12f93142-6df7-4888-9504-72347e2bed1e" containerID="195af9f354459d589d4433062562a78e184c395a07c034989d4b6ca5282ba2e6" exitCode=0
Jul 08 12:11:18.129579 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:18.129153    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bbd6859f5-9ccqc" event={"ID":"12f93142-6df7-4888-9504-72347e2bed1e","Type":"ContainerDied","Data":"195af9f354459d589d4433062562a78e184c395a07c034989d4b6ca5282ba2e6"}
Jul 08 12:11:18.129579 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:18.129189    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bbd6859f5-9ccqc" event={"ID":"12f93142-6df7-4888-9504-72347e2bed1e","Type":"ContainerDied","Data":"6d504baccf9c8960d2f14b4ff6cc3d95ba597d3e4711f08ba5849d0b154f5fe6"}
Jul 08 12:11:18.129579 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:18.129209    2579 scope.go:117] "RemoveContainer" containerID="195af9f354459d589d4433062562a78e184c395a07c034989d4b6ca5282ba2e6"
Jul 08 12:11:18.129579 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:18.129155    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bbd6859f5-9ccqc"
Jul 08 12:11:18.138243 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:18.138226    2579 scope.go:117] "RemoveContainer" containerID="195af9f354459d589d4433062562a78e184c395a07c034989d4b6ca5282ba2e6"
Jul 08 12:11:18.138503 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:11:18.138486    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"195af9f354459d589d4433062562a78e184c395a07c034989d4b6ca5282ba2e6\": container with ID starting with 195af9f354459d589d4433062562a78e184c395a07c034989d4b6ca5282ba2e6 not found: ID does not exist" containerID="195af9f354459d589d4433062562a78e184c395a07c034989d4b6ca5282ba2e6"
Jul 08 12:11:18.138550 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:18.138512    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"195af9f354459d589d4433062562a78e184c395a07c034989d4b6ca5282ba2e6"} err="failed to get container status \"195af9f354459d589d4433062562a78e184c395a07c034989d4b6ca5282ba2e6\": rpc error: code = NotFound desc = could not find container \"195af9f354459d589d4433062562a78e184c395a07c034989d4b6ca5282ba2e6\": container with ID starting with 195af9f354459d589d4433062562a78e184c395a07c034989d4b6ca5282ba2e6 not found: ID does not exist"
Jul 08 12:11:18.152664 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:18.152641    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7bbd6859f5-9ccqc"]
Jul 08 12:11:18.158978 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:18.158956    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7bbd6859f5-9ccqc"]
Jul 08 12:11:18.716142 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:18.716109    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12f93142-6df7-4888-9504-72347e2bed1e" path="/var/lib/kubelet/pods/12f93142-6df7-4888-9504-72347e2bed1e/volumes"
Jul 08 12:11:37.618642 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:37.618607    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5d64f7d5c4-z7hd4"]
Jul 08 12:11:37.619135 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:37.618984    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12f93142-6df7-4888-9504-72347e2bed1e" containerName="manager"
Jul 08 12:11:37.619135 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:37.618995    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="12f93142-6df7-4888-9504-72347e2bed1e" containerName="manager"
Jul 08 12:11:37.619135 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:37.619058    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="12f93142-6df7-4888-9504-72347e2bed1e" containerName="manager"
Jul 08 12:11:37.625344 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:37.625323    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5d64f7d5c4-z7hd4"
Jul 08 12:11:37.628012 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:37.627993    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Jul 08 12:11:37.628116 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:37.628054    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-mp9nf\""
Jul 08 12:11:37.635150 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:37.635128    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5d64f7d5c4-z7hd4"]
Jul 08 12:11:37.725344 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:37.725312    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9eb5f60-4762-4331-9772-43500a84659d-maas-api-tls\") pod \"maas-api-5d64f7d5c4-z7hd4\" (UID: \"d9eb5f60-4762-4331-9772-43500a84659d\") " pod="odh-ai-gateway-infra/maas-api-5d64f7d5c4-z7hd4"
Jul 08 12:11:37.725538 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:37.725392    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgllc\" (UniqueName: \"kubernetes.io/projected/d9eb5f60-4762-4331-9772-43500a84659d-kube-api-access-pgllc\") pod \"maas-api-5d64f7d5c4-z7hd4\" (UID: \"d9eb5f60-4762-4331-9772-43500a84659d\") " pod="odh-ai-gateway-infra/maas-api-5d64f7d5c4-z7hd4"
Jul 08 12:11:37.725538 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:37.725423    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9eb5f60-4762-4331-9772-43500a84659d-openshift-service-ca\") pod \"maas-api-5d64f7d5c4-z7hd4\" (UID: \"d9eb5f60-4762-4331-9772-43500a84659d\") " pod="odh-ai-gateway-infra/maas-api-5d64f7d5c4-z7hd4"
Jul 08 12:11:37.826092 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:37.826051    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgllc\" (UniqueName: \"kubernetes.io/projected/d9eb5f60-4762-4331-9772-43500a84659d-kube-api-access-pgllc\") pod \"maas-api-5d64f7d5c4-z7hd4\" (UID: \"d9eb5f60-4762-4331-9772-43500a84659d\") " pod="odh-ai-gateway-infra/maas-api-5d64f7d5c4-z7hd4"
Jul 08 12:11:37.826289 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:37.826106    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9eb5f60-4762-4331-9772-43500a84659d-openshift-service-ca\") pod \"maas-api-5d64f7d5c4-z7hd4\" (UID: \"d9eb5f60-4762-4331-9772-43500a84659d\") " pod="odh-ai-gateway-infra/maas-api-5d64f7d5c4-z7hd4"
Jul 08 12:11:37.826289 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:37.826154    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9eb5f60-4762-4331-9772-43500a84659d-maas-api-tls\") pod \"maas-api-5d64f7d5c4-z7hd4\" (UID: \"d9eb5f60-4762-4331-9772-43500a84659d\") " pod="odh-ai-gateway-infra/maas-api-5d64f7d5c4-z7hd4"
Jul 08 12:11:37.826411 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:11:37.826313    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found
Jul 08 12:11:37.826411 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:11:37.826389    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9eb5f60-4762-4331-9772-43500a84659d-maas-api-tls podName:d9eb5f60-4762-4331-9772-43500a84659d nodeName:}" failed. No retries permitted until 2026-07-08 12:11:38.326365637 +0000 UTC m=+588.056740607 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d9eb5f60-4762-4331-9772-43500a84659d-maas-api-tls") pod "maas-api-5d64f7d5c4-z7hd4" (UID: "d9eb5f60-4762-4331-9772-43500a84659d") : secret "maas-api-serving-cert" not found
Jul 08 12:11:37.826841 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:37.826810    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9eb5f60-4762-4331-9772-43500a84659d-openshift-service-ca\") pod \"maas-api-5d64f7d5c4-z7hd4\" (UID: \"d9eb5f60-4762-4331-9772-43500a84659d\") " pod="odh-ai-gateway-infra/maas-api-5d64f7d5c4-z7hd4"
Jul 08 12:11:37.835483 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:37.835459    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgllc\" (UniqueName: \"kubernetes.io/projected/d9eb5f60-4762-4331-9772-43500a84659d-kube-api-access-pgllc\") pod \"maas-api-5d64f7d5c4-z7hd4\" (UID: \"d9eb5f60-4762-4331-9772-43500a84659d\") " pod="odh-ai-gateway-infra/maas-api-5d64f7d5c4-z7hd4"
Jul 08 12:11:38.219045 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:38.219014    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-2s49w"]
Jul 08 12:11:38.222823 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:38.222805    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-2s49w"
Jul 08 12:11:38.226232 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:38.226208    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-9vxw9\""
Jul 08 12:11:38.226366 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:38.226232    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 08 12:11:38.236191 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:38.236169    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-2s49w"]
Jul 08 12:11:38.265005 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:38.264972    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-hq45b"]
Jul 08 12:11:38.268283 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:38.268267    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-hq45b"
Jul 08 12:11:38.283032 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:38.283005    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-hq45b"]
Jul 08 12:11:38.331071 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:38.331041    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dbe08ff-b44a-48f4-911a-ca873bdd44a8-plugins-config-volume\") pod \"payload-processing-58f86c7798-2s49w\" (UID: \"9dbe08ff-b44a-48f4-911a-ca873bdd44a8\") " pod="openshift-ingress/payload-processing-58f86c7798-2s49w"
Jul 08 12:11:38.331242 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:38.331097    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6bmz\" (UniqueName: \"kubernetes.io/projected/30ac9628-8965-48bd-b98e-d632fc85a69a-kube-api-access-t6bmz\") pod \"payload-pre-processing-7964457696-hq45b\" (UID: \"30ac9628-8965-48bd-b98e-d632fc85a69a\") " pod="openshift-ingress/payload-pre-processing-7964457696-hq45b"
Jul 08 12:11:38.331242 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:38.331123    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngps9\" (UniqueName: \"kubernetes.io/projected/9dbe08ff-b44a-48f4-911a-ca873bdd44a8-kube-api-access-ngps9\") pod \"payload-processing-58f86c7798-2s49w\" (UID: \"9dbe08ff-b44a-48f4-911a-ca873bdd44a8\") " pod="openshift-ingress/payload-processing-58f86c7798-2s49w"
Jul 08 12:11:38.331242 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:38.331194    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30ac9628-8965-48bd-b98e-d632fc85a69a-plugins-config-volume\") pod \"payload-pre-processing-7964457696-hq45b\" (UID: \"30ac9628-8965-48bd-b98e-d632fc85a69a\") " pod="openshift-ingress/payload-pre-processing-7964457696-hq45b"
Jul 08 12:11:38.331383 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:38.331252    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9eb5f60-4762-4331-9772-43500a84659d-maas-api-tls\") pod \"maas-api-5d64f7d5c4-z7hd4\" (UID: \"d9eb5f60-4762-4331-9772-43500a84659d\") " pod="odh-ai-gateway-infra/maas-api-5d64f7d5c4-z7hd4"
Jul 08 12:11:38.333681 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:38.333660    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9eb5f60-4762-4331-9772-43500a84659d-maas-api-tls\") pod \"maas-api-5d64f7d5c4-z7hd4\" (UID: \"d9eb5f60-4762-4331-9772-43500a84659d\") " pod="odh-ai-gateway-infra/maas-api-5d64f7d5c4-z7hd4"
Jul 08 12:11:38.420734 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:38.420697    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:11:38.432563 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:38.432535    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dbe08ff-b44a-48f4-911a-ca873bdd44a8-plugins-config-volume\") pod \"payload-processing-58f86c7798-2s49w\" (UID: \"9dbe08ff-b44a-48f4-911a-ca873bdd44a8\") " pod="openshift-ingress/payload-processing-58f86c7798-2s49w"
Jul 08 12:11:38.432710 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:38.432584    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6bmz\" (UniqueName: \"kubernetes.io/projected/30ac9628-8965-48bd-b98e-d632fc85a69a-kube-api-access-t6bmz\") pod \"payload-pre-processing-7964457696-hq45b\" (UID: \"30ac9628-8965-48bd-b98e-d632fc85a69a\") " pod="openshift-ingress/payload-pre-processing-7964457696-hq45b"
Jul 08 12:11:38.432710 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:38.432616    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngps9\" (UniqueName: \"kubernetes.io/projected/9dbe08ff-b44a-48f4-911a-ca873bdd44a8-kube-api-access-ngps9\") pod \"payload-processing-58f86c7798-2s49w\" (UID: \"9dbe08ff-b44a-48f4-911a-ca873bdd44a8\") " pod="openshift-ingress/payload-processing-58f86c7798-2s49w"
Jul 08 12:11:38.432710 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:38.432660    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30ac9628-8965-48bd-b98e-d632fc85a69a-plugins-config-volume\") pod \"payload-pre-processing-7964457696-hq45b\" (UID: \"30ac9628-8965-48bd-b98e-d632fc85a69a\") " pod="openshift-ingress/payload-pre-processing-7964457696-hq45b"
Jul 08 12:11:38.433494 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:38.433467    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dbe08ff-b44a-48f4-911a-ca873bdd44a8-plugins-config-volume\") pod \"payload-processing-58f86c7798-2s49w\" (UID: \"9dbe08ff-b44a-48f4-911a-ca873bdd44a8\") " pod="openshift-ingress/payload-processing-58f86c7798-2s49w"
Jul 08 12:11:38.433603 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:38.433467    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30ac9628-8965-48bd-b98e-d632fc85a69a-plugins-config-volume\") pod \"payload-pre-processing-7964457696-hq45b\" (UID: \"30ac9628-8965-48bd-b98e-d632fc85a69a\") " pod="openshift-ingress/payload-pre-processing-7964457696-hq45b"
Jul 08 12:11:38.443402 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:38.443365    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngps9\" (UniqueName: \"kubernetes.io/projected/9dbe08ff-b44a-48f4-911a-ca873bdd44a8-kube-api-access-ngps9\") pod \"payload-processing-58f86c7798-2s49w\" (UID: \"9dbe08ff-b44a-48f4-911a-ca873bdd44a8\") " pod="openshift-ingress/payload-processing-58f86c7798-2s49w"
Jul 08 12:11:38.444370 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:38.444355    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6bmz\" (UniqueName: \"kubernetes.io/projected/30ac9628-8965-48bd-b98e-d632fc85a69a-kube-api-access-t6bmz\") pod \"payload-pre-processing-7964457696-hq45b\" (UID: \"30ac9628-8965-48bd-b98e-d632fc85a69a\") " pod="openshift-ingress/payload-pre-processing-7964457696-hq45b"
Jul 08 12:11:38.533593 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:38.533505    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-2s49w"
Jul 08 12:11:38.536714 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:38.536689    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5d64f7d5c4-z7hd4"
Jul 08 12:11:38.578117 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:38.577784    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-hq45b"
Jul 08 12:11:38.701008 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:38.700980    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-2s49w"]
Jul 08 12:11:38.703594 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:11:38.703541    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dbe08ff_b44a_48f4_911a_ca873bdd44a8.slice/crio-095f4562ee3e876b37e779ae641693624e1964dda4fd1e15ce405a1c230797ca WatchSource:0}: Error finding container 095f4562ee3e876b37e779ae641693624e1964dda4fd1e15ce405a1c230797ca: Status 404 returned error can't find the container with id 095f4562ee3e876b37e779ae641693624e1964dda4fd1e15ce405a1c230797ca
Jul 08 12:11:38.728137 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:38.728116    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5d64f7d5c4-z7hd4"]
Jul 08 12:11:38.730595 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:11:38.730561    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9eb5f60_4762_4331_9772_43500a84659d.slice/crio-5e36632ac96ff4053413839bb065c7795555a378cc0efbb55f0c059f2067a18c WatchSource:0}: Error finding container 5e36632ac96ff4053413839bb065c7795555a378cc0efbb55f0c059f2067a18c: Status 404 returned error can't find the container with id 5e36632ac96ff4053413839bb065c7795555a378cc0efbb55f0c059f2067a18c
Jul 08 12:11:38.749481 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:38.749460    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-hq45b"]
Jul 08 12:11:38.751128 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:11:38.751104    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30ac9628_8965_48bd_b98e_d632fc85a69a.slice/crio-c5e91135fc55b0b906d25ac1d80026624063e49fd8a66779b2372efc045e0dd6 WatchSource:0}: Error finding container c5e91135fc55b0b906d25ac1d80026624063e49fd8a66779b2372efc045e0dd6: Status 404 returned error can't find the container with id c5e91135fc55b0b906d25ac1d80026624063e49fd8a66779b2372efc045e0dd6
Jul 08 12:11:39.226805 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:39.226744    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-2s49w" event={"ID":"9dbe08ff-b44a-48f4-911a-ca873bdd44a8","Type":"ContainerStarted","Data":"095f4562ee3e876b37e779ae641693624e1964dda4fd1e15ce405a1c230797ca"}
Jul 08 12:11:39.228017 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:39.227991    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-hq45b" event={"ID":"30ac9628-8965-48bd-b98e-d632fc85a69a","Type":"ContainerStarted","Data":"c5e91135fc55b0b906d25ac1d80026624063e49fd8a66779b2372efc045e0dd6"}
Jul 08 12:11:39.229157 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:39.229133    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5d64f7d5c4-z7hd4" event={"ID":"d9eb5f60-4762-4331-9772-43500a84659d","Type":"ContainerStarted","Data":"5e36632ac96ff4053413839bb065c7795555a378cc0efbb55f0c059f2067a18c"}
Jul 08 12:11:41.241179 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:41.241142    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-hq45b" event={"ID":"30ac9628-8965-48bd-b98e-d632fc85a69a","Type":"ContainerStarted","Data":"c93c4b25ac20c595a4b78d2298008dca836302ff079f212980c4d8d8e147a7f9"}
Jul 08 12:11:41.241657 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:41.241285    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7964457696-hq45b"
Jul 08 12:11:41.242626 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:41.242603    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5d64f7d5c4-z7hd4" event={"ID":"d9eb5f60-4762-4331-9772-43500a84659d","Type":"ContainerStarted","Data":"8f727eade9d3c26d48ac66c7e3fded50c6ce33200e14bc4c26e922951d42711d"}
Jul 08 12:11:41.242740 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:41.242669    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5d64f7d5c4-z7hd4"
Jul 08 12:11:41.243954 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:41.243936    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-2s49w" event={"ID":"9dbe08ff-b44a-48f4-911a-ca873bdd44a8","Type":"ContainerStarted","Data":"c38eac74f4fdfcdebe805736f16be583e5591d70e0c40190e23889f0907685db"}
Jul 08 12:11:41.244034 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:41.243983    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-58f86c7798-2s49w"
Jul 08 12:11:41.265323 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:41.265279    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7964457696-hq45b" podStartSLOduration=1.428442704 podStartE2EDuration="3.26526385s" totalImagesPullingTime="1.836821146s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 12:11:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 12:11:41.263969126 +0000 UTC m=+590.994344099" watchObservedRunningTime="2026-07-08 12:11:41.26526385 +0000 UTC m=+590.995638888"
Jul 08 12:11:41.288173 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:41.288128    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5d64f7d5c4-z7hd4" podStartSLOduration=2.432623678 podStartE2EDuration="4.288113978s" totalImagesPullingTime="1.8554903s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 12:11:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 12:11:41.286634413 +0000 UTC m=+591.017009384" watchObservedRunningTime="2026-07-08 12:11:41.288113978 +0000 UTC m=+591.018488949"
Jul 08 12:11:41.308172 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:41.308115    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-58f86c7798-2s49w" podStartSLOduration=1.426376864 podStartE2EDuration="3.308101012s" totalImagesPullingTime="1.881724148s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 12:11:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 12:11:41.307750046 +0000 UTC m=+591.038125016" watchObservedRunningTime="2026-07-08 12:11:41.308101012 +0000 UTC m=+591.038475983"
Jul 08 12:11:47.255997 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:47.255963    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5d64f7d5c4-z7hd4"
Jul 08 12:11:52.252560 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:52.252527    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-58f86c7798-2s49w"
Jul 08 12:11:52.253071 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:52.252611    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7964457696-hq45b"
Jul 08 12:11:56.935657 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:11:56.935625    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:12:08.330269 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:12:08.330236    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-s4kld"]
Jul 08 12:12:08.333956 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:12:08.333937    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s4kld"
Jul 08 12:12:08.336808 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:12:08.336785    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-f4pdd\""
Jul 08 12:12:08.336917 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:12:08.336789    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 08 12:12:08.337722 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:12:08.337705    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 08 12:12:08.337799 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:12:08.337728    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Jul 08 12:12:08.344999 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:12:08.344978    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-s4kld"]
Jul 08 12:12:08.394889 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:12:08.394862    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe12af73-3e74-4e9a-88c9-03935365fec2-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s4kld\" (UID: \"fe12af73-3e74-4e9a-88c9-03935365fec2\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s4kld"
Jul 08 12:12:08.395061 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:12:08.394893    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zkdj\" (UniqueName: \"kubernetes.io/projected/fe12af73-3e74-4e9a-88c9-03935365fec2-kube-api-access-2zkdj\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s4kld\" (UID: \"fe12af73-3e74-4e9a-88c9-03935365fec2\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s4kld"
Jul 08 12:12:08.395061 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:12:08.394931    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe12af73-3e74-4e9a-88c9-03935365fec2-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s4kld\" (UID: \"fe12af73-3e74-4e9a-88c9-03935365fec2\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s4kld"
Jul 08 12:12:08.395061 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:12:08.394951    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe12af73-3e74-4e9a-88c9-03935365fec2-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s4kld\" (UID: \"fe12af73-3e74-4e9a-88c9-03935365fec2\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s4kld"
Jul 08 12:12:08.395061 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:12:08.394978    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe12af73-3e74-4e9a-88c9-03935365fec2-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s4kld\" (UID: \"fe12af73-3e74-4e9a-88c9-03935365fec2\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s4kld"
Jul 08 12:12:08.395061 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:12:08.395004    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe12af73-3e74-4e9a-88c9-03935365fec2-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s4kld\" (UID: \"fe12af73-3e74-4e9a-88c9-03935365fec2\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s4kld"
Jul 08 12:12:08.495859 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:12:08.495823    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe12af73-3e74-4e9a-88c9-03935365fec2-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s4kld\" (UID: \"fe12af73-3e74-4e9a-88c9-03935365fec2\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s4kld"
Jul 08 12:12:08.496028 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:12:08.495881    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe12af73-3e74-4e9a-88c9-03935365fec2-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s4kld\" (UID: \"fe12af73-3e74-4e9a-88c9-03935365fec2\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s4kld"
Jul 08 12:12:08.496028 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:12:08.495930    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe12af73-3e74-4e9a-88c9-03935365fec2-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s4kld\" (UID: \"fe12af73-3e74-4e9a-88c9-03935365fec2\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s4kld"
Jul 08 12:12:08.496028 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:12:08.495957    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zkdj\" (UniqueName: \"kubernetes.io/projected/fe12af73-3e74-4e9a-88c9-03935365fec2-kube-api-access-2zkdj\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s4kld\" (UID: \"fe12af73-3e74-4e9a-88c9-03935365fec2\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s4kld"
Jul 08 12:12:08.496137 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:12:08.496102    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe12af73-3e74-4e9a-88c9-03935365fec2-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s4kld\" (UID: \"fe12af73-3e74-4e9a-88c9-03935365fec2\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s4kld"
Jul 08 12:12:08.496186 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:12:08.496138    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe12af73-3e74-4e9a-88c9-03935365fec2-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s4kld\" (UID: \"fe12af73-3e74-4e9a-88c9-03935365fec2\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s4kld"
Jul 08 12:12:08.496350 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:12:08.496331    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe12af73-3e74-4e9a-88c9-03935365fec2-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s4kld\" (UID: \"fe12af73-3e74-4e9a-88c9-03935365fec2\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s4kld"
Jul 08 12:12:08.496441 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:12:08.496419    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe12af73-3e74-4e9a-88c9-03935365fec2-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s4kld\" (UID: \"fe12af73-3e74-4e9a-88c9-03935365fec2\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s4kld"
Jul 08 12:12:08.496489 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:12:08.496471    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe12af73-3e74-4e9a-88c9-03935365fec2-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s4kld\" (UID: \"fe12af73-3e74-4e9a-88c9-03935365fec2\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s4kld"
Jul 08 12:12:08.498310 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:12:08.498292    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe12af73-3e74-4e9a-88c9-03935365fec2-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s4kld\" (UID: \"fe12af73-3e74-4e9a-88c9-03935365fec2\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s4kld"
Jul 08 12:12:08.498572 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:12:08.498556    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe12af73-3e74-4e9a-88c9-03935365fec2-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s4kld\" (UID: \"fe12af73-3e74-4e9a-88c9-03935365fec2\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s4kld"
Jul 08 12:12:08.505821 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:12:08.505796    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zkdj\" (UniqueName: \"kubernetes.io/projected/fe12af73-3e74-4e9a-88c9-03935365fec2-kube-api-access-2zkdj\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s4kld\" (UID: \"fe12af73-3e74-4e9a-88c9-03935365fec2\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s4kld"
Jul 08 12:12:08.644647 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:12:08.644561    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s4kld"
Jul 08 12:12:08.787965 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:12:08.787934    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-s4kld"]
Jul 08 12:12:08.789827 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:12:08.789799    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe12af73_3e74_4e9a_88c9_03935365fec2.slice/crio-c6eeeb6f66668f34408d42ceae7078f59e11c4585a7136606ec07e4ea9823491 WatchSource:0}: Error finding container c6eeeb6f66668f34408d42ceae7078f59e11c4585a7136606ec07e4ea9823491: Status 404 returned error can't find the container with id c6eeeb6f66668f34408d42ceae7078f59e11c4585a7136606ec07e4ea9823491
Jul 08 12:12:08.791078 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:12:08.791058    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 08 12:12:09.384456 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:12:09.384413    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s4kld" event={"ID":"fe12af73-3e74-4e9a-88c9-03935365fec2","Type":"ContainerStarted","Data":"c6eeeb6f66668f34408d42ceae7078f59e11c4585a7136606ec07e4ea9823491"}
Jul 08 12:12:09.518248 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:12:09.518212    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:12:12.400662 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:12:12.400620    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s4kld" event={"ID":"fe12af73-3e74-4e9a-88c9-03935365fec2","Type":"ContainerStarted","Data":"f23e05d804c1448668a3acc6a9f8cfb944d1a95b3d17e7f3f5efe0ca9b3e6cd2"}
Jul 08 12:12:12.425504 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:12:12.425456    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s4kld" podStartSLOduration=1.6613798339999999 podStartE2EDuration="4.425437744s" totalImagesPullingTime="2.76405791s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 12:12:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 12:12:12.424826758 +0000 UTC m=+622.155201732" watchObservedRunningTime="2026-07-08 12:12:12.425437744 +0000 UTC m=+622.155812714"
Jul 08 12:12:18.645343 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:12:18.645305    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s4kld"
Jul 08 12:12:18.645829 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:12:18.645410    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s4kld"
Jul 08 12:12:18.657485 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:12:18.657455    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s4kld"
Jul 08 12:12:19.446502 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:12:19.446476    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s4kld"
Jul 08 12:12:25.021338 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:12:25.021305    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:12:37.617512 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:12:37.617479    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:12:46.260805 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:12:46.260759    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-cf6tp"]
Jul 08 12:12:46.266618 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:12:46.266601    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cf6tp"
Jul 08 12:12:46.275433 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:12:46.275407    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Jul 08 12:12:46.288127 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:12:46.288094    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-cf6tp"]
Jul 08 12:12:46.341924 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:12:46.341888    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ada9676-dcfc-4722-82d6-90f7bd74d7ec-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cf6tp\" (UID: \"7ada9676-dcfc-4722-82d6-90f7bd74d7ec\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cf6tp"
Jul 08 12:12:46.341924 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:12:46.341926    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ada9676-dcfc-4722-82d6-90f7bd74d7ec-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cf6tp\" (UID: \"7ada9676-dcfc-4722-82d6-90f7bd74d7ec\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cf6tp"
Jul 08 12:12:46.342161 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:12:46.342028    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ada9676-dcfc-4722-82d6-90f7bd74d7ec-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cf6tp\" (UID: \"7ada9676-dcfc-4722-82d6-90f7bd74d7ec\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cf6tp"
Jul 08 12:12:46.342161 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:12:46.342116    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ada9676-dcfc-4722-82d6-90f7bd74d7ec-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cf6tp\" (UID: \"7ada9676-dcfc-4722-82d6-90f7bd74d7ec\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cf6tp"
Jul 08 12:12:46.342161 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:12:46.342153    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ada9676-dcfc-4722-82d6-90f7bd74d7ec-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cf6tp\" (UID: \"7ada9676-dcfc-4722-82d6-90f7bd74d7ec\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cf6tp"
Jul 08 12:12:46.342261 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:12:46.342245    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbkt2\" (UniqueName: \"kubernetes.io/projected/7ada9676-dcfc-4722-82d6-90f7bd74d7ec-kube-api-access-hbkt2\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cf6tp\" (UID: \"7ada9676-dcfc-4722-82d6-90f7bd74d7ec\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cf6tp"
Jul 08 12:12:46.443156 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:12:46.443116    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ada9676-dcfc-4722-82d6-90f7bd74d7ec-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cf6tp\" (UID: \"7ada9676-dcfc-4722-82d6-90f7bd74d7ec\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cf6tp"
Jul 08 12:12:46.443156 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:12:46.443161    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ada9676-dcfc-4722-82d6-90f7bd74d7ec-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cf6tp\" (UID: \"7ada9676-dcfc-4722-82d6-90f7bd74d7ec\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cf6tp"
Jul 08 12:12:46.443423 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:12:46.443181    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ada9676-dcfc-4722-82d6-90f7bd74d7ec-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cf6tp\" (UID: \"7ada9676-dcfc-4722-82d6-90f7bd74d7ec\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cf6tp"
Jul 08 12:12:46.443423 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:12:46.443225    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbkt2\" (UniqueName: \"kubernetes.io/projected/7ada9676-dcfc-4722-82d6-90f7bd74d7ec-kube-api-access-hbkt2\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cf6tp\" (UID: \"7ada9676-dcfc-4722-82d6-90f7bd74d7ec\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cf6tp"
Jul 08 12:12:46.443423 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:12:46.443252    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ada9676-dcfc-4722-82d6-90f7bd74d7ec-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cf6tp\" (UID: \"7ada9676-dcfc-4722-82d6-90f7bd74d7ec\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cf6tp"
Jul 08 12:12:46.443423 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:12:46.443269    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ada9676-dcfc-4722-82d6-90f7bd74d7ec-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cf6tp\" (UID: \"7ada9676-dcfc-4722-82d6-90f7bd74d7ec\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cf6tp"
Jul 08 12:12:46.443649 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:12:46.443614    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ada9676-dcfc-4722-82d6-90f7bd74d7ec-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cf6tp\" (UID: \"7ada9676-dcfc-4722-82d6-90f7bd74d7ec\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cf6tp"
Jul 08 12:12:46.443700 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:12:46.443626    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ada9676-dcfc-4722-82d6-90f7bd74d7ec-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cf6tp\" (UID: \"7ada9676-dcfc-4722-82d6-90f7bd74d7ec\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cf6tp"
Jul 08 12:12:46.443700 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:12:46.443693    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ada9676-dcfc-4722-82d6-90f7bd74d7ec-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cf6tp\" (UID: \"7ada9676-dcfc-4722-82d6-90f7bd74d7ec\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cf6tp"
Jul 08 12:12:46.445592 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:12:46.445571    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ada9676-dcfc-4722-82d6-90f7bd74d7ec-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cf6tp\" (UID: \"7ada9676-dcfc-4722-82d6-90f7bd74d7ec\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cf6tp"
Jul 08 12:12:46.445970 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:12:46.445953    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ada9676-dcfc-4722-82d6-90f7bd74d7ec-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cf6tp\" (UID: \"7ada9676-dcfc-4722-82d6-90f7bd74d7ec\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cf6tp"
Jul 08 12:12:46.477437 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:12:46.477405    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbkt2\" (UniqueName: \"kubernetes.io/projected/7ada9676-dcfc-4722-82d6-90f7bd74d7ec-kube-api-access-hbkt2\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cf6tp\" (UID: \"7ada9676-dcfc-4722-82d6-90f7bd74d7ec\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cf6tp"
Jul 08 12:12:46.577638 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:12:46.577537    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cf6tp"
Jul 08 12:12:46.728044 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:12:46.728015    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-cf6tp"]
Jul 08 12:12:46.731454 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:12:46.731425    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ada9676_dcfc_4722_82d6_90f7bd74d7ec.slice/crio-0b24501d5fa4c2c7598d954e50e6e45898a79c75a3bfa6ee606d555b181b163e WatchSource:0}: Error finding container 0b24501d5fa4c2c7598d954e50e6e45898a79c75a3bfa6ee606d555b181b163e: Status 404 returned error can't find the container with id 0b24501d5fa4c2c7598d954e50e6e45898a79c75a3bfa6ee606d555b181b163e
Jul 08 12:12:47.572067 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:12:47.572028    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cf6tp" event={"ID":"7ada9676-dcfc-4722-82d6-90f7bd74d7ec","Type":"ContainerStarted","Data":"15d750829aa167ffa374e1216fdf66b549e5443900c3d6777532e094f473daf0"}
Jul 08 12:12:47.572475 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:12:47.572087    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cf6tp" event={"ID":"7ada9676-dcfc-4722-82d6-90f7bd74d7ec","Type":"ContainerStarted","Data":"0b24501d5fa4c2c7598d954e50e6e45898a79c75a3bfa6ee606d555b181b163e"}
Jul 08 12:12:47.597881 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:12:47.597829    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cf6tp" podStartSLOduration=1.2975006599999999 podStartE2EDuration="1.597814524s" totalImagesPullingTime="300.313864ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 12:12:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 12:12:47.59699055 +0000 UTC m=+657.327365521" watchObservedRunningTime="2026-07-08 12:12:47.597814524 +0000 UTC m=+657.328189494"
Jul 08 12:12:48.358167 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:12:48.358132    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:12:56.578573 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:12:56.578522    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cf6tp"
Jul 08 12:12:56.579095 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:12:56.578609    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cf6tp"
Jul 08 12:12:56.591342 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:12:56.591313    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cf6tp"
Jul 08 12:12:56.629620 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:12:56.629585    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cf6tp"
Jul 08 12:13:12.448385 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:13:12.448344    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:15:20.717968 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:15:20.717883    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:15:30.513587 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:15:30.513549    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:15:40.015912 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:15:40.015875    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:15:50.025851 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:15:50.025819    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:15:58.168849 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:15:58.168816    2579 scope.go:117] "RemoveContainer" containerID="bb09688dd903927e2e62a075c93760d92a79a08d63fa146dc34df921b972ccc3"
Jul 08 12:15:58.178077 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:15:58.178055    2579 scope.go:117] "RemoveContainer" containerID="79ffb6104d00536d8e4604b21ab732a5523ac536dfcfe2aad71e3a6b2d636960"
Jul 08 12:16:00.720160 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:16:00.720125    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:16:02.536837 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:16:02.536800    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-858769f6f8-vgwvn"]
Jul 08 12:16:02.537287 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:16:02.537006    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-858769f6f8-vgwvn" podUID="72b4701f-eb6c-47e6-abfd-cfbe250778ab" containerName="manager" containerID="cri-o://f7671ee1098beff1a21d9efd6e412599197bb72766a3040514249b9d6a0ce64f" gracePeriod=30
Jul 08 12:16:02.803458 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:16:02.803431    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-858769f6f8-vgwvn"
Jul 08 12:16:02.882809 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:16:02.882750    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dw9dl\" (UniqueName: \"kubernetes.io/projected/72b4701f-eb6c-47e6-abfd-cfbe250778ab-kube-api-access-dw9dl\") pod \"72b4701f-eb6c-47e6-abfd-cfbe250778ab\" (UID: \"72b4701f-eb6c-47e6-abfd-cfbe250778ab\") "
Jul 08 12:16:02.883008 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:16:02.882876    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/72b4701f-eb6c-47e6-abfd-cfbe250778ab-webhook-cert\") pod \"72b4701f-eb6c-47e6-abfd-cfbe250778ab\" (UID: \"72b4701f-eb6c-47e6-abfd-cfbe250778ab\") "
Jul 08 12:16:02.885182 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:16:02.885138    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72b4701f-eb6c-47e6-abfd-cfbe250778ab-kube-api-access-dw9dl" (OuterVolumeSpecName: "kube-api-access-dw9dl") pod "72b4701f-eb6c-47e6-abfd-cfbe250778ab" (UID: "72b4701f-eb6c-47e6-abfd-cfbe250778ab"). InnerVolumeSpecName "kube-api-access-dw9dl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 12:16:02.885310 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:16:02.885252    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72b4701f-eb6c-47e6-abfd-cfbe250778ab-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "72b4701f-eb6c-47e6-abfd-cfbe250778ab" (UID: "72b4701f-eb6c-47e6-abfd-cfbe250778ab"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 12:16:02.983563 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:16:02.983524    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dw9dl\" (UniqueName: \"kubernetes.io/projected/72b4701f-eb6c-47e6-abfd-cfbe250778ab-kube-api-access-dw9dl\") on node \"ip-10-0-138-89.ec2.internal\" DevicePath \"\""
Jul 08 12:16:02.983563 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:16:02.983557    2579 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/72b4701f-eb6c-47e6-abfd-cfbe250778ab-webhook-cert\") on node \"ip-10-0-138-89.ec2.internal\" DevicePath \"\""
Jul 08 12:16:03.554098 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:16:03.554058    2579 generic.go:356] "Generic (PLEG): container finished" podID="72b4701f-eb6c-47e6-abfd-cfbe250778ab" containerID="f7671ee1098beff1a21d9efd6e412599197bb72766a3040514249b9d6a0ce64f" exitCode=0
Jul 08 12:16:03.554623 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:16:03.554132    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-858769f6f8-vgwvn"
Jul 08 12:16:03.554623 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:16:03.554146    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-858769f6f8-vgwvn" event={"ID":"72b4701f-eb6c-47e6-abfd-cfbe250778ab","Type":"ContainerDied","Data":"f7671ee1098beff1a21d9efd6e412599197bb72766a3040514249b9d6a0ce64f"}
Jul 08 12:16:03.554623 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:16:03.554230    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-858769f6f8-vgwvn" event={"ID":"72b4701f-eb6c-47e6-abfd-cfbe250778ab","Type":"ContainerDied","Data":"c1d3643dc8082493096908f6ef5059a017080d24fc483c7fdd757ac77218081a"}
Jul 08 12:16:03.554623 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:16:03.554254    2579 scope.go:117] "RemoveContainer" containerID="f7671ee1098beff1a21d9efd6e412599197bb72766a3040514249b9d6a0ce64f"
Jul 08 12:16:03.564330 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:16:03.564300    2579 scope.go:117] "RemoveContainer" containerID="f7671ee1098beff1a21d9efd6e412599197bb72766a3040514249b9d6a0ce64f"
Jul 08 12:16:03.564640 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:16:03.564619    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7671ee1098beff1a21d9efd6e412599197bb72766a3040514249b9d6a0ce64f\": container with ID starting with f7671ee1098beff1a21d9efd6e412599197bb72766a3040514249b9d6a0ce64f not found: ID does not exist" containerID="f7671ee1098beff1a21d9efd6e412599197bb72766a3040514249b9d6a0ce64f"
Jul 08 12:16:03.564701 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:16:03.564650    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7671ee1098beff1a21d9efd6e412599197bb72766a3040514249b9d6a0ce64f"} err="failed to get container status \"f7671ee1098beff1a21d9efd6e412599197bb72766a3040514249b9d6a0ce64f\": rpc error: code = NotFound desc = could not find container \"f7671ee1098beff1a21d9efd6e412599197bb72766a3040514249b9d6a0ce64f\": container with ID starting with f7671ee1098beff1a21d9efd6e412599197bb72766a3040514249b9d6a0ce64f not found: ID does not exist"
Jul 08 12:16:03.579577 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:16:03.579545    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-858769f6f8-vgwvn"]
Jul 08 12:16:03.583492 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:16:03.583461    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-858769f6f8-vgwvn"]
Jul 08 12:16:04.362156 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:16:04.362120    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-858769f6f8-p5nz2"]
Jul 08 12:16:04.362580 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:16:04.362564    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72b4701f-eb6c-47e6-abfd-cfbe250778ab" containerName="manager"
Jul 08 12:16:04.362629 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:16:04.362583    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="72b4701f-eb6c-47e6-abfd-cfbe250778ab" containerName="manager"
Jul 08 12:16:04.362662 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:16:04.362657    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="72b4701f-eb6c-47e6-abfd-cfbe250778ab" containerName="manager"
Jul 08 12:16:04.366023 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:16:04.366004    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-858769f6f8-p5nz2"
Jul 08 12:16:04.368933 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:16:04.368907    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-25m4d\""
Jul 08 12:16:04.369979 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:16:04.369960    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 08 12:16:04.370066 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:16:04.370051    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 08 12:16:04.376563 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:16:04.376537    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-858769f6f8-p5nz2"]
Jul 08 12:16:04.497389 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:16:04.497346    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/391eb116-aa03-4f92-bd56-514f552816f5-webhook-cert\") pod \"maas-controller-858769f6f8-p5nz2\" (UID: \"391eb116-aa03-4f92-bd56-514f552816f5\") " pod="opendatahub/maas-controller-858769f6f8-p5nz2"
Jul 08 12:16:04.497575 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:16:04.497441    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4869s\" (UniqueName: \"kubernetes.io/projected/391eb116-aa03-4f92-bd56-514f552816f5-kube-api-access-4869s\") pod \"maas-controller-858769f6f8-p5nz2\" (UID: \"391eb116-aa03-4f92-bd56-514f552816f5\") " pod="opendatahub/maas-controller-858769f6f8-p5nz2"
Jul 08 12:16:04.598436 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:16:04.598395    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/391eb116-aa03-4f92-bd56-514f552816f5-webhook-cert\") pod \"maas-controller-858769f6f8-p5nz2\" (UID: \"391eb116-aa03-4f92-bd56-514f552816f5\") " pod="opendatahub/maas-controller-858769f6f8-p5nz2"
Jul 08 12:16:04.598908 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:16:04.598467    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4869s\" (UniqueName: \"kubernetes.io/projected/391eb116-aa03-4f92-bd56-514f552816f5-kube-api-access-4869s\") pod \"maas-controller-858769f6f8-p5nz2\" (UID: \"391eb116-aa03-4f92-bd56-514f552816f5\") " pod="opendatahub/maas-controller-858769f6f8-p5nz2"
Jul 08 12:16:04.601279 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:16:04.601254    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/391eb116-aa03-4f92-bd56-514f552816f5-webhook-cert\") pod \"maas-controller-858769f6f8-p5nz2\" (UID: \"391eb116-aa03-4f92-bd56-514f552816f5\") " pod="opendatahub/maas-controller-858769f6f8-p5nz2"
Jul 08 12:16:04.609969 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:16:04.609940    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4869s\" (UniqueName: \"kubernetes.io/projected/391eb116-aa03-4f92-bd56-514f552816f5-kube-api-access-4869s\") pod \"maas-controller-858769f6f8-p5nz2\" (UID: \"391eb116-aa03-4f92-bd56-514f552816f5\") " pod="opendatahub/maas-controller-858769f6f8-p5nz2"
Jul 08 12:16:04.677026 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:16:04.676996    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-858769f6f8-p5nz2"
Jul 08 12:16:04.717034 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:16:04.717004    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72b4701f-eb6c-47e6-abfd-cfbe250778ab" path="/var/lib/kubelet/pods/72b4701f-eb6c-47e6-abfd-cfbe250778ab/volumes"
Jul 08 12:16:04.829020 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:16:04.828974    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-858769f6f8-p5nz2"]
Jul 08 12:16:04.832015 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:16:04.831967    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod391eb116_aa03_4f92_bd56_514f552816f5.slice/crio-ee5beabdfd7146d1f2176e302489c4e2f38036abd3642689b322f3f182562353 WatchSource:0}: Error finding container ee5beabdfd7146d1f2176e302489c4e2f38036abd3642689b322f3f182562353: Status 404 returned error can't find the container with id ee5beabdfd7146d1f2176e302489c4e2f38036abd3642689b322f3f182562353
Jul 08 12:16:05.567871 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:16:05.567820    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-858769f6f8-p5nz2" event={"ID":"391eb116-aa03-4f92-bd56-514f552816f5","Type":"ContainerStarted","Data":"99148120c64c68b437fd09b8b949f0b1f074f335f458342432701eedbb2ba67b"}
Jul 08 12:16:05.568065 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:16:05.568031    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-858769f6f8-p5nz2"
Jul 08 12:16:05.568124 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:16:05.568063    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-858769f6f8-p5nz2" event={"ID":"391eb116-aa03-4f92-bd56-514f552816f5","Type":"ContainerStarted","Data":"ee5beabdfd7146d1f2176e302489c4e2f38036abd3642689b322f3f182562353"}
Jul 08 12:16:05.590674 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:16:05.590573    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-858769f6f8-p5nz2" podStartSLOduration=1.060089689 podStartE2EDuration="1.590555766s" totalImagesPullingTime="530.466077ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 12:16:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 12:16:05.590304098 +0000 UTC m=+855.320679069" watchObservedRunningTime="2026-07-08 12:16:05.590555766 +0000 UTC m=+855.320930715"
Jul 08 12:16:16.581183 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:16:16.581147    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-858769f6f8-p5nz2"
Jul 08 12:16:27.042861 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:16:27.042821    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:16:40.319451 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:16:40.319411    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:17:14.951980 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:17:14.951946    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:17:24.817152 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:17:24.817105    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:17:35.519051 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:17:35.519016    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:17:43.305651 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:17:43.305615    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:17:55.316652 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:17:55.316609    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:18:05.237888 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:18:05.237854    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:18:16.932560 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:18:16.932477    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:18:27.410377 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:18:27.410329    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:18:49.508296 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:18:49.508260    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:18:59.405733 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:18:59.405682    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:19:22.103957 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:19:22.103918    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:19:33.309860 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:19:33.309800    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:19:49.126403 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:19:49.126317    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:19:59.515133 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:19:59.515095    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:20:09.615933 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:20:09.615895    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:20:19.817395 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:20:19.817354    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:20:29.209006 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:20:29.208973    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:22:17.076724 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:22:17.076689    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:22:27.638863 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:22:27.638828    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:22:38.225464 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:22:38.225425    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:22:51.563363 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:22:51.563271    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:23:03.467294 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:23:03.467255    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:23:13.637454 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:23:13.637419    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:23:22.634346 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:23:22.634310    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:23:34.421252 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:23:34.421216    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:23:44.508702 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:23:44.508660    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:23:52.807188 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:23:52.807148    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:24:04.521320 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:24:04.521276    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:24:28.314441 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:24:28.314396    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:24:36.326752 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:24:36.326711    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:24:46.520362 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:24:46.520330    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:24:56.505610 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:24:56.505574    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:25:02.305888 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:25:02.305843    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:25:13.900814 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:25:13.900763    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:25:23.705138 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:25:23.705096    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:25:35.205107 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:25:35.205069    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:25:49.204011 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:25:49.203922    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:25:59.510304 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:25:59.510259    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:26:09.107481 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:26:09.107435    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:26:19.011900 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:26:19.011852    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:26:31.599919 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:26:31.599871    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:26:41.202126 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:26:41.202087    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:26:51.302841 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:26:51.302802    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:27:04.707981 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:27:04.707941    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:27:14.600037 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:27:14.599959    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:27:25.495635 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:27:25.495595    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:27:37.110660 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:27:37.110621    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:27:49.305485 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:27:49.305447    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:28:03.309867 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:03.309832    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:28:12.601534 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:12.601494    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:28:34.001294 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:34.001260    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:28:40.836086 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:40.836057    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3a1658d8-64cb555d85-hf6nh"]
Jul 08 12:28:40.839646 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:40.839621    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a1658d8-64cb555d85-hf6nh"
Jul 08 12:28:40.843416 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:40.843389    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-3a1658d8-serving-cert\""
Jul 08 12:28:40.852175 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:40.852149    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3a1658d8-64cb555d85-hf6nh"]
Jul 08 12:28:40.894427 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:40.894392    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd2e166f-686c-4912-ba6d-2408c75864c2-maas-api-tls\") pod \"maas-api-e2e-ait-3a1658d8-64cb555d85-hf6nh\" (UID: \"bd2e166f-686c-4912-ba6d-2408c75864c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a1658d8-64cb555d85-hf6nh"
Jul 08 12:28:40.894427 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:40.894429    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd2e166f-686c-4912-ba6d-2408c75864c2-openshift-service-ca\") pod \"maas-api-e2e-ait-3a1658d8-64cb555d85-hf6nh\" (UID: \"bd2e166f-686c-4912-ba6d-2408c75864c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a1658d8-64cb555d85-hf6nh"
Jul 08 12:28:40.894656 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:40.894446    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2vq8\" (UniqueName: \"kubernetes.io/projected/bd2e166f-686c-4912-ba6d-2408c75864c2-kube-api-access-x2vq8\") pod \"maas-api-e2e-ait-3a1658d8-64cb555d85-hf6nh\" (UID: \"bd2e166f-686c-4912-ba6d-2408c75864c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a1658d8-64cb555d85-hf6nh"
Jul 08 12:28:40.995712 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:40.995675    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd2e166f-686c-4912-ba6d-2408c75864c2-maas-api-tls\") pod \"maas-api-e2e-ait-3a1658d8-64cb555d85-hf6nh\" (UID: \"bd2e166f-686c-4912-ba6d-2408c75864c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a1658d8-64cb555d85-hf6nh"
Jul 08 12:28:40.995712 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:40.995713    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd2e166f-686c-4912-ba6d-2408c75864c2-openshift-service-ca\") pod \"maas-api-e2e-ait-3a1658d8-64cb555d85-hf6nh\" (UID: \"bd2e166f-686c-4912-ba6d-2408c75864c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a1658d8-64cb555d85-hf6nh"
Jul 08 12:28:40.996014 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:40.995730    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2vq8\" (UniqueName: \"kubernetes.io/projected/bd2e166f-686c-4912-ba6d-2408c75864c2-kube-api-access-x2vq8\") pod \"maas-api-e2e-ait-3a1658d8-64cb555d85-hf6nh\" (UID: \"bd2e166f-686c-4912-ba6d-2408c75864c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a1658d8-64cb555d85-hf6nh"
Jul 08 12:28:40.996014 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:28:40.995875    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-3a1658d8-serving-cert: secret "maas-api-e2e-ait-3a1658d8-serving-cert" not found
Jul 08 12:28:40.996014 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:28:40.995945    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd2e166f-686c-4912-ba6d-2408c75864c2-maas-api-tls podName:bd2e166f-686c-4912-ba6d-2408c75864c2 nodeName:}" failed. No retries permitted until 2026-07-08 12:28:41.495928907 +0000 UTC m=+1611.226303856 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bd2e166f-686c-4912-ba6d-2408c75864c2-maas-api-tls") pod "maas-api-e2e-ait-3a1658d8-64cb555d85-hf6nh" (UID: "bd2e166f-686c-4912-ba6d-2408c75864c2") : secret "maas-api-e2e-ait-3a1658d8-serving-cert" not found
Jul 08 12:28:40.996355 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:40.996336    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd2e166f-686c-4912-ba6d-2408c75864c2-openshift-service-ca\") pod \"maas-api-e2e-ait-3a1658d8-64cb555d85-hf6nh\" (UID: \"bd2e166f-686c-4912-ba6d-2408c75864c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a1658d8-64cb555d85-hf6nh"
Jul 08 12:28:41.009713 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:41.009680    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2vq8\" (UniqueName: \"kubernetes.io/projected/bd2e166f-686c-4912-ba6d-2408c75864c2-kube-api-access-x2vq8\") pod \"maas-api-e2e-ait-3a1658d8-64cb555d85-hf6nh\" (UID: \"bd2e166f-686c-4912-ba6d-2408c75864c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a1658d8-64cb555d85-hf6nh"
Jul 08 12:28:41.501040 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:41.500952    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd2e166f-686c-4912-ba6d-2408c75864c2-maas-api-tls\") pod \"maas-api-e2e-ait-3a1658d8-64cb555d85-hf6nh\" (UID: \"bd2e166f-686c-4912-ba6d-2408c75864c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a1658d8-64cb555d85-hf6nh"
Jul 08 12:28:41.503542 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:41.503510    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd2e166f-686c-4912-ba6d-2408c75864c2-maas-api-tls\") pod \"maas-api-e2e-ait-3a1658d8-64cb555d85-hf6nh\" (UID: \"bd2e166f-686c-4912-ba6d-2408c75864c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a1658d8-64cb555d85-hf6nh"
Jul 08 12:28:41.750124 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:41.750091    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a1658d8-64cb555d85-hf6nh"
Jul 08 12:28:41.897188 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:41.897161    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3a1658d8-64cb555d85-hf6nh"]
Jul 08 12:28:41.899138 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:28:41.899113    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd2e166f_686c_4912_ba6d_2408c75864c2.slice/crio-308722640e2001ca77f6f5d1667c660f53be0e405e5349ed5f071778280604ef WatchSource:0}: Error finding container 308722640e2001ca77f6f5d1667c660f53be0e405e5349ed5f071778280604ef: Status 404 returned error can't find the container with id 308722640e2001ca77f6f5d1667c660f53be0e405e5349ed5f071778280604ef
Jul 08 12:28:41.900410 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:41.900387    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 08 12:28:42.260160 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:42.260122    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a1658d8-64cb555d85-hf6nh" event={"ID":"bd2e166f-686c-4912-ba6d-2408c75864c2","Type":"ContainerStarted","Data":"308722640e2001ca77f6f5d1667c660f53be0e405e5349ed5f071778280604ef"}
Jul 08 12:28:43.266149 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:43.266114    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a1658d8-64cb555d85-hf6nh" event={"ID":"bd2e166f-686c-4912-ba6d-2408c75864c2","Type":"ContainerStarted","Data":"fa623f8c344c6e665458ab8078e2bfc5f2d8efe794ad5f852ca07e68c5eace6d"}
Jul 08 12:28:43.266651 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:43.266272    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a1658d8-64cb555d85-hf6nh"
Jul 08 12:28:43.288869 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:43.288813    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a1658d8-64cb555d85-hf6nh" podStartSLOduration=2.834738916 podStartE2EDuration="3.288797302s" totalImagesPullingTime="454.058386ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 12:28:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 12:28:43.287107382 +0000 UTC m=+1613.017482353" watchObservedRunningTime="2026-07-08 12:28:43.288797302 +0000 UTC m=+1613.019172307"
Jul 08 12:28:43.813026 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:43.812990    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:28:46.327894 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:46.327852    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3a1658d8-64cb555d85-hf6nh"]
Jul 08 12:28:46.328361 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:46.328087    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a1658d8-64cb555d85-hf6nh" podUID="bd2e166f-686c-4912-ba6d-2408c75864c2" containerName="maas-api" containerID="cri-o://fa623f8c344c6e665458ab8078e2bfc5f2d8efe794ad5f852ca07e68c5eace6d" gracePeriod=30
Jul 08 12:28:46.588471 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:46.588398    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a1658d8-64cb555d85-hf6nh"
Jul 08 12:28:46.652369 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:46.652338    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd2e166f-686c-4912-ba6d-2408c75864c2-maas-api-tls\") pod \"bd2e166f-686c-4912-ba6d-2408c75864c2\" (UID: \"bd2e166f-686c-4912-ba6d-2408c75864c2\") "
Jul 08 12:28:46.652571 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:46.652406    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd2e166f-686c-4912-ba6d-2408c75864c2-openshift-service-ca\") pod \"bd2e166f-686c-4912-ba6d-2408c75864c2\" (UID: \"bd2e166f-686c-4912-ba6d-2408c75864c2\") "
Jul 08 12:28:46.652571 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:46.652497    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x2vq8\" (UniqueName: \"kubernetes.io/projected/bd2e166f-686c-4912-ba6d-2408c75864c2-kube-api-access-x2vq8\") pod \"bd2e166f-686c-4912-ba6d-2408c75864c2\" (UID: \"bd2e166f-686c-4912-ba6d-2408c75864c2\") "
Jul 08 12:28:46.652764 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:46.652739    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd2e166f-686c-4912-ba6d-2408c75864c2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bd2e166f-686c-4912-ba6d-2408c75864c2" (UID: "bd2e166f-686c-4912-ba6d-2408c75864c2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 12:28:46.654818 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:46.654783    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd2e166f-686c-4912-ba6d-2408c75864c2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bd2e166f-686c-4912-ba6d-2408c75864c2" (UID: "bd2e166f-686c-4912-ba6d-2408c75864c2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 12:28:46.655083 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:46.655063    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd2e166f-686c-4912-ba6d-2408c75864c2-kube-api-access-x2vq8" (OuterVolumeSpecName: "kube-api-access-x2vq8") pod "bd2e166f-686c-4912-ba6d-2408c75864c2" (UID: "bd2e166f-686c-4912-ba6d-2408c75864c2"). InnerVolumeSpecName "kube-api-access-x2vq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 12:28:46.753922 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:46.753889    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2vq8\" (UniqueName: \"kubernetes.io/projected/bd2e166f-686c-4912-ba6d-2408c75864c2-kube-api-access-x2vq8\") on node \"ip-10-0-138-89.ec2.internal\" DevicePath \"\""
Jul 08 12:28:46.753922 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:46.753916    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd2e166f-686c-4912-ba6d-2408c75864c2-maas-api-tls\") on node \"ip-10-0-138-89.ec2.internal\" DevicePath \"\""
Jul 08 12:28:46.753922 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:46.753925    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd2e166f-686c-4912-ba6d-2408c75864c2-openshift-service-ca\") on node \"ip-10-0-138-89.ec2.internal\" DevicePath \"\""
Jul 08 12:28:47.286510 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:47.286476    2579 generic.go:356] "Generic (PLEG): container finished" podID="bd2e166f-686c-4912-ba6d-2408c75864c2" containerID="fa623f8c344c6e665458ab8078e2bfc5f2d8efe794ad5f852ca07e68c5eace6d" exitCode=0
Jul 08 12:28:47.286510 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:47.286516    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a1658d8-64cb555d85-hf6nh"
Jul 08 12:28:47.286829 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:47.286562    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a1658d8-64cb555d85-hf6nh" event={"ID":"bd2e166f-686c-4912-ba6d-2408c75864c2","Type":"ContainerDied","Data":"fa623f8c344c6e665458ab8078e2bfc5f2d8efe794ad5f852ca07e68c5eace6d"}
Jul 08 12:28:47.286829 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:47.286622    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a1658d8-64cb555d85-hf6nh" event={"ID":"bd2e166f-686c-4912-ba6d-2408c75864c2","Type":"ContainerDied","Data":"308722640e2001ca77f6f5d1667c660f53be0e405e5349ed5f071778280604ef"}
Jul 08 12:28:47.286829 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:47.286650    2579 scope.go:117] "RemoveContainer" containerID="fa623f8c344c6e665458ab8078e2bfc5f2d8efe794ad5f852ca07e68c5eace6d"
Jul 08 12:28:47.295724 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:47.295705    2579 scope.go:117] "RemoveContainer" containerID="fa623f8c344c6e665458ab8078e2bfc5f2d8efe794ad5f852ca07e68c5eace6d"
Jul 08 12:28:47.296027 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:28:47.296008    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa623f8c344c6e665458ab8078e2bfc5f2d8efe794ad5f852ca07e68c5eace6d\": container with ID starting with fa623f8c344c6e665458ab8078e2bfc5f2d8efe794ad5f852ca07e68c5eace6d not found: ID does not exist" containerID="fa623f8c344c6e665458ab8078e2bfc5f2d8efe794ad5f852ca07e68c5eace6d"
Jul 08 12:28:47.296113 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:47.296035    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa623f8c344c6e665458ab8078e2bfc5f2d8efe794ad5f852ca07e68c5eace6d"} err="failed to get container status \"fa623f8c344c6e665458ab8078e2bfc5f2d8efe794ad5f852ca07e68c5eace6d\": rpc error: code = NotFound desc = could not find container \"fa623f8c344c6e665458ab8078e2bfc5f2d8efe794ad5f852ca07e68c5eace6d\": container with ID starting with fa623f8c344c6e665458ab8078e2bfc5f2d8efe794ad5f852ca07e68c5eace6d not found: ID does not exist"
Jul 08 12:28:47.306178 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:47.306152    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3a1658d8-64cb555d85-hf6nh"]
Jul 08 12:28:47.310432 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:47.310409    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3a1658d8-64cb555d85-hf6nh"]
Jul 08 12:28:48.717646 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:48.717614    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd2e166f-686c-4912-ba6d-2408c75864c2" path="/var/lib/kubelet/pods/bd2e166f-686c-4912-ba6d-2408c75864c2/volumes"
Jul 08 12:28:54.121534 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:54.121499    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-16272b8a-6b74957f4b-5rs5p"]
Jul 08 12:28:54.121985 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:54.121971    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd2e166f-686c-4912-ba6d-2408c75864c2" containerName="maas-api"
Jul 08 12:28:54.122029 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:54.121987    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd2e166f-686c-4912-ba6d-2408c75864c2" containerName="maas-api"
Jul 08 12:28:54.122063 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:54.122049    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd2e166f-686c-4912-ba6d-2408c75864c2" containerName="maas-api"
Jul 08 12:28:54.125144 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:54.125117    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-16272b8a-6b74957f4b-5rs5p"
Jul 08 12:28:54.128527 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:54.128469    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-16272b8a-serving-cert\""
Jul 08 12:28:54.146880 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:54.146850    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-16272b8a-6b74957f4b-5rs5p"]
Jul 08 12:28:54.224905 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:54.224865    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dbb93c9-b23f-43dc-997f-8e8be7eda69a-openshift-service-ca\") pod \"maas-api-e2e-ait-16272b8a-6b74957f4b-5rs5p\" (UID: \"7dbb93c9-b23f-43dc-997f-8e8be7eda69a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-16272b8a-6b74957f4b-5rs5p"
Jul 08 12:28:54.224905 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:54.224913    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g48wd\" (UniqueName: \"kubernetes.io/projected/7dbb93c9-b23f-43dc-997f-8e8be7eda69a-kube-api-access-g48wd\") pod \"maas-api-e2e-ait-16272b8a-6b74957f4b-5rs5p\" (UID: \"7dbb93c9-b23f-43dc-997f-8e8be7eda69a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-16272b8a-6b74957f4b-5rs5p"
Jul 08 12:28:54.225156 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:54.224937    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7dbb93c9-b23f-43dc-997f-8e8be7eda69a-maas-api-tls\") pod \"maas-api-e2e-ait-16272b8a-6b74957f4b-5rs5p\" (UID: \"7dbb93c9-b23f-43dc-997f-8e8be7eda69a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-16272b8a-6b74957f4b-5rs5p"
Jul 08 12:28:54.325765 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:54.325726    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dbb93c9-b23f-43dc-997f-8e8be7eda69a-openshift-service-ca\") pod \"maas-api-e2e-ait-16272b8a-6b74957f4b-5rs5p\" (UID: \"7dbb93c9-b23f-43dc-997f-8e8be7eda69a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-16272b8a-6b74957f4b-5rs5p"
Jul 08 12:28:54.325954 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:54.325809    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g48wd\" (UniqueName: \"kubernetes.io/projected/7dbb93c9-b23f-43dc-997f-8e8be7eda69a-kube-api-access-g48wd\") pod \"maas-api-e2e-ait-16272b8a-6b74957f4b-5rs5p\" (UID: \"7dbb93c9-b23f-43dc-997f-8e8be7eda69a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-16272b8a-6b74957f4b-5rs5p"
Jul 08 12:28:54.325954 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:54.325841    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7dbb93c9-b23f-43dc-997f-8e8be7eda69a-maas-api-tls\") pod \"maas-api-e2e-ait-16272b8a-6b74957f4b-5rs5p\" (UID: \"7dbb93c9-b23f-43dc-997f-8e8be7eda69a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-16272b8a-6b74957f4b-5rs5p"
Jul 08 12:28:54.326516 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:54.326488    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dbb93c9-b23f-43dc-997f-8e8be7eda69a-openshift-service-ca\") pod \"maas-api-e2e-ait-16272b8a-6b74957f4b-5rs5p\" (UID: \"7dbb93c9-b23f-43dc-997f-8e8be7eda69a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-16272b8a-6b74957f4b-5rs5p"
Jul 08 12:28:54.328538 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:54.328506    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7dbb93c9-b23f-43dc-997f-8e8be7eda69a-maas-api-tls\") pod \"maas-api-e2e-ait-16272b8a-6b74957f4b-5rs5p\" (UID: \"7dbb93c9-b23f-43dc-997f-8e8be7eda69a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-16272b8a-6b74957f4b-5rs5p"
Jul 08 12:28:54.340076 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:54.340047    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g48wd\" (UniqueName: \"kubernetes.io/projected/7dbb93c9-b23f-43dc-997f-8e8be7eda69a-kube-api-access-g48wd\") pod \"maas-api-e2e-ait-16272b8a-6b74957f4b-5rs5p\" (UID: \"7dbb93c9-b23f-43dc-997f-8e8be7eda69a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-16272b8a-6b74957f4b-5rs5p"
Jul 08 12:28:54.439239 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:54.439196    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-16272b8a-6b74957f4b-5rs5p"
Jul 08 12:28:54.603659 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:54.603506    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-16272b8a-6b74957f4b-5rs5p"]
Jul 08 12:28:54.606134 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:28:54.606093    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dbb93c9_b23f_43dc_997f_8e8be7eda69a.slice/crio-430a5738b337fe72ec55643370c83c075ef24b0ad766c9605d19598c2c6be698 WatchSource:0}: Error finding container 430a5738b337fe72ec55643370c83c075ef24b0ad766c9605d19598c2c6be698: Status 404 returned error can't find the container with id 430a5738b337fe72ec55643370c83c075ef24b0ad766c9605d19598c2c6be698
Jul 08 12:28:55.330432 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:55.330396    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-16272b8a-6b74957f4b-5rs5p" event={"ID":"7dbb93c9-b23f-43dc-997f-8e8be7eda69a","Type":"ContainerStarted","Data":"f5a8d02a457c1962b1a73d3eff7bc95cff4ec17d98dfdb0d2bd488d5783080c2"}
Jul 08 12:28:55.330852 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:55.330457    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-16272b8a-6b74957f4b-5rs5p" event={"ID":"7dbb93c9-b23f-43dc-997f-8e8be7eda69a","Type":"ContainerStarted","Data":"430a5738b337fe72ec55643370c83c075ef24b0ad766c9605d19598c2c6be698"}
Jul 08 12:28:55.330852 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:55.330480    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-16272b8a-6b74957f4b-5rs5p"
Jul 08 12:28:55.373995 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:55.373940    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-16272b8a-6b74957f4b-5rs5p" podStartSLOduration=0.848163051 podStartE2EDuration="1.373923004s" totalImagesPullingTime="525.759953ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 12:28:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 12:28:55.373057646 +0000 UTC m=+1625.103432615" watchObservedRunningTime="2026-07-08 12:28:55.373923004 +0000 UTC m=+1625.104297974"
Jul 08 12:28:57.582648 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:57.582605    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:28:59.404738 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:59.404701    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-16272b8a-6b74957f4b-5rs5p"]
Jul 08 12:28:59.405190 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:59.405013    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-16272b8a-6b74957f4b-5rs5p" podUID="7dbb93c9-b23f-43dc-997f-8e8be7eda69a" containerName="maas-api" containerID="cri-o://f5a8d02a457c1962b1a73d3eff7bc95cff4ec17d98dfdb0d2bd488d5783080c2" gracePeriod=30
Jul 08 12:28:59.662876 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:59.662801    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-16272b8a-6b74957f4b-5rs5p"
Jul 08 12:28:59.778060 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:59.778007    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dbb93c9-b23f-43dc-997f-8e8be7eda69a-openshift-service-ca\") pod \"7dbb93c9-b23f-43dc-997f-8e8be7eda69a\" (UID: \"7dbb93c9-b23f-43dc-997f-8e8be7eda69a\") "
Jul 08 12:28:59.778274 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:59.778161    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g48wd\" (UniqueName: \"kubernetes.io/projected/7dbb93c9-b23f-43dc-997f-8e8be7eda69a-kube-api-access-g48wd\") pod \"7dbb93c9-b23f-43dc-997f-8e8be7eda69a\" (UID: \"7dbb93c9-b23f-43dc-997f-8e8be7eda69a\") "
Jul 08 12:28:59.778274 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:59.778186    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7dbb93c9-b23f-43dc-997f-8e8be7eda69a-maas-api-tls\") pod \"7dbb93c9-b23f-43dc-997f-8e8be7eda69a\" (UID: \"7dbb93c9-b23f-43dc-997f-8e8be7eda69a\") "
Jul 08 12:28:59.778504 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:59.778475    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dbb93c9-b23f-43dc-997f-8e8be7eda69a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7dbb93c9-b23f-43dc-997f-8e8be7eda69a" (UID: "7dbb93c9-b23f-43dc-997f-8e8be7eda69a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 12:28:59.780478 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:59.780444    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dbb93c9-b23f-43dc-997f-8e8be7eda69a-kube-api-access-g48wd" (OuterVolumeSpecName: "kube-api-access-g48wd") pod "7dbb93c9-b23f-43dc-997f-8e8be7eda69a" (UID: "7dbb93c9-b23f-43dc-997f-8e8be7eda69a"). InnerVolumeSpecName "kube-api-access-g48wd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 12:28:59.780478 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:59.780447    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7dbb93c9-b23f-43dc-997f-8e8be7eda69a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7dbb93c9-b23f-43dc-997f-8e8be7eda69a" (UID: "7dbb93c9-b23f-43dc-997f-8e8be7eda69a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 12:28:59.879838 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:59.879753    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dbb93c9-b23f-43dc-997f-8e8be7eda69a-openshift-service-ca\") on node \"ip-10-0-138-89.ec2.internal\" DevicePath \"\""
Jul 08 12:28:59.879838 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:59.879831    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g48wd\" (UniqueName: \"kubernetes.io/projected/7dbb93c9-b23f-43dc-997f-8e8be7eda69a-kube-api-access-g48wd\") on node \"ip-10-0-138-89.ec2.internal\" DevicePath \"\""
Jul 08 12:28:59.879838 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:28:59.879842    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7dbb93c9-b23f-43dc-997f-8e8be7eda69a-maas-api-tls\") on node \"ip-10-0-138-89.ec2.internal\" DevicePath \"\""
Jul 08 12:29:00.357235 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:29:00.357197    2579 generic.go:356] "Generic (PLEG): container finished" podID="7dbb93c9-b23f-43dc-997f-8e8be7eda69a" containerID="f5a8d02a457c1962b1a73d3eff7bc95cff4ec17d98dfdb0d2bd488d5783080c2" exitCode=0
Jul 08 12:29:00.357235 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:29:00.357241    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-16272b8a-6b74957f4b-5rs5p"
Jul 08 12:29:00.357447 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:29:00.357254    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-16272b8a-6b74957f4b-5rs5p" event={"ID":"7dbb93c9-b23f-43dc-997f-8e8be7eda69a","Type":"ContainerDied","Data":"f5a8d02a457c1962b1a73d3eff7bc95cff4ec17d98dfdb0d2bd488d5783080c2"}
Jul 08 12:29:00.357447 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:29:00.357312    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-16272b8a-6b74957f4b-5rs5p" event={"ID":"7dbb93c9-b23f-43dc-997f-8e8be7eda69a","Type":"ContainerDied","Data":"430a5738b337fe72ec55643370c83c075ef24b0ad766c9605d19598c2c6be698"}
Jul 08 12:29:00.357447 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:29:00.357341    2579 scope.go:117] "RemoveContainer" containerID="f5a8d02a457c1962b1a73d3eff7bc95cff4ec17d98dfdb0d2bd488d5783080c2"
Jul 08 12:29:00.370699 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:29:00.370676    2579 scope.go:117] "RemoveContainer" containerID="f5a8d02a457c1962b1a73d3eff7bc95cff4ec17d98dfdb0d2bd488d5783080c2"
Jul 08 12:29:00.371019 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:29:00.371000    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5a8d02a457c1962b1a73d3eff7bc95cff4ec17d98dfdb0d2bd488d5783080c2\": container with ID starting with f5a8d02a457c1962b1a73d3eff7bc95cff4ec17d98dfdb0d2bd488d5783080c2 not found: ID does not exist" containerID="f5a8d02a457c1962b1a73d3eff7bc95cff4ec17d98dfdb0d2bd488d5783080c2"
Jul 08 12:29:00.371073 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:29:00.371029    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5a8d02a457c1962b1a73d3eff7bc95cff4ec17d98dfdb0d2bd488d5783080c2"} err="failed to get container status \"f5a8d02a457c1962b1a73d3eff7bc95cff4ec17d98dfdb0d2bd488d5783080c2\": rpc error: code = NotFound desc = could not find container \"f5a8d02a457c1962b1a73d3eff7bc95cff4ec17d98dfdb0d2bd488d5783080c2\": container with ID starting with f5a8d02a457c1962b1a73d3eff7bc95cff4ec17d98dfdb0d2bd488d5783080c2 not found: ID does not exist"
Jul 08 12:29:00.381719 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:29:00.381688    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-16272b8a-6b74957f4b-5rs5p"]
Jul 08 12:29:00.386217 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:29:00.386181    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-16272b8a-6b74957f4b-5rs5p"]
Jul 08 12:29:00.716548 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:29:00.716506    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7dbb93c9-b23f-43dc-997f-8e8be7eda69a" path="/var/lib/kubelet/pods/7dbb93c9-b23f-43dc-997f-8e8be7eda69a/volumes"
Jul 08 12:29:09.818820 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:29:09.818783    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:29:13.774921 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:29:13.774886    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3cb60f5b-799d667687-kvqjr"]
Jul 08 12:29:13.775318 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:29:13.775284    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7dbb93c9-b23f-43dc-997f-8e8be7eda69a" containerName="maas-api"
Jul 08 12:29:13.775318 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:29:13.775294    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7dbb93c9-b23f-43dc-997f-8e8be7eda69a" containerName="maas-api"
Jul 08 12:29:13.775396 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:29:13.775361    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7dbb93c9-b23f-43dc-997f-8e8be7eda69a" containerName="maas-api"
Jul 08 12:29:13.780069 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:29:13.780045    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3cb60f5b-799d667687-kvqjr"
Jul 08 12:29:13.783167 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:29:13.783145    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-3cb60f5b-serving-cert\""
Jul 08 12:29:13.797863 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:29:13.797664    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3cb60f5b-799d667687-kvqjr"]
Jul 08 12:29:13.922806 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:29:13.922727    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21ad121a-b7bf-4b19-b183-c9ef4861a297-openshift-service-ca\") pod \"maas-api-e2e-derive-3cb60f5b-799d667687-kvqjr\" (UID: \"21ad121a-b7bf-4b19-b183-c9ef4861a297\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3cb60f5b-799d667687-kvqjr"
Jul 08 12:29:13.922806 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:29:13.922798    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkkfp\" (UniqueName: \"kubernetes.io/projected/21ad121a-b7bf-4b19-b183-c9ef4861a297-kube-api-access-vkkfp\") pod \"maas-api-e2e-derive-3cb60f5b-799d667687-kvqjr\" (UID: \"21ad121a-b7bf-4b19-b183-c9ef4861a297\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3cb60f5b-799d667687-kvqjr"
Jul 08 12:29:13.923044 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:29:13.922886    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21ad121a-b7bf-4b19-b183-c9ef4861a297-maas-api-tls\") pod \"maas-api-e2e-derive-3cb60f5b-799d667687-kvqjr\" (UID: \"21ad121a-b7bf-4b19-b183-c9ef4861a297\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3cb60f5b-799d667687-kvqjr"
Jul 08 12:29:14.023736 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:29:14.023694    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21ad121a-b7bf-4b19-b183-c9ef4861a297-openshift-service-ca\") pod \"maas-api-e2e-derive-3cb60f5b-799d667687-kvqjr\" (UID: \"21ad121a-b7bf-4b19-b183-c9ef4861a297\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3cb60f5b-799d667687-kvqjr"
Jul 08 12:29:14.023736 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:29:14.023738    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkkfp\" (UniqueName: \"kubernetes.io/projected/21ad121a-b7bf-4b19-b183-c9ef4861a297-kube-api-access-vkkfp\") pod \"maas-api-e2e-derive-3cb60f5b-799d667687-kvqjr\" (UID: \"21ad121a-b7bf-4b19-b183-c9ef4861a297\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3cb60f5b-799d667687-kvqjr"
Jul 08 12:29:14.024141 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:29:14.023794    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21ad121a-b7bf-4b19-b183-c9ef4861a297-maas-api-tls\") pod \"maas-api-e2e-derive-3cb60f5b-799d667687-kvqjr\" (UID: \"21ad121a-b7bf-4b19-b183-c9ef4861a297\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3cb60f5b-799d667687-kvqjr"
Jul 08 12:29:14.024141 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:29:14.023928    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-3cb60f5b-serving-cert: secret "maas-api-e2e-derive-3cb60f5b-serving-cert" not found
Jul 08 12:29:14.024141 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:29:14.024002    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21ad121a-b7bf-4b19-b183-c9ef4861a297-maas-api-tls podName:21ad121a-b7bf-4b19-b183-c9ef4861a297 nodeName:}" failed. No retries permitted until 2026-07-08 12:29:14.523979492 +0000 UTC m=+1644.254354441 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/21ad121a-b7bf-4b19-b183-c9ef4861a297-maas-api-tls") pod "maas-api-e2e-derive-3cb60f5b-799d667687-kvqjr" (UID: "21ad121a-b7bf-4b19-b183-c9ef4861a297") : secret "maas-api-e2e-derive-3cb60f5b-serving-cert" not found
Jul 08 12:29:14.024499 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:29:14.024472    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21ad121a-b7bf-4b19-b183-c9ef4861a297-openshift-service-ca\") pod \"maas-api-e2e-derive-3cb60f5b-799d667687-kvqjr\" (UID: \"21ad121a-b7bf-4b19-b183-c9ef4861a297\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3cb60f5b-799d667687-kvqjr"
Jul 08 12:29:14.038230 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:29:14.038165    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkkfp\" (UniqueName: \"kubernetes.io/projected/21ad121a-b7bf-4b19-b183-c9ef4861a297-kube-api-access-vkkfp\") pod \"maas-api-e2e-derive-3cb60f5b-799d667687-kvqjr\" (UID: \"21ad121a-b7bf-4b19-b183-c9ef4861a297\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3cb60f5b-799d667687-kvqjr"
Jul 08 12:29:14.528977 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:29:14.528945    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21ad121a-b7bf-4b19-b183-c9ef4861a297-maas-api-tls\") pod \"maas-api-e2e-derive-3cb60f5b-799d667687-kvqjr\" (UID: \"21ad121a-b7bf-4b19-b183-c9ef4861a297\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3cb60f5b-799d667687-kvqjr"
Jul 08 12:29:14.531638 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:29:14.531611    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21ad121a-b7bf-4b19-b183-c9ef4861a297-maas-api-tls\") pod \"maas-api-e2e-derive-3cb60f5b-799d667687-kvqjr\" (UID: \"21ad121a-b7bf-4b19-b183-c9ef4861a297\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3cb60f5b-799d667687-kvqjr"
Jul 08 12:29:14.691728 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:29:14.691683    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3cb60f5b-799d667687-kvqjr"
Jul 08 12:29:14.857606 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:29:14.857558    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21ad121a_b7bf_4b19_b183_c9ef4861a297.slice/crio-f44704b3150b236f0be8042f9ecbdef816e3e2d0391a5905b5a2a0d06f54d8ff WatchSource:0}: Error finding container f44704b3150b236f0be8042f9ecbdef816e3e2d0391a5905b5a2a0d06f54d8ff: Status 404 returned error can't find the container with id f44704b3150b236f0be8042f9ecbdef816e3e2d0391a5905b5a2a0d06f54d8ff
Jul 08 12:29:14.858043 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:29:14.857673    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3cb60f5b-799d667687-kvqjr"]
Jul 08 12:29:15.433249 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:29:15.433212    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3cb60f5b-799d667687-kvqjr" event={"ID":"21ad121a-b7bf-4b19-b183-c9ef4861a297","Type":"ContainerStarted","Data":"2c740fb70405e21640f9364331ccc53443f5f069b46abf201ce97edc343af150"}
Jul 08 12:29:15.433453 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:29:15.433311    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3cb60f5b-799d667687-kvqjr" event={"ID":"21ad121a-b7bf-4b19-b183-c9ef4861a297","Type":"ContainerStarted","Data":"f44704b3150b236f0be8042f9ecbdef816e3e2d0391a5905b5a2a0d06f54d8ff"}
Jul 08 12:29:15.433453 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:29:15.433366    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3cb60f5b-799d667687-kvqjr"
Jul 08 12:29:15.453301 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:29:15.453241    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3cb60f5b-799d667687-kvqjr" podStartSLOduration=2.037096305 podStartE2EDuration="2.453222111s" totalImagesPullingTime="416.125806ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 12:29:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 12:29:15.450980083 +0000 UTC m=+1645.181355054" watchObservedRunningTime="2026-07-08 12:29:15.453222111 +0000 UTC m=+1645.183597081"
Jul 08 12:29:18.020679 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:29:18.020630    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:29:18.566243 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:29:18.566203    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3cb60f5b-799d667687-kvqjr"]
Jul 08 12:29:18.566468 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:29:18.566445    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3cb60f5b-799d667687-kvqjr" podUID="21ad121a-b7bf-4b19-b183-c9ef4861a297" containerName="maas-api" containerID="cri-o://2c740fb70405e21640f9364331ccc53443f5f069b46abf201ce97edc343af150" gracePeriod=30
Jul 08 12:29:18.829414 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:29:18.829336    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3cb60f5b-799d667687-kvqjr"
Jul 08 12:29:18.972164 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:29:18.972127    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vkkfp\" (UniqueName: \"kubernetes.io/projected/21ad121a-b7bf-4b19-b183-c9ef4861a297-kube-api-access-vkkfp\") pod \"21ad121a-b7bf-4b19-b183-c9ef4861a297\" (UID: \"21ad121a-b7bf-4b19-b183-c9ef4861a297\") "
Jul 08 12:29:18.972346 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:29:18.972260    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21ad121a-b7bf-4b19-b183-c9ef4861a297-maas-api-tls\") pod \"21ad121a-b7bf-4b19-b183-c9ef4861a297\" (UID: \"21ad121a-b7bf-4b19-b183-c9ef4861a297\") "
Jul 08 12:29:18.972346 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:29:18.972288    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21ad121a-b7bf-4b19-b183-c9ef4861a297-openshift-service-ca\") pod \"21ad121a-b7bf-4b19-b183-c9ef4861a297\" (UID: \"21ad121a-b7bf-4b19-b183-c9ef4861a297\") "
Jul 08 12:29:18.972676 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:29:18.972647    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21ad121a-b7bf-4b19-b183-c9ef4861a297-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "21ad121a-b7bf-4b19-b183-c9ef4861a297" (UID: "21ad121a-b7bf-4b19-b183-c9ef4861a297"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 12:29:18.974519 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:29:18.974488    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21ad121a-b7bf-4b19-b183-c9ef4861a297-kube-api-access-vkkfp" (OuterVolumeSpecName: "kube-api-access-vkkfp") pod "21ad121a-b7bf-4b19-b183-c9ef4861a297" (UID: "21ad121a-b7bf-4b19-b183-c9ef4861a297"). InnerVolumeSpecName "kube-api-access-vkkfp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 12:29:18.974656 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:29:18.974487    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21ad121a-b7bf-4b19-b183-c9ef4861a297-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "21ad121a-b7bf-4b19-b183-c9ef4861a297" (UID: "21ad121a-b7bf-4b19-b183-c9ef4861a297"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 12:29:19.073353 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:29:19.073310    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21ad121a-b7bf-4b19-b183-c9ef4861a297-maas-api-tls\") on node \"ip-10-0-138-89.ec2.internal\" DevicePath \"\""
Jul 08 12:29:19.073353 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:29:19.073347    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21ad121a-b7bf-4b19-b183-c9ef4861a297-openshift-service-ca\") on node \"ip-10-0-138-89.ec2.internal\" DevicePath \"\""
Jul 08 12:29:19.073353 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:29:19.073356    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkkfp\" (UniqueName: \"kubernetes.io/projected/21ad121a-b7bf-4b19-b183-c9ef4861a297-kube-api-access-vkkfp\") on node \"ip-10-0-138-89.ec2.internal\" DevicePath \"\""
Jul 08 12:29:19.455438 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:29:19.455405    2579 generic.go:356] "Generic (PLEG): container finished" podID="21ad121a-b7bf-4b19-b183-c9ef4861a297" containerID="2c740fb70405e21640f9364331ccc53443f5f069b46abf201ce97edc343af150" exitCode=0
Jul 08 12:29:19.455626 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:29:19.455471    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3cb60f5b-799d667687-kvqjr" event={"ID":"21ad121a-b7bf-4b19-b183-c9ef4861a297","Type":"ContainerDied","Data":"2c740fb70405e21640f9364331ccc53443f5f069b46abf201ce97edc343af150"}
Jul 08 12:29:19.455626 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:29:19.455511    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3cb60f5b-799d667687-kvqjr" event={"ID":"21ad121a-b7bf-4b19-b183-c9ef4861a297","Type":"ContainerDied","Data":"f44704b3150b236f0be8042f9ecbdef816e3e2d0391a5905b5a2a0d06f54d8ff"}
Jul 08 12:29:19.455626 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:29:19.455533    2579 scope.go:117] "RemoveContainer" containerID="2c740fb70405e21640f9364331ccc53443f5f069b46abf201ce97edc343af150"
Jul 08 12:29:19.455626 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:29:19.455456    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3cb60f5b-799d667687-kvqjr"
Jul 08 12:29:19.465151 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:29:19.465126    2579 scope.go:117] "RemoveContainer" containerID="2c740fb70405e21640f9364331ccc53443f5f069b46abf201ce97edc343af150"
Jul 08 12:29:19.465463 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:29:19.465437    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c740fb70405e21640f9364331ccc53443f5f069b46abf201ce97edc343af150\": container with ID starting with 2c740fb70405e21640f9364331ccc53443f5f069b46abf201ce97edc343af150 not found: ID does not exist" containerID="2c740fb70405e21640f9364331ccc53443f5f069b46abf201ce97edc343af150"
Jul 08 12:29:19.465538 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:29:19.465475    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c740fb70405e21640f9364331ccc53443f5f069b46abf201ce97edc343af150"} err="failed to get container status \"2c740fb70405e21640f9364331ccc53443f5f069b46abf201ce97edc343af150\": rpc error: code = NotFound desc = could not find container \"2c740fb70405e21640f9364331ccc53443f5f069b46abf201ce97edc343af150\": container with ID starting with 2c740fb70405e21640f9364331ccc53443f5f069b46abf201ce97edc343af150 not found: ID does not exist"
Jul 08 12:29:19.478915 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:29:19.478876    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3cb60f5b-799d667687-kvqjr"]
Jul 08 12:29:19.482705 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:29:19.482677    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3cb60f5b-799d667687-kvqjr"]
Jul 08 12:29:20.717084 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:29:20.717049    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21ad121a-b7bf-4b19-b183-c9ef4861a297" path="/var/lib/kubelet/pods/21ad121a-b7bf-4b19-b183-c9ef4861a297/volumes"
Jul 08 12:29:29.228905 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:29:29.228863    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:30:52.114338 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:30:52.114249    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:30:59.906155 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:30:59.906112    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:31:08.911357 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:31:08.911313    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:31:09.080480 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:31:09.080442    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-49ca3975-7f499bf4fc-v58xl"]
Jul 08 12:31:09.080916 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:31:09.080901    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21ad121a-b7bf-4b19-b183-c9ef4861a297" containerName="maas-api"
Jul 08 12:31:09.080974 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:31:09.080919    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="21ad121a-b7bf-4b19-b183-c9ef4861a297" containerName="maas-api"
Jul 08 12:31:09.081014 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:31:09.080997    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="21ad121a-b7bf-4b19-b183-c9ef4861a297" containerName="maas-api"
Jul 08 12:31:09.084401 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:31:09.084376    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-49ca3975-7f499bf4fc-v58xl"
Jul 08 12:31:09.087142 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:31:09.087117    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-49ca3975-serving-cert\""
Jul 08 12:31:09.094426 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:31:09.094399    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-49ca3975-7f499bf4fc-v58xl"]
Jul 08 12:31:09.186713 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:31:09.186677    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdbf7\" (UniqueName: \"kubernetes.io/projected/00cf80ef-9a09-44b5-a26e-b36a29641b36-kube-api-access-fdbf7\") pod \"maas-api-e2e-mt-49ca3975-7f499bf4fc-v58xl\" (UID: \"00cf80ef-9a09-44b5-a26e-b36a29641b36\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-49ca3975-7f499bf4fc-v58xl"
Jul 08 12:31:09.186713 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:31:09.186713    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00cf80ef-9a09-44b5-a26e-b36a29641b36-openshift-service-ca\") pod \"maas-api-e2e-mt-49ca3975-7f499bf4fc-v58xl\" (UID: \"00cf80ef-9a09-44b5-a26e-b36a29641b36\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-49ca3975-7f499bf4fc-v58xl"
Jul 08 12:31:09.186978 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:31:09.186741    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00cf80ef-9a09-44b5-a26e-b36a29641b36-maas-api-tls\") pod \"maas-api-e2e-mt-49ca3975-7f499bf4fc-v58xl\" (UID: \"00cf80ef-9a09-44b5-a26e-b36a29641b36\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-49ca3975-7f499bf4fc-v58xl"
Jul 08 12:31:09.287872 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:31:09.287829    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdbf7\" (UniqueName: \"kubernetes.io/projected/00cf80ef-9a09-44b5-a26e-b36a29641b36-kube-api-access-fdbf7\") pod \"maas-api-e2e-mt-49ca3975-7f499bf4fc-v58xl\" (UID: \"00cf80ef-9a09-44b5-a26e-b36a29641b36\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-49ca3975-7f499bf4fc-v58xl"
Jul 08 12:31:09.288075 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:31:09.287886    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00cf80ef-9a09-44b5-a26e-b36a29641b36-openshift-service-ca\") pod \"maas-api-e2e-mt-49ca3975-7f499bf4fc-v58xl\" (UID: \"00cf80ef-9a09-44b5-a26e-b36a29641b36\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-49ca3975-7f499bf4fc-v58xl"
Jul 08 12:31:09.288075 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:31:09.287924    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00cf80ef-9a09-44b5-a26e-b36a29641b36-maas-api-tls\") pod \"maas-api-e2e-mt-49ca3975-7f499bf4fc-v58xl\" (UID: \"00cf80ef-9a09-44b5-a26e-b36a29641b36\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-49ca3975-7f499bf4fc-v58xl"
Jul 08 12:31:09.288503 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:31:09.288480    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00cf80ef-9a09-44b5-a26e-b36a29641b36-openshift-service-ca\") pod \"maas-api-e2e-mt-49ca3975-7f499bf4fc-v58xl\" (UID: \"00cf80ef-9a09-44b5-a26e-b36a29641b36\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-49ca3975-7f499bf4fc-v58xl"
Jul 08 12:31:09.290613 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:31:09.290587    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00cf80ef-9a09-44b5-a26e-b36a29641b36-maas-api-tls\") pod \"maas-api-e2e-mt-49ca3975-7f499bf4fc-v58xl\" (UID: \"00cf80ef-9a09-44b5-a26e-b36a29641b36\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-49ca3975-7f499bf4fc-v58xl"
Jul 08 12:31:09.300114 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:31:09.300088    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdbf7\" (UniqueName: \"kubernetes.io/projected/00cf80ef-9a09-44b5-a26e-b36a29641b36-kube-api-access-fdbf7\") pod \"maas-api-e2e-mt-49ca3975-7f499bf4fc-v58xl\" (UID: \"00cf80ef-9a09-44b5-a26e-b36a29641b36\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-49ca3975-7f499bf4fc-v58xl"
Jul 08 12:31:09.397496 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:31:09.397457    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-49ca3975-7f499bf4fc-v58xl"
Jul 08 12:31:09.543181 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:31:09.543156    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-49ca3975-7f499bf4fc-v58xl"]
Jul 08 12:31:09.545132 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:31:09.545096    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00cf80ef_9a09_44b5_a26e_b36a29641b36.slice/crio-1b710e2918952b5515952ee8f9a6250c187b138daa0bb9c8d96bbbf15c502c20 WatchSource:0}: Error finding container 1b710e2918952b5515952ee8f9a6250c187b138daa0bb9c8d96bbbf15c502c20: Status 404 returned error can't find the container with id 1b710e2918952b5515952ee8f9a6250c187b138daa0bb9c8d96bbbf15c502c20
Jul 08 12:31:10.012862 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:31:10.012821    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-49ca3975-7f499bf4fc-v58xl" event={"ID":"00cf80ef-9a09-44b5-a26e-b36a29641b36","Type":"ContainerStarted","Data":"1b710e2918952b5515952ee8f9a6250c187b138daa0bb9c8d96bbbf15c502c20"}
Jul 08 12:31:11.021843 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:31:11.021807    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-49ca3975-7f499bf4fc-v58xl" event={"ID":"00cf80ef-9a09-44b5-a26e-b36a29641b36","Type":"ContainerStarted","Data":"1b9fb736bfed8333e941efd29b3469ac9c8d19f8f1c1bf4a1e8f92e3c93493bc"}
Jul 08 12:31:11.022276 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:31:11.021930    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-49ca3975-7f499bf4fc-v58xl"
Jul 08 12:31:11.044246 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:31:11.044191    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-49ca3975-7f499bf4fc-v58xl" podStartSLOduration=1.471159217 podStartE2EDuration="2.044173082s" totalImagesPullingTime="573.013865ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 12:31:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 12:31:11.041961576 +0000 UTC m=+1760.772336547" watchObservedRunningTime="2026-07-08 12:31:11.044173082 +0000 UTC m=+1760.774548059"
Jul 08 12:31:17.033341 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:31:17.033304    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-49ca3975-7f499bf4fc-v58xl"
Jul 08 12:31:21.253511 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:31:21.253480    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:31:21.780978 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:31:21.780940    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-49ca3975-7f499bf4fc-v58xl"]
Jul 08 12:31:21.781247 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:31:21.781216    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-49ca3975-7f499bf4fc-v58xl" podUID="00cf80ef-9a09-44b5-a26e-b36a29641b36" containerName="maas-api" containerID="cri-o://1b9fb736bfed8333e941efd29b3469ac9c8d19f8f1c1bf4a1e8f92e3c93493bc" gracePeriod=30
Jul 08 12:31:22.044985 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:31:22.044906    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-49ca3975-7f499bf4fc-v58xl"
Jul 08 12:31:22.080234 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:31:22.080197    2579 generic.go:356] "Generic (PLEG): container finished" podID="00cf80ef-9a09-44b5-a26e-b36a29641b36" containerID="1b9fb736bfed8333e941efd29b3469ac9c8d19f8f1c1bf4a1e8f92e3c93493bc" exitCode=0
Jul 08 12:31:22.080414 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:31:22.080256    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-49ca3975-7f499bf4fc-v58xl"
Jul 08 12:31:22.080414 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:31:22.080277    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-49ca3975-7f499bf4fc-v58xl" event={"ID":"00cf80ef-9a09-44b5-a26e-b36a29641b36","Type":"ContainerDied","Data":"1b9fb736bfed8333e941efd29b3469ac9c8d19f8f1c1bf4a1e8f92e3c93493bc"}
Jul 08 12:31:22.080414 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:31:22.080332    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-49ca3975-7f499bf4fc-v58xl" event={"ID":"00cf80ef-9a09-44b5-a26e-b36a29641b36","Type":"ContainerDied","Data":"1b710e2918952b5515952ee8f9a6250c187b138daa0bb9c8d96bbbf15c502c20"}
Jul 08 12:31:22.080414 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:31:22.080359    2579 scope.go:117] "RemoveContainer" containerID="1b9fb736bfed8333e941efd29b3469ac9c8d19f8f1c1bf4a1e8f92e3c93493bc"
Jul 08 12:31:22.090972 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:31:22.090954    2579 scope.go:117] "RemoveContainer" containerID="1b9fb736bfed8333e941efd29b3469ac9c8d19f8f1c1bf4a1e8f92e3c93493bc"
Jul 08 12:31:22.091288 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:31:22.091270    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b9fb736bfed8333e941efd29b3469ac9c8d19f8f1c1bf4a1e8f92e3c93493bc\": container with ID starting with 1b9fb736bfed8333e941efd29b3469ac9c8d19f8f1c1bf4a1e8f92e3c93493bc not found: ID does not exist" containerID="1b9fb736bfed8333e941efd29b3469ac9c8d19f8f1c1bf4a1e8f92e3c93493bc"
Jul 08 12:31:22.091359 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:31:22.091302    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b9fb736bfed8333e941efd29b3469ac9c8d19f8f1c1bf4a1e8f92e3c93493bc"} err="failed to get container status \"1b9fb736bfed8333e941efd29b3469ac9c8d19f8f1c1bf4a1e8f92e3c93493bc\": rpc error: code = NotFound desc = could not find container \"1b9fb736bfed8333e941efd29b3469ac9c8d19f8f1c1bf4a1e8f92e3c93493bc\": container with ID starting with 1b9fb736bfed8333e941efd29b3469ac9c8d19f8f1c1bf4a1e8f92e3c93493bc not found: ID does not exist"
Jul 08 12:31:22.219127 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:31:22.219091    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00cf80ef-9a09-44b5-a26e-b36a29641b36-maas-api-tls\") pod \"00cf80ef-9a09-44b5-a26e-b36a29641b36\" (UID: \"00cf80ef-9a09-44b5-a26e-b36a29641b36\") "
Jul 08 12:31:22.219311 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:31:22.219148    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fdbf7\" (UniqueName: \"kubernetes.io/projected/00cf80ef-9a09-44b5-a26e-b36a29641b36-kube-api-access-fdbf7\") pod \"00cf80ef-9a09-44b5-a26e-b36a29641b36\" (UID: \"00cf80ef-9a09-44b5-a26e-b36a29641b36\") "
Jul 08 12:31:22.219311 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:31:22.219197    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00cf80ef-9a09-44b5-a26e-b36a29641b36-openshift-service-ca\") pod \"00cf80ef-9a09-44b5-a26e-b36a29641b36\" (UID: \"00cf80ef-9a09-44b5-a26e-b36a29641b36\") "
Jul 08 12:31:22.219614 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:31:22.219589    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00cf80ef-9a09-44b5-a26e-b36a29641b36-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "00cf80ef-9a09-44b5-a26e-b36a29641b36" (UID: "00cf80ef-9a09-44b5-a26e-b36a29641b36"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 12:31:22.221338 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:31:22.221309    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00cf80ef-9a09-44b5-a26e-b36a29641b36-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "00cf80ef-9a09-44b5-a26e-b36a29641b36" (UID: "00cf80ef-9a09-44b5-a26e-b36a29641b36"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 12:31:22.221445 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:31:22.221372    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00cf80ef-9a09-44b5-a26e-b36a29641b36-kube-api-access-fdbf7" (OuterVolumeSpecName: "kube-api-access-fdbf7") pod "00cf80ef-9a09-44b5-a26e-b36a29641b36" (UID: "00cf80ef-9a09-44b5-a26e-b36a29641b36"). InnerVolumeSpecName "kube-api-access-fdbf7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 12:31:22.320133 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:31:22.320048    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00cf80ef-9a09-44b5-a26e-b36a29641b36-maas-api-tls\") on node \"ip-10-0-138-89.ec2.internal\" DevicePath \"\""
Jul 08 12:31:22.320133 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:31:22.320078    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdbf7\" (UniqueName: \"kubernetes.io/projected/00cf80ef-9a09-44b5-a26e-b36a29641b36-kube-api-access-fdbf7\") on node \"ip-10-0-138-89.ec2.internal\" DevicePath \"\""
Jul 08 12:31:22.320133 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:31:22.320089    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00cf80ef-9a09-44b5-a26e-b36a29641b36-openshift-service-ca\") on node \"ip-10-0-138-89.ec2.internal\" DevicePath \"\""
Jul 08 12:31:22.403292 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:31:22.403255    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-49ca3975-7f499bf4fc-v58xl"]
Jul 08 12:31:22.406158 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:31:22.406130    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-49ca3975-7f499bf4fc-v58xl"]
Jul 08 12:31:22.716949 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:31:22.716899    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00cf80ef-9a09-44b5-a26e-b36a29641b36" path="/var/lib/kubelet/pods/00cf80ef-9a09-44b5-a26e-b36a29641b36/volumes"
Jul 08 12:31:25.028838 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:31:25.028766    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-49ca3975-7f499bf4fc-v58xl" podUID="00cf80ef-9a09-44b5-a26e-b36a29641b36" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.50:8443/health\": context deadline exceeded"
Jul 08 12:31:31.209614 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:31:31.209567    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:31:52.285990 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:31:52.285947    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cbca5e6f-695fb748df-68thg"]
Jul 08 12:31:52.286393 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:31:52.286368    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00cf80ef-9a09-44b5-a26e-b36a29641b36" containerName="maas-api"
Jul 08 12:31:52.286393 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:31:52.286379    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="00cf80ef-9a09-44b5-a26e-b36a29641b36" containerName="maas-api"
Jul 08 12:31:52.286463 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:31:52.286439    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="00cf80ef-9a09-44b5-a26e-b36a29641b36" containerName="maas-api"
Jul 08 12:31:52.289441 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:31:52.289419    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cbca5e6f-695fb748df-68thg"
Jul 08 12:31:52.292595 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:31:52.292571    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-cbca5e6f-serving-cert\""
Jul 08 12:31:52.302363 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:31:52.302331    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cbca5e6f-695fb748df-68thg"]
Jul 08 12:31:52.422344 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:31:52.422300    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f708b90-ef1f-40cc-be58-ba3bb2634375-openshift-service-ca\") pod \"maas-api-e2e-mt-cbca5e6f-695fb748df-68thg\" (UID: \"9f708b90-ef1f-40cc-be58-ba3bb2634375\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cbca5e6f-695fb748df-68thg"
Jul 08 12:31:52.422344 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:31:52.422342    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56ck8\" (UniqueName: \"kubernetes.io/projected/9f708b90-ef1f-40cc-be58-ba3bb2634375-kube-api-access-56ck8\") pod \"maas-api-e2e-mt-cbca5e6f-695fb748df-68thg\" (UID: \"9f708b90-ef1f-40cc-be58-ba3bb2634375\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cbca5e6f-695fb748df-68thg"
Jul 08 12:31:52.422615 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:31:52.422444    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f708b90-ef1f-40cc-be58-ba3bb2634375-maas-api-tls\") pod \"maas-api-e2e-mt-cbca5e6f-695fb748df-68thg\" (UID: \"9f708b90-ef1f-40cc-be58-ba3bb2634375\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cbca5e6f-695fb748df-68thg"
Jul 08 12:31:52.524024 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:31:52.523985    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f708b90-ef1f-40cc-be58-ba3bb2634375-maas-api-tls\") pod \"maas-api-e2e-mt-cbca5e6f-695fb748df-68thg\" (UID: \"9f708b90-ef1f-40cc-be58-ba3bb2634375\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cbca5e6f-695fb748df-68thg"
Jul 08 12:31:52.524216 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:31:52.524063    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f708b90-ef1f-40cc-be58-ba3bb2634375-openshift-service-ca\") pod \"maas-api-e2e-mt-cbca5e6f-695fb748df-68thg\" (UID: \"9f708b90-ef1f-40cc-be58-ba3bb2634375\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cbca5e6f-695fb748df-68thg"
Jul 08 12:31:52.524216 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:31:52.524080    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56ck8\" (UniqueName: \"kubernetes.io/projected/9f708b90-ef1f-40cc-be58-ba3bb2634375-kube-api-access-56ck8\") pod \"maas-api-e2e-mt-cbca5e6f-695fb748df-68thg\" (UID: \"9f708b90-ef1f-40cc-be58-ba3bb2634375\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cbca5e6f-695fb748df-68thg"
Jul 08 12:31:52.524801 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:31:52.524748    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f708b90-ef1f-40cc-be58-ba3bb2634375-openshift-service-ca\") pod \"maas-api-e2e-mt-cbca5e6f-695fb748df-68thg\" (UID: \"9f708b90-ef1f-40cc-be58-ba3bb2634375\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cbca5e6f-695fb748df-68thg"
Jul 08 12:31:52.526571 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:31:52.526546    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f708b90-ef1f-40cc-be58-ba3bb2634375-maas-api-tls\") pod \"maas-api-e2e-mt-cbca5e6f-695fb748df-68thg\" (UID: \"9f708b90-ef1f-40cc-be58-ba3bb2634375\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cbca5e6f-695fb748df-68thg"
Jul 08 12:31:52.533265 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:31:52.533235    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56ck8\" (UniqueName: \"kubernetes.io/projected/9f708b90-ef1f-40cc-be58-ba3bb2634375-kube-api-access-56ck8\") pod \"maas-api-e2e-mt-cbca5e6f-695fb748df-68thg\" (UID: \"9f708b90-ef1f-40cc-be58-ba3bb2634375\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cbca5e6f-695fb748df-68thg"
Jul 08 12:31:52.600965 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:31:52.600865    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cbca5e6f-695fb748df-68thg"
Jul 08 12:31:52.753412 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:31:52.753385    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cbca5e6f-695fb748df-68thg"]
Jul 08 12:31:52.755553 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:31:52.755525    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f708b90_ef1f_40cc_be58_ba3bb2634375.slice/crio-82667b2fb1ccf6857549eba8c3d74adae75872e3c69fc6701ec73de881cf1b52 WatchSource:0}: Error finding container 82667b2fb1ccf6857549eba8c3d74adae75872e3c69fc6701ec73de881cf1b52: Status 404 returned error can't find the container with id 82667b2fb1ccf6857549eba8c3d74adae75872e3c69fc6701ec73de881cf1b52
Jul 08 12:31:53.238595 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:31:53.238560    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cbca5e6f-695fb748df-68thg" event={"ID":"9f708b90-ef1f-40cc-be58-ba3bb2634375","Type":"ContainerStarted","Data":"82667b2fb1ccf6857549eba8c3d74adae75872e3c69fc6701ec73de881cf1b52"}
Jul 08 12:31:54.252016 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:31:54.251977    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cbca5e6f-695fb748df-68thg" event={"ID":"9f708b90-ef1f-40cc-be58-ba3bb2634375","Type":"ContainerStarted","Data":"2bda210aea323d34bfdd4c2d8b5f6dea4e62c162c93c42c27db5158c50ac4d89"}
Jul 08 12:31:54.252504 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:31:54.252074    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cbca5e6f-695fb748df-68thg"
Jul 08 12:31:54.271246 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:31:54.271194    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cbca5e6f-695fb748df-68thg" podStartSLOduration=1.5428695239999999 podStartE2EDuration="2.27117698s" totalImagesPullingTime="728.307456ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 12:31:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 12:31:54.268990752 +0000 UTC m=+1803.999365723" watchObservedRunningTime="2026-07-08 12:31:54.27117698 +0000 UTC m=+1804.001552008"
Jul 08 12:31:58.104138 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:31:58.104097    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:32:00.262946 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:32:00.262914    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cbca5e6f-695fb748df-68thg"
Jul 08 12:32:20.204256 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:32:20.204217    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:32:32.911896 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:32:32.911860    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:32:39.846584 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:32:39.846539    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cbca5e6f-695fb748df-68thg"]
Jul 08 12:32:39.847065 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:32:39.846830    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cbca5e6f-695fb748df-68thg" podUID="9f708b90-ef1f-40cc-be58-ba3bb2634375" containerName="maas-api" containerID="cri-o://2bda210aea323d34bfdd4c2d8b5f6dea4e62c162c93c42c27db5158c50ac4d89" gracePeriod=30
Jul 08 12:32:40.101640 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:32:40.101574    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cbca5e6f-695fb748df-68thg"
Jul 08 12:32:40.274234 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:32:40.274200    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f708b90-ef1f-40cc-be58-ba3bb2634375-openshift-service-ca\") pod \"9f708b90-ef1f-40cc-be58-ba3bb2634375\" (UID: \"9f708b90-ef1f-40cc-be58-ba3bb2634375\") "
Jul 08 12:32:40.274448 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:32:40.274258    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f708b90-ef1f-40cc-be58-ba3bb2634375-maas-api-tls\") pod \"9f708b90-ef1f-40cc-be58-ba3bb2634375\" (UID: \"9f708b90-ef1f-40cc-be58-ba3bb2634375\") "
Jul 08 12:32:40.274448 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:32:40.274321    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-56ck8\" (UniqueName: \"kubernetes.io/projected/9f708b90-ef1f-40cc-be58-ba3bb2634375-kube-api-access-56ck8\") pod \"9f708b90-ef1f-40cc-be58-ba3bb2634375\" (UID: \"9f708b90-ef1f-40cc-be58-ba3bb2634375\") "
Jul 08 12:32:40.274661 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:32:40.274633    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f708b90-ef1f-40cc-be58-ba3bb2634375-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9f708b90-ef1f-40cc-be58-ba3bb2634375" (UID: "9f708b90-ef1f-40cc-be58-ba3bb2634375"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 12:32:40.276710 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:32:40.276680    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f708b90-ef1f-40cc-be58-ba3bb2634375-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9f708b90-ef1f-40cc-be58-ba3bb2634375" (UID: "9f708b90-ef1f-40cc-be58-ba3bb2634375"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 12:32:40.276710 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:32:40.276682    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f708b90-ef1f-40cc-be58-ba3bb2634375-kube-api-access-56ck8" (OuterVolumeSpecName: "kube-api-access-56ck8") pod "9f708b90-ef1f-40cc-be58-ba3bb2634375" (UID: "9f708b90-ef1f-40cc-be58-ba3bb2634375"). InnerVolumeSpecName "kube-api-access-56ck8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 12:32:40.375947 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:32:40.375859    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f708b90-ef1f-40cc-be58-ba3bb2634375-openshift-service-ca\") on node \"ip-10-0-138-89.ec2.internal\" DevicePath \"\""
Jul 08 12:32:40.375947 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:32:40.375887    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f708b90-ef1f-40cc-be58-ba3bb2634375-maas-api-tls\") on node \"ip-10-0-138-89.ec2.internal\" DevicePath \"\""
Jul 08 12:32:40.375947 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:32:40.375898    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56ck8\" (UniqueName: \"kubernetes.io/projected/9f708b90-ef1f-40cc-be58-ba3bb2634375-kube-api-access-56ck8\") on node \"ip-10-0-138-89.ec2.internal\" DevicePath \"\""
Jul 08 12:32:40.489231 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:32:40.489190    2579 generic.go:356] "Generic (PLEG): container finished" podID="9f708b90-ef1f-40cc-be58-ba3bb2634375" containerID="2bda210aea323d34bfdd4c2d8b5f6dea4e62c162c93c42c27db5158c50ac4d89" exitCode=0
Jul 08 12:32:40.489406 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:32:40.489252    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cbca5e6f-695fb748df-68thg"
Jul 08 12:32:40.489406 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:32:40.489274    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cbca5e6f-695fb748df-68thg" event={"ID":"9f708b90-ef1f-40cc-be58-ba3bb2634375","Type":"ContainerDied","Data":"2bda210aea323d34bfdd4c2d8b5f6dea4e62c162c93c42c27db5158c50ac4d89"}
Jul 08 12:32:40.489406 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:32:40.489372    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cbca5e6f-695fb748df-68thg" event={"ID":"9f708b90-ef1f-40cc-be58-ba3bb2634375","Type":"ContainerDied","Data":"82667b2fb1ccf6857549eba8c3d74adae75872e3c69fc6701ec73de881cf1b52"}
Jul 08 12:32:40.489406 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:32:40.489401    2579 scope.go:117] "RemoveContainer" containerID="2bda210aea323d34bfdd4c2d8b5f6dea4e62c162c93c42c27db5158c50ac4d89"
Jul 08 12:32:40.499279 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:32:40.499261    2579 scope.go:117] "RemoveContainer" containerID="2bda210aea323d34bfdd4c2d8b5f6dea4e62c162c93c42c27db5158c50ac4d89"
Jul 08 12:32:40.499567 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:32:40.499549    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bda210aea323d34bfdd4c2d8b5f6dea4e62c162c93c42c27db5158c50ac4d89\": container with ID starting with 2bda210aea323d34bfdd4c2d8b5f6dea4e62c162c93c42c27db5158c50ac4d89 not found: ID does not exist" containerID="2bda210aea323d34bfdd4c2d8b5f6dea4e62c162c93c42c27db5158c50ac4d89"
Jul 08 12:32:40.499608 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:32:40.499577    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bda210aea323d34bfdd4c2d8b5f6dea4e62c162c93c42c27db5158c50ac4d89"} err="failed to get container status \"2bda210aea323d34bfdd4c2d8b5f6dea4e62c162c93c42c27db5158c50ac4d89\": rpc error: code = NotFound desc = could not find container \"2bda210aea323d34bfdd4c2d8b5f6dea4e62c162c93c42c27db5158c50ac4d89\": container with ID starting with 2bda210aea323d34bfdd4c2d8b5f6dea4e62c162c93c42c27db5158c50ac4d89 not found: ID does not exist"
Jul 08 12:32:40.520349 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:32:40.520315    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cbca5e6f-695fb748df-68thg"]
Jul 08 12:32:40.525314 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:32:40.525283    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cbca5e6f-695fb748df-68thg"]
Jul 08 12:32:40.717825 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:32:40.717791    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f708b90-ef1f-40cc-be58-ba3bb2634375" path="/var/lib/kubelet/pods/9f708b90-ef1f-40cc-be58-ba3bb2634375/volumes"
Jul 08 12:32:44.503838 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:32:44.503795    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:32:54.805202 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:32:54.805155    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:33:30.602861 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:33:30.602764    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:33:39.708155 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:33:39.708111    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:33:52.103659 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:33:52.103567    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:34:33.170080 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:34:33.170045    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-253a40-5b645dc899-8r2sw"]
Jul 08 12:34:33.170658 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:34:33.170522    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f708b90-ef1f-40cc-be58-ba3bb2634375" containerName="maas-api"
Jul 08 12:34:33.170658 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:34:33.170533    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f708b90-ef1f-40cc-be58-ba3bb2634375" containerName="maas-api"
Jul 08 12:34:33.170658 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:34:33.170590    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f708b90-ef1f-40cc-be58-ba3bb2634375" containerName="maas-api"
Jul 08 12:34:33.173680 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:34:33.173660    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-253a40-5b645dc899-8r2sw"
Jul 08 12:34:33.176645 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:34:33.176629    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-253a40-serving-cert\""
Jul 08 12:34:33.191536 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:34:33.191508    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-253a40-5b645dc899-8r2sw"]
Jul 08 12:34:33.270663 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:34:33.270631    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cckb6\" (UniqueName: \"kubernetes.io/projected/c5e6ad6b-1a77-4adb-840f-6da660406e6a-kube-api-access-cckb6\") pod \"maas-api-e2e-inf-a-253a40-5b645dc899-8r2sw\" (UID: \"c5e6ad6b-1a77-4adb-840f-6da660406e6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-253a40-5b645dc899-8r2sw"
Jul 08 12:34:33.270849 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:34:33.270670    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5e6ad6b-1a77-4adb-840f-6da660406e6a-maas-api-tls\") pod \"maas-api-e2e-inf-a-253a40-5b645dc899-8r2sw\" (UID: \"c5e6ad6b-1a77-4adb-840f-6da660406e6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-253a40-5b645dc899-8r2sw"
Jul 08 12:34:33.270849 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:34:33.270756    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5e6ad6b-1a77-4adb-840f-6da660406e6a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-253a40-5b645dc899-8r2sw\" (UID: \"c5e6ad6b-1a77-4adb-840f-6da660406e6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-253a40-5b645dc899-8r2sw"
Jul 08 12:34:33.372176 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:34:33.372145    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cckb6\" (UniqueName: \"kubernetes.io/projected/c5e6ad6b-1a77-4adb-840f-6da660406e6a-kube-api-access-cckb6\") pod \"maas-api-e2e-inf-a-253a40-5b645dc899-8r2sw\" (UID: \"c5e6ad6b-1a77-4adb-840f-6da660406e6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-253a40-5b645dc899-8r2sw"
Jul 08 12:34:33.372327 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:34:33.372184    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5e6ad6b-1a77-4adb-840f-6da660406e6a-maas-api-tls\") pod \"maas-api-e2e-inf-a-253a40-5b645dc899-8r2sw\" (UID: \"c5e6ad6b-1a77-4adb-840f-6da660406e6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-253a40-5b645dc899-8r2sw"
Jul 08 12:34:33.372327 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:34:33.372232    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5e6ad6b-1a77-4adb-840f-6da660406e6a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-253a40-5b645dc899-8r2sw\" (UID: \"c5e6ad6b-1a77-4adb-840f-6da660406e6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-253a40-5b645dc899-8r2sw"
Jul 08 12:34:33.372794 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:34:33.372753    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5e6ad6b-1a77-4adb-840f-6da660406e6a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-253a40-5b645dc899-8r2sw\" (UID: \"c5e6ad6b-1a77-4adb-840f-6da660406e6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-253a40-5b645dc899-8r2sw"
Jul 08 12:34:33.374747 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:34:33.374727    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5e6ad6b-1a77-4adb-840f-6da660406e6a-maas-api-tls\") pod \"maas-api-e2e-inf-a-253a40-5b645dc899-8r2sw\" (UID: \"c5e6ad6b-1a77-4adb-840f-6da660406e6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-253a40-5b645dc899-8r2sw"
Jul 08 12:34:33.383102 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:34:33.383073    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cckb6\" (UniqueName: \"kubernetes.io/projected/c5e6ad6b-1a77-4adb-840f-6da660406e6a-kube-api-access-cckb6\") pod \"maas-api-e2e-inf-a-253a40-5b645dc899-8r2sw\" (UID: \"c5e6ad6b-1a77-4adb-840f-6da660406e6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-253a40-5b645dc899-8r2sw"
Jul 08 12:34:33.483874 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:34:33.483824    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-253a40-5b645dc899-8r2sw"
Jul 08 12:34:33.621268 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:34:33.621236    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-253a40-5b645dc899-8r2sw"]
Jul 08 12:34:33.623466 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:34:33.623437    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5e6ad6b_1a77_4adb_840f_6da660406e6a.slice/crio-008da4637edb557031c6a6ed987e749434814e7215076d4797d6f75129487337 WatchSource:0}: Error finding container 008da4637edb557031c6a6ed987e749434814e7215076d4797d6f75129487337: Status 404 returned error can't find the container with id 008da4637edb557031c6a6ed987e749434814e7215076d4797d6f75129487337
Jul 08 12:34:33.624702 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:34:33.624687    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 08 12:34:34.048099 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:34:34.048059    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-253a40-5b645dc899-8r2sw" event={"ID":"c5e6ad6b-1a77-4adb-840f-6da660406e6a","Type":"ContainerStarted","Data":"008da4637edb557031c6a6ed987e749434814e7215076d4797d6f75129487337"}
Jul 08 12:34:35.053884 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:34:35.053844    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-253a40-5b645dc899-8r2sw" event={"ID":"c5e6ad6b-1a77-4adb-840f-6da660406e6a","Type":"ContainerStarted","Data":"c93a0c3d1578adba1faf19641d40b3342a15e75666b5a97f25abb00fe720b8b1"}
Jul 08 12:34:35.054253 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:34:35.053906    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-253a40-5b645dc899-8r2sw"
Jul 08 12:34:35.074513 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:34:35.074473    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-253a40-5b645dc899-8r2sw" podStartSLOduration=1.64272632 podStartE2EDuration="2.074456089s" totalImagesPullingTime="431.729769ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 12:34:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 12:34:35.073228606 +0000 UTC m=+1964.803603577" watchObservedRunningTime="2026-07-08 12:34:35.074456089 +0000 UTC m=+1964.804831058"
Jul 08 12:34:41.009252 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:34:41.009211    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:34:41.065494 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:34:41.065454    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-253a40-5b645dc899-8r2sw"
Jul 08 12:34:41.193941 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:34:41.193904    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-85707a-7fc9d594cc-ndx7g"]
Jul 08 12:34:41.197872 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:34:41.197844    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-85707a-7fc9d594cc-ndx7g"
Jul 08 12:34:41.200616 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:34:41.200592    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-85707a-serving-cert\""
Jul 08 12:34:41.211235 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:34:41.211209    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-85707a-7fc9d594cc-ndx7g"]
Jul 08 12:34:41.239476 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:34:41.239439    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h6j4\" (UniqueName: \"kubernetes.io/projected/a09196ca-ee0b-4c53-8a31-6f587f777058-kube-api-access-2h6j4\") pod \"maas-api-e2e-inf-b-85707a-7fc9d594cc-ndx7g\" (UID: \"a09196ca-ee0b-4c53-8a31-6f587f777058\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-85707a-7fc9d594cc-ndx7g"
Jul 08 12:34:41.239476 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:34:41.239481    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a09196ca-ee0b-4c53-8a31-6f587f777058-maas-api-tls\") pod \"maas-api-e2e-inf-b-85707a-7fc9d594cc-ndx7g\" (UID: \"a09196ca-ee0b-4c53-8a31-6f587f777058\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-85707a-7fc9d594cc-ndx7g"
Jul 08 12:34:41.239683 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:34:41.239554    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a09196ca-ee0b-4c53-8a31-6f587f777058-openshift-service-ca\") pod \"maas-api-e2e-inf-b-85707a-7fc9d594cc-ndx7g\" (UID: \"a09196ca-ee0b-4c53-8a31-6f587f777058\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-85707a-7fc9d594cc-ndx7g"
Jul 08 12:34:41.340676 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:34:41.340546    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h6j4\" (UniqueName: \"kubernetes.io/projected/a09196ca-ee0b-4c53-8a31-6f587f777058-kube-api-access-2h6j4\") pod \"maas-api-e2e-inf-b-85707a-7fc9d594cc-ndx7g\" (UID: \"a09196ca-ee0b-4c53-8a31-6f587f777058\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-85707a-7fc9d594cc-ndx7g"
Jul 08 12:34:41.340676 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:34:41.340597    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a09196ca-ee0b-4c53-8a31-6f587f777058-maas-api-tls\") pod \"maas-api-e2e-inf-b-85707a-7fc9d594cc-ndx7g\" (UID: \"a09196ca-ee0b-4c53-8a31-6f587f777058\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-85707a-7fc9d594cc-ndx7g"
Jul 08 12:34:41.340676 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:34:41.340658    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a09196ca-ee0b-4c53-8a31-6f587f777058-openshift-service-ca\") pod \"maas-api-e2e-inf-b-85707a-7fc9d594cc-ndx7g\" (UID: \"a09196ca-ee0b-4c53-8a31-6f587f777058\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-85707a-7fc9d594cc-ndx7g"
Jul 08 12:34:41.341018 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:34:41.340829    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-85707a-serving-cert: secret "maas-api-e2e-inf-b-85707a-serving-cert" not found
Jul 08 12:34:41.341018 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:34:41.340895    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a09196ca-ee0b-4c53-8a31-6f587f777058-maas-api-tls podName:a09196ca-ee0b-4c53-8a31-6f587f777058 nodeName:}" failed. No retries permitted until 2026-07-08 12:34:41.840875443 +0000 UTC m=+1971.571250392 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a09196ca-ee0b-4c53-8a31-6f587f777058-maas-api-tls") pod "maas-api-e2e-inf-b-85707a-7fc9d594cc-ndx7g" (UID: "a09196ca-ee0b-4c53-8a31-6f587f777058") : secret "maas-api-e2e-inf-b-85707a-serving-cert" not found
Jul 08 12:34:41.341422 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:34:41.341398    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a09196ca-ee0b-4c53-8a31-6f587f777058-openshift-service-ca\") pod \"maas-api-e2e-inf-b-85707a-7fc9d594cc-ndx7g\" (UID: \"a09196ca-ee0b-4c53-8a31-6f587f777058\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-85707a-7fc9d594cc-ndx7g"
Jul 08 12:34:41.350288 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:34:41.350267    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h6j4\" (UniqueName: \"kubernetes.io/projected/a09196ca-ee0b-4c53-8a31-6f587f777058-kube-api-access-2h6j4\") pod \"maas-api-e2e-inf-b-85707a-7fc9d594cc-ndx7g\" (UID: \"a09196ca-ee0b-4c53-8a31-6f587f777058\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-85707a-7fc9d594cc-ndx7g"
Jul 08 12:34:41.845250 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:34:41.845218    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a09196ca-ee0b-4c53-8a31-6f587f777058-maas-api-tls\") pod \"maas-api-e2e-inf-b-85707a-7fc9d594cc-ndx7g\" (UID: \"a09196ca-ee0b-4c53-8a31-6f587f777058\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-85707a-7fc9d594cc-ndx7g"
Jul 08 12:34:41.847741 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:34:41.847711    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a09196ca-ee0b-4c53-8a31-6f587f777058-maas-api-tls\") pod \"maas-api-e2e-inf-b-85707a-7fc9d594cc-ndx7g\" (UID: \"a09196ca-ee0b-4c53-8a31-6f587f777058\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-85707a-7fc9d594cc-ndx7g"
Jul 08 12:34:42.111094 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:34:42.111005    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-85707a-7fc9d594cc-ndx7g"
Jul 08 12:34:42.254673 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:34:42.254648    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-85707a-7fc9d594cc-ndx7g"]
Jul 08 12:34:42.257198 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:34:42.257162    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda09196ca_ee0b_4c53_8a31_6f587f777058.slice/crio-ccbad6095512d4ae15abcd507dcd6c1277db75d7f637b9f645c9605c19687375 WatchSource:0}: Error finding container ccbad6095512d4ae15abcd507dcd6c1277db75d7f637b9f645c9605c19687375: Status 404 returned error can't find the container with id ccbad6095512d4ae15abcd507dcd6c1277db75d7f637b9f645c9605c19687375
Jul 08 12:34:43.099978 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:34:43.099880    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-85707a-7fc9d594cc-ndx7g" event={"ID":"a09196ca-ee0b-4c53-8a31-6f587f777058","Type":"ContainerStarted","Data":"9c1dce000c69363256199ec459c63aec90fa85ffd9fed5b8d910ea818b09f3c3"}
Jul 08 12:34:43.099978 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:34:43.099947    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-85707a-7fc9d594cc-ndx7g" event={"ID":"a09196ca-ee0b-4c53-8a31-6f587f777058","Type":"ContainerStarted","Data":"ccbad6095512d4ae15abcd507dcd6c1277db75d7f637b9f645c9605c19687375"}
Jul 08 12:34:43.100185 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:34:43.100109    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-85707a-7fc9d594cc-ndx7g"
Jul 08 12:34:43.118256 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:34:43.118197    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-85707a-7fc9d594cc-ndx7g" podStartSLOduration=1.548550266 podStartE2EDuration="2.118170628s" totalImagesPullingTime="569.620362ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 12:34:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 12:34:43.117328933 +0000 UTC m=+1972.847703914" watchObservedRunningTime="2026-07-08 12:34:43.118170628 +0000 UTC m=+1972.848545590"
Jul 08 12:34:49.111099 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:34:49.111069    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-85707a-7fc9d594cc-ndx7g"
Jul 08 12:34:55.420638 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:34:55.420597    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:35:05.313020 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:35:05.312984    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:35:32.412916 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:35:32.412874    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:35:48.111541 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:35:48.111496    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:36:17.266235 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:36:17.266022    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-253a40-5b645dc899-8r2sw"]
Jul 08 12:36:17.266758 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:36:17.266292    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-253a40-5b645dc899-8r2sw" podUID="c5e6ad6b-1a77-4adb-840f-6da660406e6a" containerName="maas-api" containerID="cri-o://c93a0c3d1578adba1faf19641d40b3342a15e75666b5a97f25abb00fe720b8b1" gracePeriod=30
Jul 08 12:36:17.528240 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:36:17.528172    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-253a40-5b645dc899-8r2sw"
Jul 08 12:36:17.565883 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:36:17.565837    2579 generic.go:356] "Generic (PLEG): container finished" podID="c5e6ad6b-1a77-4adb-840f-6da660406e6a" containerID="c93a0c3d1578adba1faf19641d40b3342a15e75666b5a97f25abb00fe720b8b1" exitCode=0
Jul 08 12:36:17.565883 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:36:17.565882    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-253a40-5b645dc899-8r2sw"
Jul 08 12:36:17.566135 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:36:17.565919    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-253a40-5b645dc899-8r2sw" event={"ID":"c5e6ad6b-1a77-4adb-840f-6da660406e6a","Type":"ContainerDied","Data":"c93a0c3d1578adba1faf19641d40b3342a15e75666b5a97f25abb00fe720b8b1"}
Jul 08 12:36:17.566135 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:36:17.565972    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-253a40-5b645dc899-8r2sw" event={"ID":"c5e6ad6b-1a77-4adb-840f-6da660406e6a","Type":"ContainerDied","Data":"008da4637edb557031c6a6ed987e749434814e7215076d4797d6f75129487337"}
Jul 08 12:36:17.566135 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:36:17.565997    2579 scope.go:117] "RemoveContainer" containerID="c93a0c3d1578adba1faf19641d40b3342a15e75666b5a97f25abb00fe720b8b1"
Jul 08 12:36:17.575299 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:36:17.575278    2579 scope.go:117] "RemoveContainer" containerID="c93a0c3d1578adba1faf19641d40b3342a15e75666b5a97f25abb00fe720b8b1"
Jul 08 12:36:17.575582 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:36:17.575565    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c93a0c3d1578adba1faf19641d40b3342a15e75666b5a97f25abb00fe720b8b1\": container with ID starting with c93a0c3d1578adba1faf19641d40b3342a15e75666b5a97f25abb00fe720b8b1 not found: ID does not exist" containerID="c93a0c3d1578adba1faf19641d40b3342a15e75666b5a97f25abb00fe720b8b1"
Jul 08 12:36:17.575643 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:36:17.575590    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c93a0c3d1578adba1faf19641d40b3342a15e75666b5a97f25abb00fe720b8b1"} err="failed to get container status \"c93a0c3d1578adba1faf19641d40b3342a15e75666b5a97f25abb00fe720b8b1\": rpc error: code = NotFound desc = could not find container \"c93a0c3d1578adba1faf19641d40b3342a15e75666b5a97f25abb00fe720b8b1\": container with ID starting with c93a0c3d1578adba1faf19641d40b3342a15e75666b5a97f25abb00fe720b8b1 not found: ID does not exist"
Jul 08 12:36:17.625679 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:36:17.625644    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cckb6\" (UniqueName: \"kubernetes.io/projected/c5e6ad6b-1a77-4adb-840f-6da660406e6a-kube-api-access-cckb6\") pod \"c5e6ad6b-1a77-4adb-840f-6da660406e6a\" (UID: \"c5e6ad6b-1a77-4adb-840f-6da660406e6a\") "
Jul 08 12:36:17.625679 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:36:17.625686    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5e6ad6b-1a77-4adb-840f-6da660406e6a-openshift-service-ca\") pod \"c5e6ad6b-1a77-4adb-840f-6da660406e6a\" (UID: \"c5e6ad6b-1a77-4adb-840f-6da660406e6a\") "
Jul 08 12:36:17.625966 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:36:17.625763    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5e6ad6b-1a77-4adb-840f-6da660406e6a-maas-api-tls\") pod \"c5e6ad6b-1a77-4adb-840f-6da660406e6a\" (UID: \"c5e6ad6b-1a77-4adb-840f-6da660406e6a\") "
Jul 08 12:36:17.626087 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:36:17.626065    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5e6ad6b-1a77-4adb-840f-6da660406e6a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c5e6ad6b-1a77-4adb-840f-6da660406e6a" (UID: "c5e6ad6b-1a77-4adb-840f-6da660406e6a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 12:36:17.628041 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:36:17.628009    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5e6ad6b-1a77-4adb-840f-6da660406e6a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c5e6ad6b-1a77-4adb-840f-6da660406e6a" (UID: "c5e6ad6b-1a77-4adb-840f-6da660406e6a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 12:36:17.628146 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:36:17.628060    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5e6ad6b-1a77-4adb-840f-6da660406e6a-kube-api-access-cckb6" (OuterVolumeSpecName: "kube-api-access-cckb6") pod "c5e6ad6b-1a77-4adb-840f-6da660406e6a" (UID: "c5e6ad6b-1a77-4adb-840f-6da660406e6a"). InnerVolumeSpecName "kube-api-access-cckb6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 12:36:17.727567 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:36:17.727491    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5e6ad6b-1a77-4adb-840f-6da660406e6a-maas-api-tls\") on node \"ip-10-0-138-89.ec2.internal\" DevicePath \"\""
Jul 08 12:36:17.727567 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:36:17.727528    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cckb6\" (UniqueName: \"kubernetes.io/projected/c5e6ad6b-1a77-4adb-840f-6da660406e6a-kube-api-access-cckb6\") on node \"ip-10-0-138-89.ec2.internal\" DevicePath \"\""
Jul 08 12:36:17.727567 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:36:17.727543    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5e6ad6b-1a77-4adb-840f-6da660406e6a-openshift-service-ca\") on node \"ip-10-0-138-89.ec2.internal\" DevicePath \"\""
Jul 08 12:36:17.890256 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:36:17.890224    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-253a40-5b645dc899-8r2sw"]
Jul 08 12:36:17.894592 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:36:17.894562    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-253a40-5b645dc899-8r2sw"]
Jul 08 12:36:18.717141 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:36:18.717110    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5e6ad6b-1a77-4adb-840f-6da660406e6a" path="/var/lib/kubelet/pods/c5e6ad6b-1a77-4adb-840f-6da660406e6a/volumes"
Jul 08 12:36:20.698498 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:36:20.698459    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:37:01.170559 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:01.170524    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-85707a-7fc9d594cc-ndx7g"]
Jul 08 12:37:01.171087 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:01.170857    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-85707a-7fc9d594cc-ndx7g" podUID="a09196ca-ee0b-4c53-8a31-6f587f777058" containerName="maas-api" containerID="cri-o://9c1dce000c69363256199ec459c63aec90fa85ffd9fed5b8d910ea818b09f3c3" gracePeriod=30
Jul 08 12:37:01.431831 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:01.431754    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-85707a-7fc9d594cc-ndx7g"
Jul 08 12:37:01.569570 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:01.569528    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a09196ca-ee0b-4c53-8a31-6f587f777058-openshift-service-ca\") pod \"a09196ca-ee0b-4c53-8a31-6f587f777058\" (UID: \"a09196ca-ee0b-4c53-8a31-6f587f777058\") "
Jul 08 12:37:01.569570 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:01.569580    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a09196ca-ee0b-4c53-8a31-6f587f777058-maas-api-tls\") pod \"a09196ca-ee0b-4c53-8a31-6f587f777058\" (UID: \"a09196ca-ee0b-4c53-8a31-6f587f777058\") "
Jul 08 12:37:01.569881 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:01.569627    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2h6j4\" (UniqueName: \"kubernetes.io/projected/a09196ca-ee0b-4c53-8a31-6f587f777058-kube-api-access-2h6j4\") pod \"a09196ca-ee0b-4c53-8a31-6f587f777058\" (UID: \"a09196ca-ee0b-4c53-8a31-6f587f777058\") "
Jul 08 12:37:01.569976 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:01.569946    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a09196ca-ee0b-4c53-8a31-6f587f777058-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a09196ca-ee0b-4c53-8a31-6f587f777058" (UID: "a09196ca-ee0b-4c53-8a31-6f587f777058"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 12:37:01.571847 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:01.571823    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a09196ca-ee0b-4c53-8a31-6f587f777058-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a09196ca-ee0b-4c53-8a31-6f587f777058" (UID: "a09196ca-ee0b-4c53-8a31-6f587f777058"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 12:37:01.571949 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:01.571911    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a09196ca-ee0b-4c53-8a31-6f587f777058-kube-api-access-2h6j4" (OuterVolumeSpecName: "kube-api-access-2h6j4") pod "a09196ca-ee0b-4c53-8a31-6f587f777058" (UID: "a09196ca-ee0b-4c53-8a31-6f587f777058"). InnerVolumeSpecName "kube-api-access-2h6j4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 12:37:01.670862 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:01.670820    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h6j4\" (UniqueName: \"kubernetes.io/projected/a09196ca-ee0b-4c53-8a31-6f587f777058-kube-api-access-2h6j4\") on node \"ip-10-0-138-89.ec2.internal\" DevicePath \"\""
Jul 08 12:37:01.670862 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:01.670856    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a09196ca-ee0b-4c53-8a31-6f587f777058-openshift-service-ca\") on node \"ip-10-0-138-89.ec2.internal\" DevicePath \"\""
Jul 08 12:37:01.670862 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:01.670866    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a09196ca-ee0b-4c53-8a31-6f587f777058-maas-api-tls\") on node \"ip-10-0-138-89.ec2.internal\" DevicePath \"\""
Jul 08 12:37:01.792605 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:01.792514    2579 generic.go:356] "Generic (PLEG): container finished" podID="a09196ca-ee0b-4c53-8a31-6f587f777058" containerID="9c1dce000c69363256199ec459c63aec90fa85ffd9fed5b8d910ea818b09f3c3" exitCode=0
Jul 08 12:37:01.792605 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:01.792552    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-85707a-7fc9d594cc-ndx7g" event={"ID":"a09196ca-ee0b-4c53-8a31-6f587f777058","Type":"ContainerDied","Data":"9c1dce000c69363256199ec459c63aec90fa85ffd9fed5b8d910ea818b09f3c3"}
Jul 08 12:37:01.792605 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:01.792558    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-85707a-7fc9d594cc-ndx7g"
Jul 08 12:37:01.792605 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:01.792589    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-85707a-7fc9d594cc-ndx7g" event={"ID":"a09196ca-ee0b-4c53-8a31-6f587f777058","Type":"ContainerDied","Data":"ccbad6095512d4ae15abcd507dcd6c1277db75d7f637b9f645c9605c19687375"}
Jul 08 12:37:01.792963 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:01.792611    2579 scope.go:117] "RemoveContainer" containerID="9c1dce000c69363256199ec459c63aec90fa85ffd9fed5b8d910ea818b09f3c3"
Jul 08 12:37:01.801764 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:01.801743    2579 scope.go:117] "RemoveContainer" containerID="9c1dce000c69363256199ec459c63aec90fa85ffd9fed5b8d910ea818b09f3c3"
Jul 08 12:37:01.802059 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:37:01.802040    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c1dce000c69363256199ec459c63aec90fa85ffd9fed5b8d910ea818b09f3c3\": container with ID starting with 9c1dce000c69363256199ec459c63aec90fa85ffd9fed5b8d910ea818b09f3c3 not found: ID does not exist" containerID="9c1dce000c69363256199ec459c63aec90fa85ffd9fed5b8d910ea818b09f3c3"
Jul 08 12:37:01.802161 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:01.802070    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c1dce000c69363256199ec459c63aec90fa85ffd9fed5b8d910ea818b09f3c3"} err="failed to get container status \"9c1dce000c69363256199ec459c63aec90fa85ffd9fed5b8d910ea818b09f3c3\": rpc error: code = NotFound desc = could not find container \"9c1dce000c69363256199ec459c63aec90fa85ffd9fed5b8d910ea818b09f3c3\": container with ID starting with 9c1dce000c69363256199ec459c63aec90fa85ffd9fed5b8d910ea818b09f3c3 not found: ID does not exist"
Jul 08 12:37:01.814228 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:01.814195    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-85707a-7fc9d594cc-ndx7g"]
Jul 08 12:37:01.820942 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:01.820916    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-85707a-7fc9d594cc-ndx7g"]
Jul 08 12:37:02.717292 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:02.717257    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a09196ca-ee0b-4c53-8a31-6f587f777058" path="/var/lib/kubelet/pods/a09196ca-ee0b-4c53-8a31-6f587f777058/volumes"
Jul 08 12:37:04.405628 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:04.405590    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:37:10.904430 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:10.904394    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c1cfd9-6545b75565-tl4mf"]
Jul 08 12:37:10.904913 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:10.904889    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a09196ca-ee0b-4c53-8a31-6f587f777058" containerName="maas-api"
Jul 08 12:37:10.904913 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:10.904913    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a09196ca-ee0b-4c53-8a31-6f587f777058" containerName="maas-api"
Jul 08 12:37:10.905117 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:10.904939    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5e6ad6b-1a77-4adb-840f-6da660406e6a" containerName="maas-api"
Jul 08 12:37:10.905117 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:10.904948    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5e6ad6b-1a77-4adb-840f-6da660406e6a" containerName="maas-api"
Jul 08 12:37:10.905117 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:10.905078    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5e6ad6b-1a77-4adb-840f-6da660406e6a" containerName="maas-api"
Jul 08 12:37:10.905117 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:10.905094    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a09196ca-ee0b-4c53-8a31-6f587f777058" containerName="maas-api"
Jul 08 12:37:10.909576 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:10.909553    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c1cfd9-6545b75565-tl4mf"
Jul 08 12:37:10.912236 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:10.912214    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-c1cfd9-serving-cert\""
Jul 08 12:37:10.919443 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:10.919418    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c1cfd9-6545b75565-tl4mf"]
Jul 08 12:37:11.055094 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:11.055050    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jmxn\" (UniqueName: \"kubernetes.io/projected/e82ffe41-a266-4115-8692-1000c501a335-kube-api-access-9jmxn\") pod \"maas-api-e2e-api-a-c1cfd9-6545b75565-tl4mf\" (UID: \"e82ffe41-a266-4115-8692-1000c501a335\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c1cfd9-6545b75565-tl4mf"
Jul 08 12:37:11.055282 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:11.055107    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e82ffe41-a266-4115-8692-1000c501a335-maas-api-tls\") pod \"maas-api-e2e-api-a-c1cfd9-6545b75565-tl4mf\" (UID: \"e82ffe41-a266-4115-8692-1000c501a335\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c1cfd9-6545b75565-tl4mf"
Jul 08 12:37:11.055282 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:11.055135    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e82ffe41-a266-4115-8692-1000c501a335-openshift-service-ca\") pod \"maas-api-e2e-api-a-c1cfd9-6545b75565-tl4mf\" (UID: \"e82ffe41-a266-4115-8692-1000c501a335\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c1cfd9-6545b75565-tl4mf"
Jul 08 12:37:11.155706 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:11.155607    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jmxn\" (UniqueName: \"kubernetes.io/projected/e82ffe41-a266-4115-8692-1000c501a335-kube-api-access-9jmxn\") pod \"maas-api-e2e-api-a-c1cfd9-6545b75565-tl4mf\" (UID: \"e82ffe41-a266-4115-8692-1000c501a335\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c1cfd9-6545b75565-tl4mf"
Jul 08 12:37:11.155706 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:11.155658    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e82ffe41-a266-4115-8692-1000c501a335-maas-api-tls\") pod \"maas-api-e2e-api-a-c1cfd9-6545b75565-tl4mf\" (UID: \"e82ffe41-a266-4115-8692-1000c501a335\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c1cfd9-6545b75565-tl4mf"
Jul 08 12:37:11.155940 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:11.155812    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e82ffe41-a266-4115-8692-1000c501a335-openshift-service-ca\") pod \"maas-api-e2e-api-a-c1cfd9-6545b75565-tl4mf\" (UID: \"e82ffe41-a266-4115-8692-1000c501a335\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c1cfd9-6545b75565-tl4mf"
Jul 08 12:37:11.156435 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:11.156409    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e82ffe41-a266-4115-8692-1000c501a335-openshift-service-ca\") pod \"maas-api-e2e-api-a-c1cfd9-6545b75565-tl4mf\" (UID: \"e82ffe41-a266-4115-8692-1000c501a335\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c1cfd9-6545b75565-tl4mf"
Jul 08 12:37:11.158231 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:11.158209    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e82ffe41-a266-4115-8692-1000c501a335-maas-api-tls\") pod \"maas-api-e2e-api-a-c1cfd9-6545b75565-tl4mf\" (UID: \"e82ffe41-a266-4115-8692-1000c501a335\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c1cfd9-6545b75565-tl4mf"
Jul 08 12:37:11.164928 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:11.164899    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jmxn\" (UniqueName: \"kubernetes.io/projected/e82ffe41-a266-4115-8692-1000c501a335-kube-api-access-9jmxn\") pod \"maas-api-e2e-api-a-c1cfd9-6545b75565-tl4mf\" (UID: \"e82ffe41-a266-4115-8692-1000c501a335\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c1cfd9-6545b75565-tl4mf"
Jul 08 12:37:11.220859 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:11.220814    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c1cfd9-6545b75565-tl4mf"
Jul 08 12:37:11.374444 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:11.374404    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c1cfd9-6545b75565-tl4mf"]
Jul 08 12:37:11.379664 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:37:11.379625    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode82ffe41_a266_4115_8692_1000c501a335.slice/crio-560532002e5d982a3266647e0c89981eb527d899c8362a79f8265350cc2aedf6 WatchSource:0}: Error finding container 560532002e5d982a3266647e0c89981eb527d899c8362a79f8265350cc2aedf6: Status 404 returned error can't find the container with id 560532002e5d982a3266647e0c89981eb527d899c8362a79f8265350cc2aedf6
Jul 08 12:37:11.850237 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:11.850201    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c1cfd9-6545b75565-tl4mf" event={"ID":"e82ffe41-a266-4115-8692-1000c501a335","Type":"ContainerStarted","Data":"560532002e5d982a3266647e0c89981eb527d899c8362a79f8265350cc2aedf6"}
Jul 08 12:37:12.856261 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:12.856219    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c1cfd9-6545b75565-tl4mf" event={"ID":"e82ffe41-a266-4115-8692-1000c501a335","Type":"ContainerStarted","Data":"725d4a31aa4e66309d4606d7966fd9161babe9356872cf0ad14656cd4e737625"}
Jul 08 12:37:12.856716 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:12.856287    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c1cfd9-6545b75565-tl4mf"
Jul 08 12:37:12.886520 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:12.886469    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c1cfd9-6545b75565-tl4mf" podStartSLOduration=2.435110006 podStartE2EDuration="2.886451731s" totalImagesPullingTime="451.341725ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 12:37:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 12:37:12.885383215 +0000 UTC m=+2122.615758199" watchObservedRunningTime="2026-07-08 12:37:12.886451731 +0000 UTC m=+2122.616826702"
Jul 08 12:37:18.866728 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:18.866694    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c1cfd9-6545b75565-tl4mf"
Jul 08 12:37:19.118072 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:19.117981    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f4d1bb-77dd6d57b9-7fc7m"]
Jul 08 12:37:19.126564 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:19.126531    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f4d1bb-77dd6d57b9-7fc7m"
Jul 08 12:37:19.128985 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:19.128956    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-f4d1bb-serving-cert\""
Jul 08 12:37:19.135728 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:19.135698    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f4d1bb-77dd6d57b9-7fc7m"]
Jul 08 12:37:19.209647 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:19.209605    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:37:19.231480 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:19.231442    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca184373-588b-4845-b303-379df16916d5-maas-api-tls\") pod \"maas-api-e2e-api-b-f4d1bb-77dd6d57b9-7fc7m\" (UID: \"ca184373-588b-4845-b303-379df16916d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f4d1bb-77dd6d57b9-7fc7m"
Jul 08 12:37:19.231480 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:19.231477    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd4ht\" (UniqueName: \"kubernetes.io/projected/ca184373-588b-4845-b303-379df16916d5-kube-api-access-vd4ht\") pod \"maas-api-e2e-api-b-f4d1bb-77dd6d57b9-7fc7m\" (UID: \"ca184373-588b-4845-b303-379df16916d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f4d1bb-77dd6d57b9-7fc7m"
Jul 08 12:37:19.231730 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:19.231598    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca184373-588b-4845-b303-379df16916d5-openshift-service-ca\") pod \"maas-api-e2e-api-b-f4d1bb-77dd6d57b9-7fc7m\" (UID: \"ca184373-588b-4845-b303-379df16916d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f4d1bb-77dd6d57b9-7fc7m"
Jul 08 12:37:19.332606 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:19.332562    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca184373-588b-4845-b303-379df16916d5-maas-api-tls\") pod \"maas-api-e2e-api-b-f4d1bb-77dd6d57b9-7fc7m\" (UID: \"ca184373-588b-4845-b303-379df16916d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f4d1bb-77dd6d57b9-7fc7m"
Jul 08 12:37:19.332606 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:19.332603    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd4ht\" (UniqueName: \"kubernetes.io/projected/ca184373-588b-4845-b303-379df16916d5-kube-api-access-vd4ht\") pod \"maas-api-e2e-api-b-f4d1bb-77dd6d57b9-7fc7m\" (UID: \"ca184373-588b-4845-b303-379df16916d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f4d1bb-77dd6d57b9-7fc7m"
Jul 08 12:37:19.332937 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:19.332751    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca184373-588b-4845-b303-379df16916d5-openshift-service-ca\") pod \"maas-api-e2e-api-b-f4d1bb-77dd6d57b9-7fc7m\" (UID: \"ca184373-588b-4845-b303-379df16916d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f4d1bb-77dd6d57b9-7fc7m"
Jul 08 12:37:19.333413 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:19.333390    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca184373-588b-4845-b303-379df16916d5-openshift-service-ca\") pod \"maas-api-e2e-api-b-f4d1bb-77dd6d57b9-7fc7m\" (UID: \"ca184373-588b-4845-b303-379df16916d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f4d1bb-77dd6d57b9-7fc7m"
Jul 08 12:37:19.335217 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:19.335194    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca184373-588b-4845-b303-379df16916d5-maas-api-tls\") pod \"maas-api-e2e-api-b-f4d1bb-77dd6d57b9-7fc7m\" (UID: \"ca184373-588b-4845-b303-379df16916d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f4d1bb-77dd6d57b9-7fc7m"
Jul 08 12:37:19.344114 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:19.344083    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd4ht\" (UniqueName: \"kubernetes.io/projected/ca184373-588b-4845-b303-379df16916d5-kube-api-access-vd4ht\") pod \"maas-api-e2e-api-b-f4d1bb-77dd6d57b9-7fc7m\" (UID: \"ca184373-588b-4845-b303-379df16916d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f4d1bb-77dd6d57b9-7fc7m"
Jul 08 12:37:19.438342 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:19.438311    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f4d1bb-77dd6d57b9-7fc7m"
Jul 08 12:37:19.589247 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:19.589220    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f4d1bb-77dd6d57b9-7fc7m"]
Jul 08 12:37:19.591704 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:37:19.591678    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca184373_588b_4845_b303_379df16916d5.slice/crio-0e6847e44b79399b0a8b8986b603429f9c46060a7d8b79b38f0525392fbe6991 WatchSource:0}: Error finding container 0e6847e44b79399b0a8b8986b603429f9c46060a7d8b79b38f0525392fbe6991: Status 404 returned error can't find the container with id 0e6847e44b79399b0a8b8986b603429f9c46060a7d8b79b38f0525392fbe6991
Jul 08 12:37:19.895005 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:19.894914    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f4d1bb-77dd6d57b9-7fc7m" event={"ID":"ca184373-588b-4845-b303-379df16916d5","Type":"ContainerStarted","Data":"0e6847e44b79399b0a8b8986b603429f9c46060a7d8b79b38f0525392fbe6991"}
Jul 08 12:37:20.901795 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:20.901727    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f4d1bb-77dd6d57b9-7fc7m" event={"ID":"ca184373-588b-4845-b303-379df16916d5","Type":"ContainerStarted","Data":"0ead775d9e6e0f885647bf8644988a9fa70ad1c853b6f38287ff76ae6ac080eb"}
Jul 08 12:37:20.902205 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:20.901907    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f4d1bb-77dd6d57b9-7fc7m"
Jul 08 12:37:20.924314 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:20.924254    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f4d1bb-77dd6d57b9-7fc7m" podStartSLOduration=1.5694932050000001 podStartE2EDuration="1.924237768s" totalImagesPullingTime="354.744563ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 12:37:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 12:37:20.922244101 +0000 UTC m=+2130.652619099" watchObservedRunningTime="2026-07-08 12:37:20.924237768 +0000 UTC m=+2130.654612756"
Jul 08 12:37:26.916668 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:26.916638    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f4d1bb-77dd6d57b9-7fc7m"
Jul 08 12:37:32.383029 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:32.382990    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c1cfd9-6545b75565-tl4mf"]
Jul 08 12:37:32.383616 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:32.383244    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c1cfd9-6545b75565-tl4mf" podUID="e82ffe41-a266-4115-8692-1000c501a335" containerName="maas-api" containerID="cri-o://725d4a31aa4e66309d4606d7966fd9161babe9356872cf0ad14656cd4e737625" gracePeriod=30
Jul 08 12:37:32.652131 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:32.652099    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c1cfd9-6545b75565-tl4mf"
Jul 08 12:37:32.753056 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:32.753018    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e82ffe41-a266-4115-8692-1000c501a335-maas-api-tls\") pod \"e82ffe41-a266-4115-8692-1000c501a335\" (UID: \"e82ffe41-a266-4115-8692-1000c501a335\") "
Jul 08 12:37:32.753254 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:32.753101    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9jmxn\" (UniqueName: \"kubernetes.io/projected/e82ffe41-a266-4115-8692-1000c501a335-kube-api-access-9jmxn\") pod \"e82ffe41-a266-4115-8692-1000c501a335\" (UID: \"e82ffe41-a266-4115-8692-1000c501a335\") "
Jul 08 12:37:32.753254 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:32.753139    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e82ffe41-a266-4115-8692-1000c501a335-openshift-service-ca\") pod \"e82ffe41-a266-4115-8692-1000c501a335\" (UID: \"e82ffe41-a266-4115-8692-1000c501a335\") "
Jul 08 12:37:32.753562 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:32.753536    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e82ffe41-a266-4115-8692-1000c501a335-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e82ffe41-a266-4115-8692-1000c501a335" (UID: "e82ffe41-a266-4115-8692-1000c501a335"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 12:37:32.755314 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:32.755279    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e82ffe41-a266-4115-8692-1000c501a335-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e82ffe41-a266-4115-8692-1000c501a335" (UID: "e82ffe41-a266-4115-8692-1000c501a335"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 12:37:32.755314 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:32.755296    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e82ffe41-a266-4115-8692-1000c501a335-kube-api-access-9jmxn" (OuterVolumeSpecName: "kube-api-access-9jmxn") pod "e82ffe41-a266-4115-8692-1000c501a335" (UID: "e82ffe41-a266-4115-8692-1000c501a335"). InnerVolumeSpecName "kube-api-access-9jmxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 12:37:32.854213 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:32.854171    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e82ffe41-a266-4115-8692-1000c501a335-maas-api-tls\") on node \"ip-10-0-138-89.ec2.internal\" DevicePath \"\""
Jul 08 12:37:32.854213 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:32.854206    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jmxn\" (UniqueName: \"kubernetes.io/projected/e82ffe41-a266-4115-8692-1000c501a335-kube-api-access-9jmxn\") on node \"ip-10-0-138-89.ec2.internal\" DevicePath \"\""
Jul 08 12:37:32.854213 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:32.854218    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e82ffe41-a266-4115-8692-1000c501a335-openshift-service-ca\") on node \"ip-10-0-138-89.ec2.internal\" DevicePath \"\""
Jul 08 12:37:32.962671 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:32.962639    2579 generic.go:356] "Generic (PLEG): container finished" podID="e82ffe41-a266-4115-8692-1000c501a335" containerID="725d4a31aa4e66309d4606d7966fd9161babe9356872cf0ad14656cd4e737625" exitCode=0
Jul 08 12:37:32.964700 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:32.962685    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c1cfd9-6545b75565-tl4mf"
Jul 08 12:37:32.964700 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:32.962726    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c1cfd9-6545b75565-tl4mf" event={"ID":"e82ffe41-a266-4115-8692-1000c501a335","Type":"ContainerDied","Data":"725d4a31aa4e66309d4606d7966fd9161babe9356872cf0ad14656cd4e737625"}
Jul 08 12:37:32.964700 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:32.962806    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c1cfd9-6545b75565-tl4mf" event={"ID":"e82ffe41-a266-4115-8692-1000c501a335","Type":"ContainerDied","Data":"560532002e5d982a3266647e0c89981eb527d899c8362a79f8265350cc2aedf6"}
Jul 08 12:37:32.964700 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:32.962831    2579 scope.go:117] "RemoveContainer" containerID="725d4a31aa4e66309d4606d7966fd9161babe9356872cf0ad14656cd4e737625"
Jul 08 12:37:32.972832 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:32.972812    2579 scope.go:117] "RemoveContainer" containerID="725d4a31aa4e66309d4606d7966fd9161babe9356872cf0ad14656cd4e737625"
Jul 08 12:37:32.973137 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:37:32.973115    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"725d4a31aa4e66309d4606d7966fd9161babe9356872cf0ad14656cd4e737625\": container with ID starting with 725d4a31aa4e66309d4606d7966fd9161babe9356872cf0ad14656cd4e737625 not found: ID does not exist" containerID="725d4a31aa4e66309d4606d7966fd9161babe9356872cf0ad14656cd4e737625"
Jul 08 12:37:32.973206 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:32.973145    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"725d4a31aa4e66309d4606d7966fd9161babe9356872cf0ad14656cd4e737625"} err="failed to get container status \"725d4a31aa4e66309d4606d7966fd9161babe9356872cf0ad14656cd4e737625\": rpc error: code = NotFound desc = could not find container \"725d4a31aa4e66309d4606d7966fd9161babe9356872cf0ad14656cd4e737625\": container with ID starting with 725d4a31aa4e66309d4606d7966fd9161babe9356872cf0ad14656cd4e737625 not found: ID does not exist"
Jul 08 12:37:32.986680 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:32.986648    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c1cfd9-6545b75565-tl4mf"]
Jul 08 12:37:32.988468 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:32.988449    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c1cfd9-6545b75565-tl4mf"]
Jul 08 12:37:34.717120 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:34.717090    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e82ffe41-a266-4115-8692-1000c501a335" path="/var/lib/kubelet/pods/e82ffe41-a266-4115-8692-1000c501a335/volumes"
Jul 08 12:37:35.109474 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:35.109388    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:37:39.552932 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:39.552896    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f4d1bb-77dd6d57b9-7fc7m"]
Jul 08 12:37:39.553300 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:39.553213    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f4d1bb-77dd6d57b9-7fc7m" podUID="ca184373-588b-4845-b303-379df16916d5" containerName="maas-api" containerID="cri-o://0ead775d9e6e0f885647bf8644988a9fa70ad1c853b6f38287ff76ae6ac080eb" gracePeriod=30
Jul 08 12:37:39.807413 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:39.807338    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f4d1bb-77dd6d57b9-7fc7m"
Jul 08 12:37:39.921509 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:39.921467    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vd4ht\" (UniqueName: \"kubernetes.io/projected/ca184373-588b-4845-b303-379df16916d5-kube-api-access-vd4ht\") pod \"ca184373-588b-4845-b303-379df16916d5\" (UID: \"ca184373-588b-4845-b303-379df16916d5\") "
Jul 08 12:37:39.921509 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:39.921517    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca184373-588b-4845-b303-379df16916d5-openshift-service-ca\") pod \"ca184373-588b-4845-b303-379df16916d5\" (UID: \"ca184373-588b-4845-b303-379df16916d5\") "
Jul 08 12:37:39.921813 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:39.921635    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca184373-588b-4845-b303-379df16916d5-maas-api-tls\") pod \"ca184373-588b-4845-b303-379df16916d5\" (UID: \"ca184373-588b-4845-b303-379df16916d5\") "
Jul 08 12:37:39.921975 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:39.921941    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca184373-588b-4845-b303-379df16916d5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ca184373-588b-4845-b303-379df16916d5" (UID: "ca184373-588b-4845-b303-379df16916d5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 12:37:39.923963 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:39.923926    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca184373-588b-4845-b303-379df16916d5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ca184373-588b-4845-b303-379df16916d5" (UID: "ca184373-588b-4845-b303-379df16916d5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 12:37:39.924067 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:39.923962    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca184373-588b-4845-b303-379df16916d5-kube-api-access-vd4ht" (OuterVolumeSpecName: "kube-api-access-vd4ht") pod "ca184373-588b-4845-b303-379df16916d5" (UID: "ca184373-588b-4845-b303-379df16916d5"). InnerVolumeSpecName "kube-api-access-vd4ht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 12:37:39.999258 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:39.999221    2579 generic.go:356] "Generic (PLEG): container finished" podID="ca184373-588b-4845-b303-379df16916d5" containerID="0ead775d9e6e0f885647bf8644988a9fa70ad1c853b6f38287ff76ae6ac080eb" exitCode=0
Jul 08 12:37:39.999258 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:39.999261    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f4d1bb-77dd6d57b9-7fc7m"
Jul 08 12:37:39.999489 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:39.999293    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f4d1bb-77dd6d57b9-7fc7m" event={"ID":"ca184373-588b-4845-b303-379df16916d5","Type":"ContainerDied","Data":"0ead775d9e6e0f885647bf8644988a9fa70ad1c853b6f38287ff76ae6ac080eb"}
Jul 08 12:37:39.999489 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:39.999366    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f4d1bb-77dd6d57b9-7fc7m" event={"ID":"ca184373-588b-4845-b303-379df16916d5","Type":"ContainerDied","Data":"0e6847e44b79399b0a8b8986b603429f9c46060a7d8b79b38f0525392fbe6991"}
Jul 08 12:37:39.999489 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:39.999389    2579 scope.go:117] "RemoveContainer" containerID="0ead775d9e6e0f885647bf8644988a9fa70ad1c853b6f38287ff76ae6ac080eb"
Jul 08 12:37:40.009235 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:40.009215    2579 scope.go:117] "RemoveContainer" containerID="0ead775d9e6e0f885647bf8644988a9fa70ad1c853b6f38287ff76ae6ac080eb"
Jul 08 12:37:40.009534 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:37:40.009512    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ead775d9e6e0f885647bf8644988a9fa70ad1c853b6f38287ff76ae6ac080eb\": container with ID starting with 0ead775d9e6e0f885647bf8644988a9fa70ad1c853b6f38287ff76ae6ac080eb not found: ID does not exist" containerID="0ead775d9e6e0f885647bf8644988a9fa70ad1c853b6f38287ff76ae6ac080eb"
Jul 08 12:37:40.009629 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:40.009542    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ead775d9e6e0f885647bf8644988a9fa70ad1c853b6f38287ff76ae6ac080eb"} err="failed to get container status \"0ead775d9e6e0f885647bf8644988a9fa70ad1c853b6f38287ff76ae6ac080eb\": rpc error: code = NotFound desc = could not find container \"0ead775d9e6e0f885647bf8644988a9fa70ad1c853b6f38287ff76ae6ac080eb\": container with ID starting with 0ead775d9e6e0f885647bf8644988a9fa70ad1c853b6f38287ff76ae6ac080eb not found: ID does not exist"
Jul 08 12:37:40.023016 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:40.022987    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca184373-588b-4845-b303-379df16916d5-maas-api-tls\") on node \"ip-10-0-138-89.ec2.internal\" DevicePath \"\""
Jul 08 12:37:40.023016 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:40.023012    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vd4ht\" (UniqueName: \"kubernetes.io/projected/ca184373-588b-4845-b303-379df16916d5-kube-api-access-vd4ht\") on node \"ip-10-0-138-89.ec2.internal\" DevicePath \"\""
Jul 08 12:37:40.023016 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:40.023024    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca184373-588b-4845-b303-379df16916d5-openshift-service-ca\") on node \"ip-10-0-138-89.ec2.internal\" DevicePath \"\""
Jul 08 12:37:40.023670 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:40.023645    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f4d1bb-77dd6d57b9-7fc7m"]
Jul 08 12:37:40.027898 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:40.027876    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f4d1bb-77dd6d57b9-7fc7m"]
Jul 08 12:37:40.716798 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:40.716736    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca184373-588b-4845-b303-379df16916d5" path="/var/lib/kubelet/pods/ca184373-588b-4845-b303-379df16916d5/volumes"
Jul 08 12:37:50.133222 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:50.133128    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-b4806b-6667476f88-qsfnn"]
Jul 08 12:37:50.133833 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:50.133795    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca184373-588b-4845-b303-379df16916d5" containerName="maas-api"
Jul 08 12:37:50.133833 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:50.133821    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca184373-588b-4845-b303-379df16916d5" containerName="maas-api"
Jul 08 12:37:50.134049 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:50.133846    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e82ffe41-a266-4115-8692-1000c501a335" containerName="maas-api"
Jul 08 12:37:50.134049 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:50.133854    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e82ffe41-a266-4115-8692-1000c501a335" containerName="maas-api"
Jul 08 12:37:50.134049 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:50.133957    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca184373-588b-4845-b303-379df16916d5" containerName="maas-api"
Jul 08 12:37:50.134049 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:50.133974    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e82ffe41-a266-4115-8692-1000c501a335" containerName="maas-api"
Jul 08 12:37:50.138818 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:50.138799    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b4806b-6667476f88-qsfnn"
Jul 08 12:37:50.141344 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:50.141322    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-b4806b-serving-cert\""
Jul 08 12:37:50.148059 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:50.148035    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-b4806b-6667476f88-qsfnn"]
Jul 08 12:37:50.217975 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:50.217936    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8dd8c9aa-b41d-495c-a6eb-9ec2ecf0cf42-maas-api-tls\") pod \"maas-api-e2e-shared-a-b4806b-6667476f88-qsfnn\" (UID: \"8dd8c9aa-b41d-495c-a6eb-9ec2ecf0cf42\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b4806b-6667476f88-qsfnn"
Jul 08 12:37:50.217975 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:50.217979    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwkpx\" (UniqueName: \"kubernetes.io/projected/8dd8c9aa-b41d-495c-a6eb-9ec2ecf0cf42-kube-api-access-xwkpx\") pod \"maas-api-e2e-shared-a-b4806b-6667476f88-qsfnn\" (UID: \"8dd8c9aa-b41d-495c-a6eb-9ec2ecf0cf42\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b4806b-6667476f88-qsfnn"
Jul 08 12:37:50.218196 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:50.218069    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dd8c9aa-b41d-495c-a6eb-9ec2ecf0cf42-openshift-service-ca\") pod \"maas-api-e2e-shared-a-b4806b-6667476f88-qsfnn\" (UID: \"8dd8c9aa-b41d-495c-a6eb-9ec2ecf0cf42\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b4806b-6667476f88-qsfnn"
Jul 08 12:37:50.319188 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:50.319155    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8dd8c9aa-b41d-495c-a6eb-9ec2ecf0cf42-maas-api-tls\") pod \"maas-api-e2e-shared-a-b4806b-6667476f88-qsfnn\" (UID: \"8dd8c9aa-b41d-495c-a6eb-9ec2ecf0cf42\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b4806b-6667476f88-qsfnn"
Jul 08 12:37:50.319188 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:50.319189    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwkpx\" (UniqueName: \"kubernetes.io/projected/8dd8c9aa-b41d-495c-a6eb-9ec2ecf0cf42-kube-api-access-xwkpx\") pod \"maas-api-e2e-shared-a-b4806b-6667476f88-qsfnn\" (UID: \"8dd8c9aa-b41d-495c-a6eb-9ec2ecf0cf42\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b4806b-6667476f88-qsfnn"
Jul 08 12:37:50.319445 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:50.319247    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dd8c9aa-b41d-495c-a6eb-9ec2ecf0cf42-openshift-service-ca\") pod \"maas-api-e2e-shared-a-b4806b-6667476f88-qsfnn\" (UID: \"8dd8c9aa-b41d-495c-a6eb-9ec2ecf0cf42\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b4806b-6667476f88-qsfnn"
Jul 08 12:37:50.319445 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:37:50.319316    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-b4806b-serving-cert: secret "maas-api-e2e-shared-a-b4806b-serving-cert" not found
Jul 08 12:37:50.319445 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:37:50.319382    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8dd8c9aa-b41d-495c-a6eb-9ec2ecf0cf42-maas-api-tls podName:8dd8c9aa-b41d-495c-a6eb-9ec2ecf0cf42 nodeName:}" failed. No retries permitted until 2026-07-08 12:37:50.819365209 +0000 UTC m=+2160.549740158 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8dd8c9aa-b41d-495c-a6eb-9ec2ecf0cf42-maas-api-tls") pod "maas-api-e2e-shared-a-b4806b-6667476f88-qsfnn" (UID: "8dd8c9aa-b41d-495c-a6eb-9ec2ecf0cf42") : secret "maas-api-e2e-shared-a-b4806b-serving-cert" not found
Jul 08 12:37:50.319817 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:50.319796    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dd8c9aa-b41d-495c-a6eb-9ec2ecf0cf42-openshift-service-ca\") pod \"maas-api-e2e-shared-a-b4806b-6667476f88-qsfnn\" (UID: \"8dd8c9aa-b41d-495c-a6eb-9ec2ecf0cf42\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b4806b-6667476f88-qsfnn"
Jul 08 12:37:50.330817 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:50.330759    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwkpx\" (UniqueName: \"kubernetes.io/projected/8dd8c9aa-b41d-495c-a6eb-9ec2ecf0cf42-kube-api-access-xwkpx\") pod \"maas-api-e2e-shared-a-b4806b-6667476f88-qsfnn\" (UID: \"8dd8c9aa-b41d-495c-a6eb-9ec2ecf0cf42\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b4806b-6667476f88-qsfnn"
Jul 08 12:37:50.606681 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:50.606645    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:37:50.824667 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:50.824629    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8dd8c9aa-b41d-495c-a6eb-9ec2ecf0cf42-maas-api-tls\") pod \"maas-api-e2e-shared-a-b4806b-6667476f88-qsfnn\" (UID: \"8dd8c9aa-b41d-495c-a6eb-9ec2ecf0cf42\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b4806b-6667476f88-qsfnn"
Jul 08 12:37:50.827196 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:50.827176    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8dd8c9aa-b41d-495c-a6eb-9ec2ecf0cf42-maas-api-tls\") pod \"maas-api-e2e-shared-a-b4806b-6667476f88-qsfnn\" (UID: \"8dd8c9aa-b41d-495c-a6eb-9ec2ecf0cf42\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b4806b-6667476f88-qsfnn"
Jul 08 12:37:51.050534 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:51.050497    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b4806b-6667476f88-qsfnn"
Jul 08 12:37:51.195539 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:51.195501    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-b4806b-6667476f88-qsfnn"]
Jul 08 12:37:51.199071 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:37:51.199045    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dd8c9aa_b41d_495c_a6eb_9ec2ecf0cf42.slice/crio-dd10fb725f41aed0a44d62f9ea63fd1bfbe14390d0887881ba3ee235f3242d13 WatchSource:0}: Error finding container dd10fb725f41aed0a44d62f9ea63fd1bfbe14390d0887881ba3ee235f3242d13: Status 404 returned error can't find the container with id dd10fb725f41aed0a44d62f9ea63fd1bfbe14390d0887881ba3ee235f3242d13
Jul 08 12:37:52.062055 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:52.062016    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b4806b-6667476f88-qsfnn" event={"ID":"8dd8c9aa-b41d-495c-a6eb-9ec2ecf0cf42","Type":"ContainerStarted","Data":"a86a2cdb8b779c01dd8574bbaeb38c0fc34e4ffbb4f016884f23d891ec86ada7"}
Jul 08 12:37:52.062229 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:52.062074    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b4806b-6667476f88-qsfnn" event={"ID":"8dd8c9aa-b41d-495c-a6eb-9ec2ecf0cf42","Type":"ContainerStarted","Data":"dd10fb725f41aed0a44d62f9ea63fd1bfbe14390d0887881ba3ee235f3242d13"}
Jul 08 12:37:52.062229 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:52.062156    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b4806b-6667476f88-qsfnn"
Jul 08 12:37:52.082394 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:52.082325    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b4806b-6667476f88-qsfnn" podStartSLOduration=1.683510258 podStartE2EDuration="2.082309342s" totalImagesPullingTime="398.799084ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 12:37:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 12:37:52.080242843 +0000 UTC m=+2161.810617815" watchObservedRunningTime="2026-07-08 12:37:52.082309342 +0000 UTC m=+2161.812684319"
Jul 08 12:37:58.072665 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:58.072633    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b4806b-6667476f88-qsfnn"
Jul 08 12:37:58.172787 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:58.172747    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-58b9d8-7746cc64cf-xr9d9"]
Jul 08 12:37:58.175474 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:58.175453    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-58b9d8-7746cc64cf-xr9d9"
Jul 08 12:37:58.177984 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:58.177955    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-58b9d8-serving-cert\""
Jul 08 12:37:58.185602 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:58.185572    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-58b9d8-7746cc64cf-xr9d9"]
Jul 08 12:37:58.298536 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:58.298499    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69c4ca84-7359-4228-abab-1a5bc2cc4123-maas-api-tls\") pod \"maas-api-e2e-shared-b-58b9d8-7746cc64cf-xr9d9\" (UID: \"69c4ca84-7359-4228-abab-1a5bc2cc4123\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-58b9d8-7746cc64cf-xr9d9"
Jul 08 12:37:58.298751 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:58.298556    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69c4ca84-7359-4228-abab-1a5bc2cc4123-openshift-service-ca\") pod \"maas-api-e2e-shared-b-58b9d8-7746cc64cf-xr9d9\" (UID: \"69c4ca84-7359-4228-abab-1a5bc2cc4123\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-58b9d8-7746cc64cf-xr9d9"
Jul 08 12:37:58.298751 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:58.298677    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbqxs\" (UniqueName: \"kubernetes.io/projected/69c4ca84-7359-4228-abab-1a5bc2cc4123-kube-api-access-xbqxs\") pod \"maas-api-e2e-shared-b-58b9d8-7746cc64cf-xr9d9\" (UID: \"69c4ca84-7359-4228-abab-1a5bc2cc4123\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-58b9d8-7746cc64cf-xr9d9"
Jul 08 12:37:58.399226 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:58.399198    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69c4ca84-7359-4228-abab-1a5bc2cc4123-openshift-service-ca\") pod \"maas-api-e2e-shared-b-58b9d8-7746cc64cf-xr9d9\" (UID: \"69c4ca84-7359-4228-abab-1a5bc2cc4123\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-58b9d8-7746cc64cf-xr9d9"
Jul 08 12:37:58.399325 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:58.399263    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbqxs\" (UniqueName: \"kubernetes.io/projected/69c4ca84-7359-4228-abab-1a5bc2cc4123-kube-api-access-xbqxs\") pod \"maas-api-e2e-shared-b-58b9d8-7746cc64cf-xr9d9\" (UID: \"69c4ca84-7359-4228-abab-1a5bc2cc4123\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-58b9d8-7746cc64cf-xr9d9"
Jul 08 12:37:58.399386 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:58.399322    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69c4ca84-7359-4228-abab-1a5bc2cc4123-maas-api-tls\") pod \"maas-api-e2e-shared-b-58b9d8-7746cc64cf-xr9d9\" (UID: \"69c4ca84-7359-4228-abab-1a5bc2cc4123\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-58b9d8-7746cc64cf-xr9d9"
Jul 08 12:37:58.399995 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:58.399973    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69c4ca84-7359-4228-abab-1a5bc2cc4123-openshift-service-ca\") pod \"maas-api-e2e-shared-b-58b9d8-7746cc64cf-xr9d9\" (UID: \"69c4ca84-7359-4228-abab-1a5bc2cc4123\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-58b9d8-7746cc64cf-xr9d9"
Jul 08 12:37:58.401890 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:58.401861    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69c4ca84-7359-4228-abab-1a5bc2cc4123-maas-api-tls\") pod \"maas-api-e2e-shared-b-58b9d8-7746cc64cf-xr9d9\" (UID: \"69c4ca84-7359-4228-abab-1a5bc2cc4123\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-58b9d8-7746cc64cf-xr9d9"
Jul 08 12:37:58.408706 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:58.408679    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbqxs\" (UniqueName: \"kubernetes.io/projected/69c4ca84-7359-4228-abab-1a5bc2cc4123-kube-api-access-xbqxs\") pod \"maas-api-e2e-shared-b-58b9d8-7746cc64cf-xr9d9\" (UID: \"69c4ca84-7359-4228-abab-1a5bc2cc4123\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-58b9d8-7746cc64cf-xr9d9"
Jul 08 12:37:58.486248 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:58.486210    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-58b9d8-7746cc64cf-xr9d9"
Jul 08 12:37:58.633307 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:58.633253    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-58b9d8-7746cc64cf-xr9d9"]
Jul 08 12:37:58.635720 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:37:58.635690    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69c4ca84_7359_4228_abab_1a5bc2cc4123.slice/crio-e4af8f1ea542339fe7f7aba36c518dd4726bdb3d80b3cd294df9b7f33eeca79d WatchSource:0}: Error finding container e4af8f1ea542339fe7f7aba36c518dd4726bdb3d80b3cd294df9b7f33eeca79d: Status 404 returned error can't find the container with id e4af8f1ea542339fe7f7aba36c518dd4726bdb3d80b3cd294df9b7f33eeca79d
Jul 08 12:37:59.099450 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:37:59.099419    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-58b9d8-7746cc64cf-xr9d9" event={"ID":"69c4ca84-7359-4228-abab-1a5bc2cc4123","Type":"ContainerStarted","Data":"e4af8f1ea542339fe7f7aba36c518dd4726bdb3d80b3cd294df9b7f33eeca79d"}
Jul 08 12:38:00.105557 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:38:00.105525    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-58b9d8-7746cc64cf-xr9d9" event={"ID":"69c4ca84-7359-4228-abab-1a5bc2cc4123","Type":"ContainerStarted","Data":"3fc1af947f48d47ff7127fbc889aa600555a45ff0067c4e5a3f5eab393ee4d14"}
Jul 08 12:38:00.106001 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:38:00.105667    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-58b9d8-7746cc64cf-xr9d9"
Jul 08 12:38:00.125522 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:38:00.125466    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-58b9d8-7746cc64cf-xr9d9" podStartSLOduration=1.575375471 podStartE2EDuration="2.125447553s" totalImagesPullingTime="550.072082ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 12:37:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 12:38:00.123708084 +0000 UTC m=+2169.854083055" watchObservedRunningTime="2026-07-08 12:38:00.125447553 +0000 UTC m=+2169.855822524"
Jul 08 12:38:05.479389 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:38:05.479346    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:38:06.116317 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:38:06.116289    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-58b9d8-7746cc64cf-xr9d9"
Jul 08 12:41:08.319735 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:08.319699    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-b4806b-6667476f88-qsfnn"]
Jul 08 12:41:08.320376 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:08.319952    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b4806b-6667476f88-qsfnn" podUID="8dd8c9aa-b41d-495c-a6eb-9ec2ecf0cf42" containerName="maas-api" containerID="cri-o://a86a2cdb8b779c01dd8574bbaeb38c0fc34e4ffbb4f016884f23d891ec86ada7" gracePeriod=30
Jul 08 12:41:08.574141 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:08.574071    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b4806b-6667476f88-qsfnn"
Jul 08 12:41:08.668800 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:08.668739    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dd8c9aa-b41d-495c-a6eb-9ec2ecf0cf42-openshift-service-ca\") pod \"8dd8c9aa-b41d-495c-a6eb-9ec2ecf0cf42\" (UID: \"8dd8c9aa-b41d-495c-a6eb-9ec2ecf0cf42\") "
Jul 08 12:41:08.668996 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:08.668864    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xwkpx\" (UniqueName: \"kubernetes.io/projected/8dd8c9aa-b41d-495c-a6eb-9ec2ecf0cf42-kube-api-access-xwkpx\") pod \"8dd8c9aa-b41d-495c-a6eb-9ec2ecf0cf42\" (UID: \"8dd8c9aa-b41d-495c-a6eb-9ec2ecf0cf42\") "
Jul 08 12:41:08.668996 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:08.668943    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8dd8c9aa-b41d-495c-a6eb-9ec2ecf0cf42-maas-api-tls\") pod \"8dd8c9aa-b41d-495c-a6eb-9ec2ecf0cf42\" (UID: \"8dd8c9aa-b41d-495c-a6eb-9ec2ecf0cf42\") "
Jul 08 12:41:08.669130 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:08.669104    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dd8c9aa-b41d-495c-a6eb-9ec2ecf0cf42-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8dd8c9aa-b41d-495c-a6eb-9ec2ecf0cf42" (UID: "8dd8c9aa-b41d-495c-a6eb-9ec2ecf0cf42"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 12:41:08.669232 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:08.669218    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dd8c9aa-b41d-495c-a6eb-9ec2ecf0cf42-openshift-service-ca\") on node \"ip-10-0-138-89.ec2.internal\" DevicePath \"\""
Jul 08 12:41:08.671172 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:08.671143    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8dd8c9aa-b41d-495c-a6eb-9ec2ecf0cf42-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8dd8c9aa-b41d-495c-a6eb-9ec2ecf0cf42" (UID: "8dd8c9aa-b41d-495c-a6eb-9ec2ecf0cf42"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 12:41:08.671294 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:08.671192    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dd8c9aa-b41d-495c-a6eb-9ec2ecf0cf42-kube-api-access-xwkpx" (OuterVolumeSpecName: "kube-api-access-xwkpx") pod "8dd8c9aa-b41d-495c-a6eb-9ec2ecf0cf42" (UID: "8dd8c9aa-b41d-495c-a6eb-9ec2ecf0cf42"). InnerVolumeSpecName "kube-api-access-xwkpx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 12:41:08.770572 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:08.770536    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8dd8c9aa-b41d-495c-a6eb-9ec2ecf0cf42-maas-api-tls\") on node \"ip-10-0-138-89.ec2.internal\" DevicePath \"\""
Jul 08 12:41:08.770572 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:08.770566    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwkpx\" (UniqueName: \"kubernetes.io/projected/8dd8c9aa-b41d-495c-a6eb-9ec2ecf0cf42-kube-api-access-xwkpx\") on node \"ip-10-0-138-89.ec2.internal\" DevicePath \"\""
Jul 08 12:41:09.034879 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:09.034843    2579 generic.go:356] "Generic (PLEG): container finished" podID="8dd8c9aa-b41d-495c-a6eb-9ec2ecf0cf42" containerID="a86a2cdb8b779c01dd8574bbaeb38c0fc34e4ffbb4f016884f23d891ec86ada7" exitCode=0
Jul 08 12:41:09.035071 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:09.034894    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b4806b-6667476f88-qsfnn"
Jul 08 12:41:09.035071 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:09.034886    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b4806b-6667476f88-qsfnn" event={"ID":"8dd8c9aa-b41d-495c-a6eb-9ec2ecf0cf42","Type":"ContainerDied","Data":"a86a2cdb8b779c01dd8574bbaeb38c0fc34e4ffbb4f016884f23d891ec86ada7"}
Jul 08 12:41:09.035071 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:09.034947    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b4806b-6667476f88-qsfnn" event={"ID":"8dd8c9aa-b41d-495c-a6eb-9ec2ecf0cf42","Type":"ContainerDied","Data":"dd10fb725f41aed0a44d62f9ea63fd1bfbe14390d0887881ba3ee235f3242d13"}
Jul 08 12:41:09.035071 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:09.034959    2579 scope.go:117] "RemoveContainer" containerID="a86a2cdb8b779c01dd8574bbaeb38c0fc34e4ffbb4f016884f23d891ec86ada7"
Jul 08 12:41:09.044101 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:09.044078    2579 scope.go:117] "RemoveContainer" containerID="a86a2cdb8b779c01dd8574bbaeb38c0fc34e4ffbb4f016884f23d891ec86ada7"
Jul 08 12:41:09.044360 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:41:09.044343    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a86a2cdb8b779c01dd8574bbaeb38c0fc34e4ffbb4f016884f23d891ec86ada7\": container with ID starting with a86a2cdb8b779c01dd8574bbaeb38c0fc34e4ffbb4f016884f23d891ec86ada7 not found: ID does not exist" containerID="a86a2cdb8b779c01dd8574bbaeb38c0fc34e4ffbb4f016884f23d891ec86ada7"
Jul 08 12:41:09.044410 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:09.044369    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a86a2cdb8b779c01dd8574bbaeb38c0fc34e4ffbb4f016884f23d891ec86ada7"} err="failed to get container status \"a86a2cdb8b779c01dd8574bbaeb38c0fc34e4ffbb4f016884f23d891ec86ada7\": rpc error: code = NotFound desc = could not find container \"a86a2cdb8b779c01dd8574bbaeb38c0fc34e4ffbb4f016884f23d891ec86ada7\": container with ID starting with a86a2cdb8b779c01dd8574bbaeb38c0fc34e4ffbb4f016884f23d891ec86ada7 not found: ID does not exist"
Jul 08 12:41:09.052798 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:09.052751    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-b4806b-6667476f88-qsfnn"]
Jul 08 12:41:09.059018 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:09.058992    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-b4806b-6667476f88-qsfnn"]
Jul 08 12:41:10.717087 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:10.717052    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8dd8c9aa-b41d-495c-a6eb-9ec2ecf0cf42" path="/var/lib/kubelet/pods/8dd8c9aa-b41d-495c-a6eb-9ec2ecf0cf42/volumes"
Jul 08 12:41:11.414926 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:11.414891    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:41:15.977457 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:15.977418    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-58b9d8-7746cc64cf-xr9d9"]
Jul 08 12:41:15.977927 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:15.977703    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-58b9d8-7746cc64cf-xr9d9" podUID="69c4ca84-7359-4228-abab-1a5bc2cc4123" containerName="maas-api" containerID="cri-o://3fc1af947f48d47ff7127fbc889aa600555a45ff0067c4e5a3f5eab393ee4d14" gracePeriod=30
Jul 08 12:41:16.111850 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:16.111800    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-58b9d8-7746cc64cf-xr9d9" podUID="69c4ca84-7359-4228-abab-1a5bc2cc4123" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.57:8443/health\": dial tcp 10.133.0.57:8443: connect: connection refused"
Jul 08 12:41:16.234631 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:16.234573    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-58b9d8-7746cc64cf-xr9d9"
Jul 08 12:41:16.341296 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:16.341255    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69c4ca84-7359-4228-abab-1a5bc2cc4123-openshift-service-ca\") pod \"69c4ca84-7359-4228-abab-1a5bc2cc4123\" (UID: \"69c4ca84-7359-4228-abab-1a5bc2cc4123\") "
Jul 08 12:41:16.341495 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:16.341308    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xbqxs\" (UniqueName: \"kubernetes.io/projected/69c4ca84-7359-4228-abab-1a5bc2cc4123-kube-api-access-xbqxs\") pod \"69c4ca84-7359-4228-abab-1a5bc2cc4123\" (UID: \"69c4ca84-7359-4228-abab-1a5bc2cc4123\") "
Jul 08 12:41:16.341495 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:16.341339    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69c4ca84-7359-4228-abab-1a5bc2cc4123-maas-api-tls\") pod \"69c4ca84-7359-4228-abab-1a5bc2cc4123\" (UID: \"69c4ca84-7359-4228-abab-1a5bc2cc4123\") "
Jul 08 12:41:16.341629 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:16.341603    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69c4ca84-7359-4228-abab-1a5bc2cc4123-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "69c4ca84-7359-4228-abab-1a5bc2cc4123" (UID: "69c4ca84-7359-4228-abab-1a5bc2cc4123"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 12:41:16.343535 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:16.343504    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69c4ca84-7359-4228-abab-1a5bc2cc4123-kube-api-access-xbqxs" (OuterVolumeSpecName: "kube-api-access-xbqxs") pod "69c4ca84-7359-4228-abab-1a5bc2cc4123" (UID: "69c4ca84-7359-4228-abab-1a5bc2cc4123"). InnerVolumeSpecName "kube-api-access-xbqxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 12:41:16.343660 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:16.343608    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69c4ca84-7359-4228-abab-1a5bc2cc4123-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "69c4ca84-7359-4228-abab-1a5bc2cc4123" (UID: "69c4ca84-7359-4228-abab-1a5bc2cc4123"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 12:41:16.442693 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:16.442660    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69c4ca84-7359-4228-abab-1a5bc2cc4123-openshift-service-ca\") on node \"ip-10-0-138-89.ec2.internal\" DevicePath \"\""
Jul 08 12:41:16.442693 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:16.442687    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbqxs\" (UniqueName: \"kubernetes.io/projected/69c4ca84-7359-4228-abab-1a5bc2cc4123-kube-api-access-xbqxs\") on node \"ip-10-0-138-89.ec2.internal\" DevicePath \"\""
Jul 08 12:41:16.442693 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:16.442699    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69c4ca84-7359-4228-abab-1a5bc2cc4123-maas-api-tls\") on node \"ip-10-0-138-89.ec2.internal\" DevicePath \"\""
Jul 08 12:41:17.074347 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:17.074263    2579 generic.go:356] "Generic (PLEG): container finished" podID="69c4ca84-7359-4228-abab-1a5bc2cc4123" containerID="3fc1af947f48d47ff7127fbc889aa600555a45ff0067c4e5a3f5eab393ee4d14" exitCode=0
Jul 08 12:41:17.074347 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:17.074302    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-58b9d8-7746cc64cf-xr9d9" event={"ID":"69c4ca84-7359-4228-abab-1a5bc2cc4123","Type":"ContainerDied","Data":"3fc1af947f48d47ff7127fbc889aa600555a45ff0067c4e5a3f5eab393ee4d14"}
Jul 08 12:41:17.074347 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:17.074307    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-58b9d8-7746cc64cf-xr9d9"
Jul 08 12:41:17.074347 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:17.074342    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-58b9d8-7746cc64cf-xr9d9" event={"ID":"69c4ca84-7359-4228-abab-1a5bc2cc4123","Type":"ContainerDied","Data":"e4af8f1ea542339fe7f7aba36c518dd4726bdb3d80b3cd294df9b7f33eeca79d"}
Jul 08 12:41:17.074939 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:17.074363    2579 scope.go:117] "RemoveContainer" containerID="3fc1af947f48d47ff7127fbc889aa600555a45ff0067c4e5a3f5eab393ee4d14"
Jul 08 12:41:17.083036 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:17.083013    2579 scope.go:117] "RemoveContainer" containerID="3fc1af947f48d47ff7127fbc889aa600555a45ff0067c4e5a3f5eab393ee4d14"
Jul 08 12:41:17.083300 ip-10-0-138-89 kubenswrapper[2579]: E0708 12:41:17.083279    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fc1af947f48d47ff7127fbc889aa600555a45ff0067c4e5a3f5eab393ee4d14\": container with ID starting with 3fc1af947f48d47ff7127fbc889aa600555a45ff0067c4e5a3f5eab393ee4d14 not found: ID does not exist" containerID="3fc1af947f48d47ff7127fbc889aa600555a45ff0067c4e5a3f5eab393ee4d14"
Jul 08 12:41:17.083369 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:17.083313    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fc1af947f48d47ff7127fbc889aa600555a45ff0067c4e5a3f5eab393ee4d14"} err="failed to get container status \"3fc1af947f48d47ff7127fbc889aa600555a45ff0067c4e5a3f5eab393ee4d14\": rpc error: code = NotFound desc = could not find container \"3fc1af947f48d47ff7127fbc889aa600555a45ff0067c4e5a3f5eab393ee4d14\": container with ID starting with 3fc1af947f48d47ff7127fbc889aa600555a45ff0067c4e5a3f5eab393ee4d14 not found: ID does not exist"
Jul 08 12:41:17.107446 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:17.107414    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-58b9d8-7746cc64cf-xr9d9"]
Jul 08 12:41:17.119072 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:17.119038    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-58b9d8-7746cc64cf-xr9d9"]
Jul 08 12:41:18.716680 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:18.716637    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69c4ca84-7359-4228-abab-1a5bc2cc4123" path="/var/lib/kubelet/pods/69c4ca84-7359-4228-abab-1a5bc2cc4123/volumes"
Jul 08 12:41:24.907128 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:24.907091    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gkcph"]
Jul 08 12:41:27.381367 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:27.381344    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-9qwx5_19452e60-a504-485d-bf01-e0acffd5f18c/manager/0.log"
Jul 08 12:41:27.507532 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:27.507501    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-2g5zq_8317baa3-d235-47f8-94c4-308c96c6fb2a/manager/0.log"
Jul 08 12:41:27.627757 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:27.627728    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-858769f6f8-p5nz2_391eb116-aa03-4f92-bd56-514f552816f5/manager/0.log"
Jul 08 12:41:28.254429 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:28.254394    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6db55b68b6-wrjvx_b083fd94-85f6-42f9-a56b-ed83d1e2c629/copy-manifests/0.log"
Jul 08 12:41:28.261870 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:28.261836    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6db55b68b6-wrjvx_b083fd94-85f6-42f9-a56b-ed83d1e2c629/manager/0.log"
Jul 08 12:41:29.691059 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:29.691026    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-7xlrf_eb3b9fcd-8703-4c3a-8709-f349ea0cc5e3/manager/0.log"
Jul 08 12:41:29.800591 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:29.800561    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-64lwx_9340764e-4d7d-4140-9111-e58bedcfee6b/manager/0.log"
Jul 08 12:41:29.912595 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:29.912538    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-vhwhh_6d96c715-e042-4e8f-9adf-de115944c84f/kuadrant-console-plugin/0.log"
Jul 08 12:41:30.031693 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:30.031611    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-klrp9_06529644-27d8-4092-9f2f-4199a0132d2a/registry-server/0.log"
Jul 08 12:41:30.269147 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:30.269120    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-gkcph_13eeecbc-6a72-4f7b-8d26-99e73d820a30/limitador/0.log"
Jul 08 12:41:30.388870 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:30.388797    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-hcwbd_46272467-ba68-4376-bd7e-ffed4281cd12/manager/0.log"
Jul 08 12:41:30.965753 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:30.965728    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-765c5b48c6-kq8lm_fcb07119-dbca-41f5-98ee-bdcfef917bd3/kube-auth-proxy/0.log"
Jul 08 12:41:31.323075 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:31.322986    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-hq45b_30ac9628-8965-48bd-b98e-d632fc85a69a/payload-pre-processing/0.log"
Jul 08 12:41:31.443954 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:31.443926    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-2s49w_9dbe08ff-b44a-48f4-911a-ca873bdd44a8/payload-processing/0.log"
Jul 08 12:41:31.563966 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:31.563935    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-697fbc8d47-wlscv_67af172b-92fa-43c3-a989-4d42968b3d0e/router/0.log"
Jul 08 12:41:32.023904 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:32.023877    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-s4kld_fe12af73-3e74-4e9a-88c9-03935365fec2/main/0.log"
Jul 08 12:41:32.365822 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:32.365723    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-cf6tp_7ada9676-dcfc-4722-82d6-90f7bd74d7ec/main/0.log"
Jul 08 12:41:39.917679 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:39.917651    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-cslr6_9e4572a9-69d7-418c-b751-07c5d6759d5a/global-pull-secret-syncer/0.log"
Jul 08 12:41:40.174522 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:40.174491    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-pp86n_56220650-f743-4804-9e3e-5a0cffd4fd3a/konnectivity-agent/0.log"
Jul 08 12:41:40.310711 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:40.310679    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-89.ec2.internal_01a7cd28e5cb68af10b158e7471bca6f/haproxy/0.log"
Jul 08 12:41:45.479306 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:45.479273    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-7xlrf_eb3b9fcd-8703-4c3a-8709-f349ea0cc5e3/manager/0.log"
Jul 08 12:41:45.507641 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:45.507605    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-64lwx_9340764e-4d7d-4140-9111-e58bedcfee6b/manager/0.log"
Jul 08 12:41:45.551981 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:45.551951    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-vhwhh_6d96c715-e042-4e8f-9adf-de115944c84f/kuadrant-console-plugin/0.log"
Jul 08 12:41:45.592954 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:45.592924    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-klrp9_06529644-27d8-4092-9f2f-4199a0132d2a/registry-server/0.log"
Jul 08 12:41:45.813451 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:45.813364    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-gkcph_13eeecbc-6a72-4f7b-8d26-99e73d820a30/limitador/0.log"
Jul 08 12:41:45.878283 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:45.878251    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-hcwbd_46272467-ba68-4376-bd7e-ffed4281cd12/manager/0.log"
Jul 08 12:41:47.577157 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:47.577121    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d7ac8123-78b8-4ca2-8ed7-e118f28a3248/alertmanager/0.log"
Jul 08 12:41:47.605997 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:47.605968    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d7ac8123-78b8-4ca2-8ed7-e118f28a3248/config-reloader/0.log"
Jul 08 12:41:47.632127 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:47.632095    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d7ac8123-78b8-4ca2-8ed7-e118f28a3248/kube-rbac-proxy-web/0.log"
Jul 08 12:41:47.661464 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:47.661435    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d7ac8123-78b8-4ca2-8ed7-e118f28a3248/kube-rbac-proxy/0.log"
Jul 08 12:41:47.689682 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:47.689655    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d7ac8123-78b8-4ca2-8ed7-e118f28a3248/kube-rbac-proxy-metric/0.log"
Jul 08 12:41:47.715787 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:47.715757    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d7ac8123-78b8-4ca2-8ed7-e118f28a3248/prom-label-proxy/0.log"
Jul 08 12:41:47.742088 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:47.742062    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d7ac8123-78b8-4ca2-8ed7-e118f28a3248/init-config-reloader/0.log"
Jul 08 12:41:48.152142 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:48.152116    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hf42b_90f23b67-264d-4ec0-bac5-d87fcab5c19d/node-exporter/0.log"
Jul 08 12:41:48.176561 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:48.176534    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hf42b_90f23b67-264d-4ec0-bac5-d87fcab5c19d/kube-rbac-proxy/0.log"
Jul 08 12:41:48.204684 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:48.204658    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hf42b_90f23b67-264d-4ec0-bac5-d87fcab5c19d/init-textfile/0.log"
Jul 08 12:41:50.307015 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:50.306983    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-drpcz/perf-node-gather-daemonset-6tqr2"]
Jul 08 12:41:50.307388 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:50.307349    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69c4ca84-7359-4228-abab-1a5bc2cc4123" containerName="maas-api"
Jul 08 12:41:50.307388 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:50.307359    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="69c4ca84-7359-4228-abab-1a5bc2cc4123" containerName="maas-api"
Jul 08 12:41:50.307388 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:50.307373    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8dd8c9aa-b41d-495c-a6eb-9ec2ecf0cf42" containerName="maas-api"
Jul 08 12:41:50.307388 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:50.307379    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8dd8c9aa-b41d-495c-a6eb-9ec2ecf0cf42" containerName="maas-api"
Jul 08 12:41:50.307525 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:50.307448    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8dd8c9aa-b41d-495c-a6eb-9ec2ecf0cf42" containerName="maas-api"
Jul 08 12:41:50.307525 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:50.307457    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="69c4ca84-7359-4228-abab-1a5bc2cc4123" containerName="maas-api"
Jul 08 12:41:50.310656 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:50.310636    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-drpcz/perf-node-gather-daemonset-6tqr2"
Jul 08 12:41:50.313207 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:50.313181    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-drpcz\"/\"openshift-service-ca.crt\""
Jul 08 12:41:50.313320 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:50.313214    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-drpcz\"/\"default-dockercfg-8xw2t\""
Jul 08 12:41:50.314180 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:50.314162    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-drpcz\"/\"kube-root-ca.crt\""
Jul 08 12:41:50.320888 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:50.320865    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-drpcz/perf-node-gather-daemonset-6tqr2"]
Jul 08 12:41:50.347054 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:50.347014    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d12f391b-83a5-4112-a11b-342c7adffb73-proc\") pod \"perf-node-gather-daemonset-6tqr2\" (UID: \"d12f391b-83a5-4112-a11b-342c7adffb73\") " pod="openshift-must-gather-drpcz/perf-node-gather-daemonset-6tqr2"
Jul 08 12:41:50.347054 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:50.347049    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d12f391b-83a5-4112-a11b-342c7adffb73-podres\") pod \"perf-node-gather-daemonset-6tqr2\" (UID: \"d12f391b-83a5-4112-a11b-342c7adffb73\") " pod="openshift-must-gather-drpcz/perf-node-gather-daemonset-6tqr2"
Jul 08 12:41:50.347276 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:50.347080    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkt8g\" (UniqueName: \"kubernetes.io/projected/d12f391b-83a5-4112-a11b-342c7adffb73-kube-api-access-fkt8g\") pod \"perf-node-gather-daemonset-6tqr2\" (UID: \"d12f391b-83a5-4112-a11b-342c7adffb73\") " pod="openshift-must-gather-drpcz/perf-node-gather-daemonset-6tqr2"
Jul 08 12:41:50.347276 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:50.347156    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d12f391b-83a5-4112-a11b-342c7adffb73-sys\") pod \"perf-node-gather-daemonset-6tqr2\" (UID: \"d12f391b-83a5-4112-a11b-342c7adffb73\") " pod="openshift-must-gather-drpcz/perf-node-gather-daemonset-6tqr2"
Jul 08 12:41:50.347276 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:50.347203    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d12f391b-83a5-4112-a11b-342c7adffb73-lib-modules\") pod \"perf-node-gather-daemonset-6tqr2\" (UID: \"d12f391b-83a5-4112-a11b-342c7adffb73\") " pod="openshift-must-gather-drpcz/perf-node-gather-daemonset-6tqr2"
Jul 08 12:41:50.448586 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:50.448549    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d12f391b-83a5-4112-a11b-342c7adffb73-proc\") pod \"perf-node-gather-daemonset-6tqr2\" (UID: \"d12f391b-83a5-4112-a11b-342c7adffb73\") " pod="openshift-must-gather-drpcz/perf-node-gather-daemonset-6tqr2"
Jul 08 12:41:50.448793 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:50.448592    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d12f391b-83a5-4112-a11b-342c7adffb73-podres\") pod \"perf-node-gather-daemonset-6tqr2\" (UID: \"d12f391b-83a5-4112-a11b-342c7adffb73\") " pod="openshift-must-gather-drpcz/perf-node-gather-daemonset-6tqr2"
Jul 08 12:41:50.448793 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:50.448673    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d12f391b-83a5-4112-a11b-342c7adffb73-proc\") pod \"perf-node-gather-daemonset-6tqr2\" (UID: \"d12f391b-83a5-4112-a11b-342c7adffb73\") " pod="openshift-must-gather-drpcz/perf-node-gather-daemonset-6tqr2"
Jul 08 12:41:50.448793 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:50.448712    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkt8g\" (UniqueName: \"kubernetes.io/projected/d12f391b-83a5-4112-a11b-342c7adffb73-kube-api-access-fkt8g\") pod \"perf-node-gather-daemonset-6tqr2\" (UID: \"d12f391b-83a5-4112-a11b-342c7adffb73\") " pod="openshift-must-gather-drpcz/perf-node-gather-daemonset-6tqr2"
Jul 08 12:41:50.448793 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:50.448756    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d12f391b-83a5-4112-a11b-342c7adffb73-sys\") pod \"perf-node-gather-daemonset-6tqr2\" (UID: \"d12f391b-83a5-4112-a11b-342c7adffb73\") " pod="openshift-must-gather-drpcz/perf-node-gather-daemonset-6tqr2"
Jul 08 12:41:50.448793 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:50.448713    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d12f391b-83a5-4112-a11b-342c7adffb73-podres\") pod \"perf-node-gather-daemonset-6tqr2\" (UID: \"d12f391b-83a5-4112-a11b-342c7adffb73\") " pod="openshift-must-gather-drpcz/perf-node-gather-daemonset-6tqr2"
Jul 08 12:41:50.448793 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:50.448792    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d12f391b-83a5-4112-a11b-342c7adffb73-lib-modules\") pod \"perf-node-gather-daemonset-6tqr2\" (UID: \"d12f391b-83a5-4112-a11b-342c7adffb73\") " pod="openshift-must-gather-drpcz/perf-node-gather-daemonset-6tqr2"
Jul 08 12:41:50.449076 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:50.448938    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d12f391b-83a5-4112-a11b-342c7adffb73-sys\") pod \"perf-node-gather-daemonset-6tqr2\" (UID: \"d12f391b-83a5-4112-a11b-342c7adffb73\") " pod="openshift-must-gather-drpcz/perf-node-gather-daemonset-6tqr2"
Jul 08 12:41:50.449076 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:50.448957    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d12f391b-83a5-4112-a11b-342c7adffb73-lib-modules\") pod \"perf-node-gather-daemonset-6tqr2\" (UID: \"d12f391b-83a5-4112-a11b-342c7adffb73\") " pod="openshift-must-gather-drpcz/perf-node-gather-daemonset-6tqr2"
Jul 08 12:41:50.458548 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:50.458479    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkt8g\" (UniqueName: \"kubernetes.io/projected/d12f391b-83a5-4112-a11b-342c7adffb73-kube-api-access-fkt8g\") pod \"perf-node-gather-daemonset-6tqr2\" (UID: \"d12f391b-83a5-4112-a11b-342c7adffb73\") " pod="openshift-must-gather-drpcz/perf-node-gather-daemonset-6tqr2"
Jul 08 12:41:50.624376 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:50.624284    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-drpcz\"/\"default-dockercfg-8xw2t\""
Jul 08 12:41:50.632252 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:50.631981    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-drpcz/perf-node-gather-daemonset-6tqr2"
Jul 08 12:41:50.778892 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:50.778862    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-drpcz/perf-node-gather-daemonset-6tqr2"]
Jul 08 12:41:50.781522 ip-10-0-138-89 kubenswrapper[2579]: W0708 12:41:50.781491    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd12f391b_83a5_4112_a11b_342c7adffb73.slice/crio-59f782b8ef9c6c2d4212041a2ba44f75fbab2b4c8db696fcd484b19833a6c9ea WatchSource:0}: Error finding container 59f782b8ef9c6c2d4212041a2ba44f75fbab2b4c8db696fcd484b19833a6c9ea: Status 404 returned error can't find the container with id 59f782b8ef9c6c2d4212041a2ba44f75fbab2b4c8db696fcd484b19833a6c9ea
Jul 08 12:41:51.179010 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:51.178976    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-78887568fd-gnvfk_b6af39b1-04e6-48c4-b153-f83f77d1d28f/console/0.log"
Jul 08 12:41:51.246595 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:51.246559    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-drpcz/perf-node-gather-daemonset-6tqr2" event={"ID":"d12f391b-83a5-4112-a11b-342c7adffb73","Type":"ContainerStarted","Data":"9c013f7e396f3f08ad656f791da8632eed2f587f621e8d259bea22ef825f6bb7"}
Jul 08 12:41:51.246824 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:51.246612    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-drpcz/perf-node-gather-daemonset-6tqr2" event={"ID":"d12f391b-83a5-4112-a11b-342c7adffb73","Type":"ContainerStarted","Data":"59f782b8ef9c6c2d4212041a2ba44f75fbab2b4c8db696fcd484b19833a6c9ea"}
Jul 08 12:41:51.246824 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:51.246680    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-drpcz/perf-node-gather-daemonset-6tqr2"
Jul 08 12:41:51.266900 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:51.266844    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-drpcz/perf-node-gather-daemonset-6tqr2" podStartSLOduration=1.266827708 podStartE2EDuration="1.266827708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 12:41:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 12:41:51.265181932 +0000 UTC m=+2400.995556915" watchObservedRunningTime="2026-07-08 12:41:51.266827708 +0000 UTC m=+2400.997202678"
Jul 08 12:41:52.903110 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:52.903082    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jprx6_6fb2ecb5-0ba5-4b15-9654-76648d18ed3b/dns/0.log"
Jul 08 12:41:52.929125 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:52.929090    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jprx6_6fb2ecb5-0ba5-4b15-9654-76648d18ed3b/kube-rbac-proxy/0.log"
Jul 08 12:41:53.010654 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:53.010628    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-26hpb_c0abdbfc-0312-4fd1-9ca4-098e3fa3e7e1/dns-node-resolver/0.log"
Jul 08 12:41:53.612715 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:53.612684    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-86bdbc87b6-lrh8q_d0f99899-5a9a-443c-9ca3-3033d5699f2a/registry/0.log"
Jul 08 12:41:53.667984 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:53.667956    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-k4jsh_641b596d-df5a-491a-a40e-e269de0a676e/node-ca/0.log"
Jul 08 12:41:54.833446 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:54.833418    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-765c5b48c6-kq8lm_fcb07119-dbca-41f5-98ee-bdcfef917bd3/kube-auth-proxy/0.log"
Jul 08 12:41:54.965163 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:54.965131    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-hq45b_30ac9628-8965-48bd-b98e-d632fc85a69a/payload-pre-processing/0.log"
Jul 08 12:41:55.018112 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:55.018074    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-2s49w_9dbe08ff-b44a-48f4-911a-ca873bdd44a8/payload-processing/0.log"
Jul 08 12:41:55.044599 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:55.044564    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-697fbc8d47-wlscv_67af172b-92fa-43c3-a989-4d42968b3d0e/router/0.log"
Jul 08 12:41:55.766732 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:55.766702    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-nn7z8_d8e14b6c-7fd3-46b6-8e03-f8c03d924e2b/serve-healthcheck-canary/0.log"
Jul 08 12:41:56.298534 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:56.298498    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-9bgrq_ebb356ca-384f-4018-bf6d-9c9a1ee748bd/insights-operator/0.log"
Jul 08 12:41:56.299987 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:56.299966    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-9bgrq_ebb356ca-384f-4018-bf6d-9c9a1ee748bd/insights-operator/1.log"
Jul 08 12:41:56.411925 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:56.411888    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8mcpn_52314101-63ee-448f-963f-16b1dd3cdedb/kube-rbac-proxy/0.log"
Jul 08 12:41:56.437414 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:56.437379    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8mcpn_52314101-63ee-448f-963f-16b1dd3cdedb/exporter/0.log"
Jul 08 12:41:56.463212 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:56.463182    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8mcpn_52314101-63ee-448f-963f-16b1dd3cdedb/extractor/0.log"
Jul 08 12:41:57.261881 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:57.261851    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-drpcz/perf-node-gather-daemonset-6tqr2"
Jul 08 12:41:58.724500 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:58.724414    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-9qwx5_19452e60-a504-485d-bf01-e0acffd5f18c/manager/0.log"
Jul 08 12:41:58.850841 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:58.850802    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-2g5zq_8317baa3-d235-47f8-94c4-308c96c6fb2a/manager/0.log"
Jul 08 12:41:58.938936 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:58.938903    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-858769f6f8-p5nz2_391eb116-aa03-4f92-bd56-514f552816f5/manager/0.log"
Jul 08 12:41:59.235084 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:59.235053    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6db55b68b6-wrjvx_b083fd94-85f6-42f9-a56b-ed83d1e2c629/manager/0.log"
Jul 08 12:41:59.262604 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:41:59.262577    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6db55b68b6-wrjvx_b083fd94-85f6-42f9-a56b-ed83d1e2c629/copy-manifests/0.log"
Jul 08 12:42:00.713320 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:42:00.713295    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-765c5b48c6-kq8lm_fcb07119-dbca-41f5-98ee-bdcfef917bd3/kube-auth-proxy/0.log"
Jul 08 12:42:00.739909 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:42:00.739881    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-cnc2b_711211a9-ef16-47f7-b5f5-aa14eeafac9f/openshift-lws-operator/0.log"
Jul 08 12:42:00.810051 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:42:00.810014    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-hq45b_30ac9628-8965-48bd-b98e-d632fc85a69a/payload-pre-processing/0.log"
Jul 08 12:42:00.845393 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:42:00.845346    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-2s49w_9dbe08ff-b44a-48f4-911a-ca873bdd44a8/payload-processing/0.log"
Jul 08 12:42:00.861572 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:42:00.861541    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-697fbc8d47-wlscv_67af172b-92fa-43c3-a989-4d42968b3d0e/router/0.log"
Jul 08 12:42:02.183792 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:42:02.183754    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-765c5b48c6-kq8lm_fcb07119-dbca-41f5-98ee-bdcfef917bd3/kube-auth-proxy/0.log"
Jul 08 12:42:02.289031 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:42:02.288992    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-hq45b_30ac9628-8965-48bd-b98e-d632fc85a69a/payload-pre-processing/0.log"
Jul 08 12:42:02.329558 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:42:02.329525    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-2s49w_9dbe08ff-b44a-48f4-911a-ca873bdd44a8/payload-processing/0.log"
Jul 08 12:42:02.353970 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:42:02.353938    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-697fbc8d47-wlscv_67af172b-92fa-43c3-a989-4d42968b3d0e/router/0.log"
Jul 08 12:42:05.891258 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:42:05.891229    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-crpsg_738db72c-994e-43df-a7e7-53fba16a575a/migrator/0.log"
Jul 08 12:42:05.917910 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:42:05.917882    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-crpsg_738db72c-994e-43df-a7e7-53fba16a575a/graceful-termination/0.log"
Jul 08 12:42:06.345613 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:42:06.345577    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-xzlcm_79addf06-f231-4993-8ae6-a9bec4194e1a/kube-storage-version-migrator-operator/1.log"
Jul 08 12:42:06.346578 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:42:06.346559    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-xzlcm_79addf06-f231-4993-8ae6-a9bec4194e1a/kube-storage-version-migrator-operator/0.log"
Jul 08 12:42:07.640345 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:42:07.640317    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hrzgb_ffda1903-e8ae-4139-bb4c-d01904b23925/kube-multus-additional-cni-plugins/0.log"
Jul 08 12:42:07.665511 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:42:07.665485    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hrzgb_ffda1903-e8ae-4139-bb4c-d01904b23925/egress-router-binary-copy/0.log"
Jul 08 12:42:07.691432 ip-10-0-138-89 kubenswrapper[2579]: I0708 12:42:07.691407    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hrzgb_ffda1903-e8ae-4139-bb4c-d01904b23925/cni-plugins/0.log"