Jul 08 19:21:44.615999 ip-10-0-142-195 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 08 19:21:44.616012 ip-10-0-142-195 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 08 19:21:44.616022 ip-10-0-142-195 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 08 19:21:44.616571 ip-10-0-142-195 systemd[1]: Failed to start Kubernetes Kubelet. Jul 08 19:21:54.789971 ip-10-0-142-195 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 08 19:21:54.789988 ip-10-0-142-195 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 561ddb33cd3f45da822752a5d5683dcc -- Jul 08 19:23:52.946798 ip-10-0-142-195 systemd[1]: Starting Kubernetes Kubelet... Jul 08 19:23:53.367166 ip-10-0-142-195 kubenswrapper[2566]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 08 19:23:53.367166 ip-10-0-142-195 kubenswrapper[2566]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 08 19:23:53.367166 ip-10-0-142-195 kubenswrapper[2566]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 08 19:23:53.367166 ip-10-0-142-195 kubenswrapper[2566]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 08 19:23:53.367166 ip-10-0-142-195 kubenswrapper[2566]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 08 19:23:53.368619 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.368523 2566 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 08 19:23:53.372018 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372003 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 19:23:53.372018 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372018 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 19:23:53.372077 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372022 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 19:23:53.372077 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372025 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 19:23:53.372077 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372028 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 19:23:53.372077 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372031 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 19:23:53.372077 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372034 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 19:23:53.372077 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372039 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 19:23:53.372077 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372043 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 19:23:53.372077 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372046 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 19:23:53.372077 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372049 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 19:23:53.372077 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372052 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 19:23:53.372077 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372056 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 19:23:53.372077 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372059 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 19:23:53.372077 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372062 2566 feature_gate.go:328] unrecognized feature gate: Example Jul 08 19:23:53.372077 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372065 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 19:23:53.372077 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372068 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 19:23:53.372077 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372071 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 19:23:53.372077 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372073 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 19:23:53.372077 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372076 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 19:23:53.372077 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372079 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 19:23:53.372524 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372082 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 19:23:53.372524 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372085 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 19:23:53.372524 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372088 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 19:23:53.372524 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372091 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 19:23:53.372524 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372093 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 19:23:53.372524 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372096 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 19:23:53.372524 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372099 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 19:23:53.372524 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372101 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 19:23:53.372524 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372103 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 19:23:53.372524 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372106 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 19:23:53.372524 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372108 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 19:23:53.372524 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372111 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 19:23:53.372524 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372113 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 19:23:53.372524 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372116 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 19:23:53.372524 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372118 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 19:23:53.372524 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372121 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 19:23:53.372524 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372123 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 19:23:53.372524 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372126 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 19:23:53.372524 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372132 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 19:23:53.372974 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372134 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 19:23:53.372974 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372137 2566 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 19:23:53.372974 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372140 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 19:23:53.372974 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372142 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 19:23:53.372974 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372145 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 19:23:53.372974 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372147 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 19:23:53.372974 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372149 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 19:23:53.372974 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372152 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 19:23:53.372974 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372154 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 19:23:53.372974 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372157 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 19:23:53.372974 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372160 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 19:23:53.372974 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372162 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 19:23:53.372974 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372165 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 19:23:53.372974 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372168 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 19:23:53.372974 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372171 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 19:23:53.372974 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372174 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 19:23:53.372974 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372176 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 19:23:53.372974 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372179 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 19:23:53.372974 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372181 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 19:23:53.372974 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372184 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 19:23:53.373448 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372186 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 19:23:53.373448 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372189 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 19:23:53.373448 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372191 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 19:23:53.373448 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372194 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 19:23:53.373448 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372196 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 19:23:53.373448 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372199 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 19:23:53.373448 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372201 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 19:23:53.373448 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372204 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 19:23:53.373448 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372206 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 19:23:53.373448 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372208 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 19:23:53.373448 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372211 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 19:23:53.373448 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372213 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 19:23:53.373448 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372217 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 19:23:53.373448 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372219 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 19:23:53.373448 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372222 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 19:23:53.373448 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372224 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 19:23:53.373448 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372227 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 19:23:53.373448 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372229 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 19:23:53.373448 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372232 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 19:23:53.373448 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372234 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 19:23:53.373924 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372236 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 19:23:53.373924 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372239 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 19:23:53.373924 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372241 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 19:23:53.373924 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372244 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 19:23:53.373924 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372246 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 19:23:53.373924 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372250 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 19:23:53.373924 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372253 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 19:23:53.373924 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372255 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 19:23:53.373924 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372258 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 19:23:53.373924 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372262 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 19:23:53.373924 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372264 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 19:23:53.373924 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372267 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 19:23:53.373924 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372269 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 19:23:53.373924 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372274 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 19:23:53.373924 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372277 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 19:23:53.373924 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372280 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 19:23:53.373924 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372283 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 19:23:53.373924 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372286 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 19:23:53.373924 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372290 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 19:23:53.374361 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372989 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 19:23:53.374361 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.372996 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 19:23:53.374361 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373000 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 19:23:53.374361 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373002 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 19:23:53.374361 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373005 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 19:23:53.374361 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373008 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 19:23:53.374361 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373012 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 19:23:53.374361 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373014 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 19:23:53.374361 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373017 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 19:23:53.374361 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373020 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 19:23:53.374361 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373022 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 19:23:53.374361 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373025 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 19:23:53.374361 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373027 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 19:23:53.374361 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373030 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 19:23:53.374361 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373032 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 19:23:53.374361 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373035 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 19:23:53.374361 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373037 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 19:23:53.374361 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373040 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 19:23:53.374361 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373043 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 19:23:53.374827 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373046 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 19:23:53.374827 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373049 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 19:23:53.374827 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373051 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 19:23:53.374827 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373054 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 19:23:53.374827 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373056 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 19:23:53.374827 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373058 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 19:23:53.374827 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373061 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 19:23:53.374827 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373063 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 19:23:53.374827 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373066 2566 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 19:23:53.374827 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373070 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 19:23:53.374827 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373074 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 19:23:53.374827 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373077 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 19:23:53.374827 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373080 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 19:23:53.374827 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373082 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 19:23:53.374827 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373085 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 19:23:53.374827 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373087 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 19:23:53.374827 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373090 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 19:23:53.374827 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373092 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 19:23:53.374827 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373095 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 19:23:53.374827 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373098 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 19:23:53.375321 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373101 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 19:23:53.375321 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373103 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 19:23:53.375321 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373106 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 19:23:53.375321 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373108 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 19:23:53.375321 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373111 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 19:23:53.375321 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373113 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 19:23:53.375321 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373116 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 19:23:53.375321 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373119 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 19:23:53.375321 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373122 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 19:23:53.375321 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373124 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 19:23:53.375321 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373127 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 19:23:53.375321 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373130 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 19:23:53.375321 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373132 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 19:23:53.375321 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373135 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 19:23:53.375321 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373137 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 19:23:53.375321 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373140 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 19:23:53.375321 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373142 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 19:23:53.375321 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373145 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 19:23:53.375321 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373147 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 19:23:53.375321 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373150 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 19:23:53.375800 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373152 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 19:23:53.375800 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373155 2566 feature_gate.go:328] unrecognized feature gate: Example Jul 08 19:23:53.375800 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373157 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 19:23:53.375800 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373160 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 19:23:53.375800 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373162 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 19:23:53.375800 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373165 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 19:23:53.375800 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373167 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 19:23:53.375800 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373170 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 19:23:53.375800 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373172 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 19:23:53.375800 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373175 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 19:23:53.375800 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373177 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 19:23:53.375800 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373180 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 19:23:53.375800 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373183 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 19:23:53.375800 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373186 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 19:23:53.375800 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373188 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 19:23:53.375800 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373191 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 19:23:53.375800 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373193 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 19:23:53.375800 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373195 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 19:23:53.375800 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373198 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 19:23:53.375800 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373200 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 19:23:53.376288 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373203 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 19:23:53.376288 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373205 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 19:23:53.376288 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373208 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 19:23:53.376288 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373210 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 19:23:53.376288 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373215 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 19:23:53.376288 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373218 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 19:23:53.376288 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373222 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 19:23:53.376288 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373226 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 19:23:53.376288 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373229 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 19:23:53.376288 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373232 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 19:23:53.376288 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373235 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 19:23:53.376288 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373238 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 19:23:53.376288 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373240 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 19:23:53.376288 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373243 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 19:23:53.376288 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373245 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 19:23:53.376288 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373248 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 19:23:53.376288 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373250 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 19:23:53.376288 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373253 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 19:23:53.376288 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373256 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 19:23:53.376761 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373258 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 19:23:53.376761 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373638 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 19:23:53.376761 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373644 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 19:23:53.376761 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373646 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 19:23:53.376761 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373649 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 19:23:53.376761 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373652 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 19:23:53.376761 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373655 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 19:23:53.376761 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373658 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 19:23:53.376761 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373660 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 19:23:53.376761 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373663 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 19:23:53.376761 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373666 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 19:23:53.376761 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373668 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 19:23:53.376761 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373671 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 19:23:53.376761 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373674 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 19:23:53.376761 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373676 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 19:23:53.376761 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373679 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 19:23:53.376761 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373684 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 19:23:53.376761 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373687 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 19:23:53.376761 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373690 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 19:23:53.377238 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373694 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 19:23:53.377238 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373698 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 19:23:53.377238 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373701 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 19:23:53.377238 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373704 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 19:23:53.377238 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373706 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 19:23:53.377238 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373709 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 19:23:53.377238 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373712 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 19:23:53.377238 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373714 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 19:23:53.377238 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373717 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 19:23:53.377238 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373719 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 19:23:53.377238 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373722 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 19:23:53.377238 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373724 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 19:23:53.377238 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373727 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 19:23:53.377238 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373729 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 19:23:53.377238 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373733 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 19:23:53.377238 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373737 2566 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 19:23:53.377238 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373740 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 19:23:53.377238 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373745 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 19:23:53.377238 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373747 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 19:23:53.377720 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373750 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 19:23:53.377720 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373753 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 19:23:53.377720 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373756 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 19:23:53.377720 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373758 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 19:23:53.377720 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373761 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 19:23:53.377720 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373764 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 19:23:53.377720 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373766 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 19:23:53.377720 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373768 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 19:23:53.377720 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373771 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 19:23:53.377720 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373773 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 19:23:53.377720 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373777 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 19:23:53.377720 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373780 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 19:23:53.377720 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373782 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 19:23:53.377720 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373784 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 19:23:53.377720 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373788 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 19:23:53.377720 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373791 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 19:23:53.377720 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373794 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 19:23:53.377720 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373797 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 19:23:53.377720 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373799 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 19:23:53.378177 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373801 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 19:23:53.378177 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373804 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 19:23:53.378177 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373806 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 19:23:53.378177 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373809 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 19:23:53.378177 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373811 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 19:23:53.378177 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373814 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 19:23:53.378177 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373816 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 19:23:53.378177 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373819 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 19:23:53.378177 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373821 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 19:23:53.378177 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373824 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 19:23:53.378177 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373826 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 19:23:53.378177 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373830 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 19:23:53.378177 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373833 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 19:23:53.378177 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373836 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 19:23:53.378177 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373839 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 19:23:53.378177 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373842 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 19:23:53.378177 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373844 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 19:23:53.378177 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373846 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 19:23:53.378177 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373849 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 19:23:53.378177 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373851 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 19:23:53.378704 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373854 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 19:23:53.378704 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373856 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 19:23:53.378704 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373858 2566 feature_gate.go:328] unrecognized feature gate: Example Jul 08 19:23:53.378704 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373862 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 19:23:53.378704 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373865 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 19:23:53.378704 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373867 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 19:23:53.378704 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373870 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 19:23:53.378704 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373873 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 19:23:53.378704 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373875 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 19:23:53.378704 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373878 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 19:23:53.378704 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373880 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 19:23:53.378704 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373883 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 19:23:53.378704 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373885 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 19:23:53.378704 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373887 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 19:23:53.378704 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373890 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 19:23:53.378704 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373892 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 19:23:53.378704 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373895 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 19:23:53.378704 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373897 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 19:23:53.378704 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373900 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 19:23:53.378704 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373902 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 19:23:53.379190 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373904 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 19:23:53.379190 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373907 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 19:23:53.379190 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.373909 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 19:23:53.379190 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.373974 2566 flags.go:64] FLAG: --address="0.0.0.0" Jul 08 19:23:53.379190 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.373981 2566 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 08 19:23:53.379190 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.373987 2566 flags.go:64] FLAG: --anonymous-auth="true" Jul 08 19:23:53.379190 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.373991 2566 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 08 19:23:53.379190 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.373995 2566 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 08 19:23:53.379190 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.373998 2566 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 08 19:23:53.379190 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374003 2566 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 08 19:23:53.379190 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374006 2566 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 08 19:23:53.379190 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374009 2566 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 08 19:23:53.379190 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374026 2566 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 08 19:23:53.379190 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374030 2566 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 08 19:23:53.379190 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374034 2566 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 08 19:23:53.379190 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374038 2566 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 08 19:23:53.379190 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374042 2566 flags.go:64] FLAG: --cgroup-root="" Jul 08 19:23:53.379190 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374045 2566 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 08 19:23:53.379190 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374048 2566 flags.go:64] FLAG: --client-ca-file="" Jul 08 19:23:53.379190 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374051 2566 flags.go:64] FLAG: --cloud-provider="external" Jul 08 19:23:53.379190 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374053 2566 flags.go:64] FLAG: --cluster-dns="[]" Jul 08 19:23:53.379190 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374057 2566 flags.go:64] FLAG: --cluster-domain="" Jul 08 19:23:53.379190 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374060 2566 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 08 19:23:53.379190 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374064 2566 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 08 19:23:53.379814 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374067 2566 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 08 19:23:53.379814 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374070 2566 flags.go:64] FLAG: --container-log-max-files="5" Jul 08 19:23:53.379814 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374073 2566 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 08 19:23:53.379814 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374076 2566 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 08 19:23:53.379814 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374079 2566 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 08 19:23:53.379814 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374082 2566 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 08 19:23:53.379814 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374085 2566 flags.go:64] FLAG: --contention-profiling="false" Jul 08 19:23:53.379814 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374088 2566 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 08 19:23:53.379814 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374091 2566 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 08 19:23:53.379814 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374094 2566 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 08 19:23:53.379814 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374096 2566 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 08 19:23:53.379814 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374101 2566 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 08 19:23:53.379814 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374104 2566 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 08 19:23:53.379814 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374107 2566 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 08 19:23:53.379814 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374110 2566 flags.go:64] FLAG: --enable-load-reader="false" Jul 08 19:23:53.379814 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374112 2566 flags.go:64] FLAG: --enable-server="true" Jul 08 19:23:53.379814 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374115 2566 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 08 19:23:53.379814 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374119 2566 flags.go:64] FLAG: --event-burst="100" Jul 08 19:23:53.379814 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374122 2566 flags.go:64] FLAG: --event-qps="50" Jul 08 19:23:53.379814 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374125 2566 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 08 19:23:53.379814 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374128 2566 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 08 19:23:53.379814 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374131 2566 flags.go:64] FLAG: --eviction-hard="" Jul 08 19:23:53.379814 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374135 2566 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 08 19:23:53.379814 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374139 2566 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 08 19:23:53.380372 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374141 2566 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 08 19:23:53.380372 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374144 2566 flags.go:64] FLAG: --eviction-soft="" Jul 08 19:23:53.380372 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374147 2566 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 08 19:23:53.380372 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374150 2566 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 08 19:23:53.380372 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374152 2566 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 08 19:23:53.380372 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374155 2566 flags.go:64] FLAG: --experimental-mounter-path="" Jul 08 19:23:53.380372 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374158 2566 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 08 19:23:53.380372 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374161 2566 flags.go:64] FLAG: --fail-swap-on="true" Jul 08 19:23:53.380372 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374163 2566 flags.go:64] FLAG: --feature-gates="" Jul 08 19:23:53.380372 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374167 2566 flags.go:64] FLAG: --file-check-frequency="20s" Jul 08 19:23:53.380372 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374174 2566 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 08 19:23:53.380372 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374177 2566 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 08 19:23:53.380372 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374180 2566 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 08 19:23:53.380372 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374183 2566 flags.go:64] FLAG: --healthz-port="10248" Jul 08 19:23:53.380372 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374186 2566 flags.go:64] FLAG: --help="false" Jul 08 19:23:53.380372 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374189 2566 flags.go:64] FLAG: --hostname-override="ip-10-0-142-195.ec2.internal" Jul 08 19:23:53.380372 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374192 2566 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 08 19:23:53.380372 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374195 2566 flags.go:64] FLAG: --http-check-frequency="20s" Jul 08 19:23:53.380372 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374198 2566 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 08 19:23:53.380372 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374201 2566 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 08 19:23:53.380372 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374204 2566 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 08 19:23:53.380372 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374207 2566 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 08 19:23:53.380372 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374210 2566 flags.go:64] FLAG: --image-service-endpoint="" Jul 08 19:23:53.380959 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374213 2566 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 08 19:23:53.380959 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374216 2566 flags.go:64] FLAG: --kube-api-burst="100" Jul 08 19:23:53.380959 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374219 2566 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 08 19:23:53.380959 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374225 2566 flags.go:64] FLAG: --kube-api-qps="50" Jul 08 19:23:53.380959 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374228 2566 flags.go:64] FLAG: --kube-reserved="" Jul 08 19:23:53.380959 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374230 2566 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 08 19:23:53.380959 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374233 2566 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 08 19:23:53.380959 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374236 2566 flags.go:64] FLAG: --kubelet-cgroups="" Jul 08 19:23:53.380959 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374240 2566 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 08 19:23:53.380959 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374243 2566 flags.go:64] FLAG: --lock-file="" Jul 08 19:23:53.380959 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374246 2566 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 08 19:23:53.380959 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374249 2566 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 08 19:23:53.380959 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374251 2566 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 08 19:23:53.380959 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374257 2566 flags.go:64] FLAG: --log-json-split-stream="false" Jul 08 19:23:53.380959 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374259 2566 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 08 19:23:53.380959 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374262 2566 flags.go:64] FLAG: --log-text-split-stream="false" Jul 08 19:23:53.380959 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374265 2566 flags.go:64] FLAG: --logging-format="text" Jul 08 19:23:53.380959 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374268 2566 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 08 19:23:53.380959 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374271 2566 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 08 19:23:53.380959 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374273 2566 flags.go:64] FLAG: --manifest-url="" Jul 08 19:23:53.380959 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374276 2566 flags.go:64] FLAG: --manifest-url-header="" Jul 08 19:23:53.380959 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374280 2566 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 08 19:23:53.380959 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374283 2566 flags.go:64] FLAG: --max-open-files="1000000" Jul 08 19:23:53.380959 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374287 2566 flags.go:64] FLAG: --max-pods="110" Jul 08 19:23:53.380959 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374290 2566 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 08 19:23:53.381615 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374293 2566 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 08 19:23:53.381615 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374295 2566 flags.go:64] FLAG: --memory-manager-policy="None" Jul 08 19:23:53.381615 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374298 2566 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 08 19:23:53.381615 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374301 2566 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 08 19:23:53.381615 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374305 2566 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 08 19:23:53.381615 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374308 2566 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 08 19:23:53.381615 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374314 2566 flags.go:64] FLAG: --node-status-max-images="50" Jul 08 19:23:53.381615 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374317 2566 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 08 19:23:53.381615 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374319 2566 flags.go:64] FLAG: --oom-score-adj="-999" Jul 08 19:23:53.381615 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374322 2566 flags.go:64] FLAG: --pod-cidr="" Jul 08 19:23:53.381615 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374325 2566 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 08 19:23:53.381615 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374331 2566 flags.go:64] FLAG: --pod-manifest-path="" Jul 08 19:23:53.381615 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374334 2566 flags.go:64] FLAG: --pod-max-pids="-1" Jul 08 19:23:53.381615 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374337 2566 flags.go:64] FLAG: --pods-per-core="0" Jul 08 19:23:53.381615 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374340 2566 flags.go:64] FLAG: --port="10250" Jul 08 19:23:53.381615 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374343 2566 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 08 19:23:53.381615 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374347 2566 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-074b06587d75e6039" Jul 08 19:23:53.381615 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374351 2566 flags.go:64] FLAG: --qos-reserved="" Jul 08 19:23:53.381615 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374354 2566 flags.go:64] FLAG: --read-only-port="10255" Jul 08 19:23:53.381615 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374356 2566 flags.go:64] FLAG: --register-node="true" Jul 08 19:23:53.381615 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374359 2566 flags.go:64] FLAG: --register-with-taints="" Jul 08 19:23:53.381615 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374363 2566 flags.go:64] FLAG: --registry-burst="10" Jul 08 19:23:53.381615 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374366 2566 flags.go:64] FLAG: --registry-qps="5" Jul 08 19:23:53.381615 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374369 2566 flags.go:64] FLAG: --reserved-cpus="" Jul 08 19:23:53.382250 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374371 2566 flags.go:64] FLAG: --reserved-memory="" Jul 08 19:23:53.382250 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374375 2566 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 08 19:23:53.382250 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374378 2566 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 08 19:23:53.382250 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374381 2566 flags.go:64] FLAG: --rotate-certificates="false" Jul 08 19:23:53.382250 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374384 2566 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 08 19:23:53.382250 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374387 2566 flags.go:64] FLAG: --runonce="false" Jul 08 19:23:53.382250 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374389 2566 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 08 19:23:53.382250 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374392 2566 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 08 19:23:53.382250 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374395 2566 flags.go:64] FLAG: --seccomp-default="false" Jul 08 19:23:53.382250 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374398 2566 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 08 19:23:53.382250 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374401 2566 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 08 19:23:53.382250 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374403 2566 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 08 19:23:53.382250 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374407 2566 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 08 19:23:53.382250 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374409 2566 flags.go:64] FLAG: --storage-driver-password="root" Jul 08 19:23:53.382250 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374412 2566 flags.go:64] FLAG: --storage-driver-secure="false" Jul 08 19:23:53.382250 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374415 2566 flags.go:64] FLAG: --storage-driver-table="stats" Jul 08 19:23:53.382250 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374418 2566 flags.go:64] FLAG: --storage-driver-user="root" Jul 08 19:23:53.382250 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374420 2566 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 08 19:23:53.382250 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374423 2566 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 08 19:23:53.382250 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374426 2566 flags.go:64] FLAG: --system-cgroups="" Jul 08 19:23:53.382250 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374429 2566 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 08 19:23:53.382250 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374434 2566 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 08 19:23:53.382250 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374437 2566 flags.go:64] FLAG: --tls-cert-file="" Jul 08 19:23:53.382250 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374439 2566 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 08 19:23:53.382250 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374444 2566 flags.go:64] FLAG: --tls-min-version="" Jul 08 19:23:53.383237 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374447 2566 flags.go:64] FLAG: --tls-private-key-file="" Jul 08 19:23:53.383237 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374450 2566 flags.go:64] FLAG: --topology-manager-policy="none" Jul 08 19:23:53.383237 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374452 2566 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 08 19:23:53.383237 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374455 2566 flags.go:64] FLAG: --topology-manager-scope="container" Jul 08 19:23:53.383237 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374458 2566 flags.go:64] FLAG: --v="2" Jul 08 19:23:53.383237 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374462 2566 flags.go:64] FLAG: --version="false" Jul 08 19:23:53.383237 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374465 2566 flags.go:64] FLAG: --vmodule="" Jul 08 19:23:53.383237 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374468 2566 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 08 19:23:53.383237 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374484 2566 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 08 19:23:53.383237 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374604 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 19:23:53.383237 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374608 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 19:23:53.383237 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374611 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 19:23:53.383237 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374614 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 19:23:53.383237 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374617 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 19:23:53.383237 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374620 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 19:23:53.383237 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374623 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 19:23:53.383237 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374625 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 19:23:53.383237 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374628 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 19:23:53.383237 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374633 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 19:23:53.383237 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374636 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 19:23:53.383237 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374639 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 19:23:53.383861 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374642 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 19:23:53.383861 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374644 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 19:23:53.383861 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374647 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 19:23:53.383861 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374649 2566 feature_gate.go:328] unrecognized feature gate: Example Jul 08 19:23:53.383861 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374652 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 19:23:53.383861 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374655 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 19:23:53.383861 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374657 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 19:23:53.383861 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374660 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 19:23:53.383861 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374662 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 19:23:53.383861 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374665 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 19:23:53.383861 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374668 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 19:23:53.383861 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374671 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 19:23:53.383861 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374673 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 19:23:53.383861 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374676 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 19:23:53.383861 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374678 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 19:23:53.383861 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374681 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 19:23:53.383861 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374684 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 19:23:53.383861 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374686 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 19:23:53.383861 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374689 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 19:23:53.383861 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374691 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 19:23:53.384336 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374694 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 19:23:53.384336 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374697 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 19:23:53.384336 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374700 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 19:23:53.384336 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374702 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 19:23:53.384336 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374705 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 19:23:53.384336 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374707 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 19:23:53.384336 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374710 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 19:23:53.384336 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374712 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 19:23:53.384336 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374715 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 19:23:53.384336 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374719 2566 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 19:23:53.384336 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374721 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 19:23:53.384336 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374724 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 19:23:53.384336 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374726 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 19:23:53.384336 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374728 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 19:23:53.384336 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374731 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 19:23:53.384336 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374734 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 19:23:53.384336 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374736 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 19:23:53.384336 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374738 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 19:23:53.384336 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374741 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 19:23:53.384336 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374744 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 19:23:53.384825 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374747 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 19:23:53.384825 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374750 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 19:23:53.384825 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374753 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 19:23:53.384825 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374755 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 19:23:53.384825 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374758 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 19:23:53.384825 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374760 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 19:23:53.384825 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374763 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 19:23:53.384825 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374765 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 19:23:53.384825 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374768 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 19:23:53.384825 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374770 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 19:23:53.384825 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374772 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 19:23:53.384825 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374775 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 19:23:53.384825 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374777 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 19:23:53.384825 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374779 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 19:23:53.384825 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374782 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 19:23:53.384825 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374785 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 19:23:53.384825 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374787 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 19:23:53.384825 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374789 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 19:23:53.384825 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374792 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 19:23:53.384825 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374794 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 19:23:53.385304 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374797 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 19:23:53.385304 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374800 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 19:23:53.385304 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374803 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 19:23:53.385304 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374805 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 19:23:53.385304 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374807 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 19:23:53.385304 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374810 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 19:23:53.385304 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374813 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 19:23:53.385304 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374815 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 19:23:53.385304 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374818 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 19:23:53.385304 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374820 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 19:23:53.385304 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374823 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 19:23:53.385304 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374827 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 19:23:53.385304 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374830 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 19:23:53.385304 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374834 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 19:23:53.385304 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374836 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 19:23:53.385304 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374839 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 19:23:53.385304 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374842 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 19:23:53.385304 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374844 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 19:23:53.385304 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374847 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 19:23:53.385779 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374851 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 19:23:53.385779 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374854 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 19:23:53.385779 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374857 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 19:23:53.385779 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374860 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 19:23:53.385779 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374864 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 19:23:53.385779 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374866 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 19:23:53.385779 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374869 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 19:23:53.385779 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.374872 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 19:23:53.385779 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.374877 2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 08 19:23:53.385779 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.383715 2566 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 08 19:23:53.385779 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.383730 2566 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 08 19:23:53.385779 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.384500 2566 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 08 19:23:53.385779 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.384508 2566 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 08 19:23:53.386094 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384562 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 19:23:53.386094 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384568 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 19:23:53.386094 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384573 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 19:23:53.386094 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384576 2566 feature_gate.go:328] unrecognized feature gate: Example Jul 08 19:23:53.386094 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384579 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 19:23:53.386094 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384582 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 19:23:53.386094 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384586 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 19:23:53.386094 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384589 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 19:23:53.386094 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384592 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 19:23:53.386094 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384595 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 19:23:53.386094 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384597 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 19:23:53.386094 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384600 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 19:23:53.386094 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384603 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 19:23:53.386094 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384606 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 19:23:53.386094 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384609 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 19:23:53.386094 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384612 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 19:23:53.386094 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384614 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 19:23:53.386094 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384617 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 19:23:53.386094 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384619 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 19:23:53.386094 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384622 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 19:23:53.386686 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384625 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 19:23:53.386686 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384627 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 19:23:53.386686 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384630 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 19:23:53.386686 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384632 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 19:23:53.386686 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384635 2566 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 19:23:53.386686 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384638 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 19:23:53.386686 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384640 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 19:23:53.386686 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384643 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 19:23:53.386686 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384647 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 19:23:53.386686 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384649 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 19:23:53.386686 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384652 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 19:23:53.386686 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384654 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 19:23:53.386686 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384657 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 19:23:53.386686 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384660 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 19:23:53.386686 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384662 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 19:23:53.386686 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384665 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 19:23:53.386686 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384667 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 19:23:53.386686 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384670 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 19:23:53.386686 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384673 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 19:23:53.386686 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384675 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 19:23:53.387158 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384678 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 19:23:53.387158 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384680 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 19:23:53.387158 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384683 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 19:23:53.387158 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384685 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 19:23:53.387158 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384688 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 19:23:53.387158 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384690 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 19:23:53.387158 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384693 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 19:23:53.387158 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384696 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 19:23:53.387158 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384699 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 19:23:53.387158 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384701 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 19:23:53.387158 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384704 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 19:23:53.387158 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384707 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 19:23:53.387158 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384710 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 19:23:53.387158 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384713 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 19:23:53.387158 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384715 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 19:23:53.387158 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384718 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 19:23:53.387158 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384720 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 19:23:53.387158 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384723 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 19:23:53.387158 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384725 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 19:23:53.387158 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384728 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 19:23:53.387651 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384731 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 19:23:53.387651 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384733 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 19:23:53.387651 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384736 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 19:23:53.387651 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384738 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 19:23:53.387651 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384741 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 19:23:53.387651 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384744 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 19:23:53.387651 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384746 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 19:23:53.387651 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384748 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 19:23:53.387651 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384751 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 19:23:53.387651 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384753 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 19:23:53.387651 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384756 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 19:23:53.387651 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384758 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 19:23:53.387651 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384761 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 19:23:53.387651 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384763 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 19:23:53.387651 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384766 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 19:23:53.387651 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384770 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 19:23:53.387651 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384773 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 19:23:53.387651 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384776 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 19:23:53.387651 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384779 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 19:23:53.387651 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384782 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 19:23:53.388120 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384785 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 19:23:53.388120 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384787 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 19:23:53.388120 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384790 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 19:23:53.388120 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384792 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 19:23:53.388120 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384795 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 19:23:53.388120 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384798 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 19:23:53.388120 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384800 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 19:23:53.388120 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384802 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 19:23:53.388120 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384805 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 19:23:53.388120 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384807 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 19:23:53.388120 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384810 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 19:23:53.388120 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384812 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 19:23:53.388120 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384815 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 19:23:53.388120 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384817 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 19:23:53.388120 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384820 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 19:23:53.388120 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384823 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 19:23:53.388120 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384825 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 19:23:53.388120 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384828 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 19:23:53.388120 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384831 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 19:23:53.388605 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.384836 2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 08 19:23:53.388605 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384963 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 19:23:53.388605 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384968 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 19:23:53.388605 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384971 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 19:23:53.388605 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384974 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 19:23:53.388605 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384977 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 19:23:53.388605 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384980 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 19:23:53.388605 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384983 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 19:23:53.388605 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384986 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 19:23:53.388605 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384988 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 19:23:53.388605 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384991 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 19:23:53.388605 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384994 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 19:23:53.388605 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384996 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 19:23:53.388605 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.384999 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 19:23:53.388605 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385002 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 19:23:53.388968 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385004 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 19:23:53.388968 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385007 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 19:23:53.388968 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385010 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 19:23:53.388968 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385013 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 19:23:53.388968 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385015 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 19:23:53.388968 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385017 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 19:23:53.388968 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385020 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 19:23:53.388968 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385023 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 19:23:53.388968 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385026 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 19:23:53.388968 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385028 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 19:23:53.388968 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385031 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 19:23:53.388968 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385033 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 19:23:53.388968 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385036 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 19:23:53.388968 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385038 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 19:23:53.388968 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385040 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 19:23:53.388968 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385043 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 19:23:53.388968 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385046 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 19:23:53.388968 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385049 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 19:23:53.388968 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385051 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 19:23:53.388968 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385054 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 19:23:53.389453 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385057 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 19:23:53.389453 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385060 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 19:23:53.389453 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385062 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 19:23:53.389453 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385065 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 19:23:53.389453 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385068 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 19:23:53.389453 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385070 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 19:23:53.389453 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385073 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 19:23:53.389453 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385075 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 19:23:53.389453 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385078 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 19:23:53.389453 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385081 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 19:23:53.389453 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385084 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 19:23:53.389453 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385086 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 19:23:53.389453 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385089 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 19:23:53.389453 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385091 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 19:23:53.389453 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385097 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 19:23:53.389453 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385099 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 19:23:53.389453 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385102 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 19:23:53.389453 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385104 2566 feature_gate.go:328] unrecognized feature gate: Example Jul 08 19:23:53.389453 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385107 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 19:23:53.389453 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385109 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 19:23:53.390003 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385112 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 19:23:53.390003 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385114 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 19:23:53.390003 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385117 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 19:23:53.390003 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385119 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 19:23:53.390003 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385122 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 19:23:53.390003 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385124 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 19:23:53.390003 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385128 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 19:23:53.390003 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385131 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 19:23:53.390003 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385134 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 19:23:53.390003 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385138 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 19:23:53.390003 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385141 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 19:23:53.390003 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385144 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 19:23:53.390003 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385147 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 19:23:53.390003 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385149 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 19:23:53.390003 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385152 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 19:23:53.390003 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385154 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 19:23:53.390003 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385157 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 19:23:53.390003 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385159 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 19:23:53.390003 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385162 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 19:23:53.390464 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385164 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 19:23:53.390464 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385167 2566 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 19:23:53.390464 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385169 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 19:23:53.390464 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385172 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 19:23:53.390464 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385174 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 19:23:53.390464 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385178 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 19:23:53.390464 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385180 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 19:23:53.390464 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385186 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 19:23:53.390464 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385188 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 19:23:53.390464 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385191 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 19:23:53.390464 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385193 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 19:23:53.390464 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385197 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 19:23:53.390464 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385201 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 19:23:53.390464 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385204 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 19:23:53.390464 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385206 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 19:23:53.390464 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385209 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 19:23:53.390464 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385212 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 19:23:53.390464 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385214 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 19:23:53.390464 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385216 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 19:23:53.390928 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385219 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 19:23:53.390928 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385221 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 19:23:53.390928 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385224 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 19:23:53.390928 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385226 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 19:23:53.390928 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385229 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 19:23:53.390928 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385231 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 19:23:53.390928 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:53.385234 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 19:23:53.390928 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.385239 2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 08 19:23:53.390928 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.385330 2566 server.go:962] "Client rotation is on, will bootstrap in background" Jul 08 19:23:53.390928 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.388057 2566 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 08 19:23:53.390928 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.389246 2566 server.go:1019] "Starting client certificate rotation" Jul 08 19:23:53.390928 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.389340 2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 08 19:23:53.390928 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.389373 2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 08 19:23:53.391804 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.391791 2566 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 08 19:23:53.394482 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.394457 2566 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 08 19:23:53.402442 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.402426 2566 log.go:25] "Validated CRI v1 runtime API" Jul 08 19:23:53.404509 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.404495 2566 log.go:25] "Validated CRI v1 image API" Jul 08 19:23:53.405389 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.405378 2566 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 08 19:23:53.407700 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.407680 2566 fs.go:135] Filesystem UUIDs: map[488f9b59-b174-45c7-b50b-d436d8d944fa:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 8da920c2-f2ee-4f37-baf1-cb0df9b1af8a:/dev/nvme0n1p4] Jul 08 19:23:53.407779 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.407697 2566 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 08 19:23:53.413348 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.413234 2566 manager.go:216] Machine: {Timestamp:2026-07-08 19:23:53.412771085 +0000 UTC m=+0.330347748 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3096415 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21843e78c3c67745e2c3995c139c16 SystemUUID:ec21843e-78c3-c677-45e2-c3995c139c16 BootID:561ddb33-cd3f-45da-8227-52a5d5683dcc Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:77:ef:92:47:93 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:77:ef:92:47:93 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:96:ce:c1:e0:ca:07 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 08 19:23:53.413348 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.413340 2566 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 08 19:23:53.413457 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.413407 2566 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 08 19:23:53.415794 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.415770 2566 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 08 19:23:53.415919 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.415798 2566 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-195.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 08 19:23:53.416506 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.416493 2566 topology_manager.go:138] "Creating topology manager with none policy" Jul 08 19:23:53.416549 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.416508 2566 container_manager_linux.go:309] "Creating device plugin manager" Jul 08 19:23:53.416549 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.416520 2566 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 08 19:23:53.417203 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.417185 2566 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 08 19:23:53.418916 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.418906 2566 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 08 19:23:53.419622 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.419608 2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 08 19:23:53.421576 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.421563 2566 state_mem.go:36] "Initialized new in-memory state store" Jul 08 19:23:53.421699 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.421690 2566 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 08 19:23:53.421736 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.421717 2566 kubelet.go:486] "Attempting to sync node with API server" Jul 08 19:23:53.421736 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.421734 2566 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 08 19:23:53.421791 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.421751 2566 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 08 19:23:53.421791 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.421763 2566 kubelet.go:398] "Adding apiserver pod source" Jul 08 19:23:53.421791 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.421773 2566 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 08 19:23:53.422309 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.422292 2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 08 19:23:53.422359 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.422323 2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 08 19:23:53.427082 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.427057 2566 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 08 19:23:53.428934 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.428920 2566 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 08 19:23:53.429004 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.428950 2566 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 08 19:23:53.430034 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.430024 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 08 19:23:53.430070 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.430039 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 08 19:23:53.430070 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.430045 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 08 19:23:53.430070 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.430051 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 08 19:23:53.430070 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.430059 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 08 19:23:53.430070 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.430065 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 08 19:23:53.430199 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.430072 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 08 19:23:53.430199 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.430077 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 08 19:23:53.430199 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.430083 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 08 19:23:53.430199 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.430089 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 08 19:23:53.430199 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.430097 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 08 19:23:53.430199 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.430102 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 08 19:23:53.430862 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.430849 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 08 19:23:53.430900 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.430867 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 08 19:23:53.433016 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:23:53.432987 2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 08 19:23:53.433016 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:23:53.432994 2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-195.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 08 19:23:53.433150 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.433054 2566 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-195.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 08 19:23:53.433818 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.433807 2566 server.go:1268] "Started kubelet" Jul 08 19:23:53.433989 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.433939 2566 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 08 19:23:53.434078 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.434065 2566 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 08 19:23:53.434600 ip-10-0-142-195 systemd[1]: Started Kubernetes Kubelet. Jul 08 19:23:53.435195 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.435127 2566 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 08 19:23:53.435705 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.435692 2566 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 08 19:23:53.438633 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.438618 2566 server.go:310] "Adding debug handlers to kubelet server" Jul 08 19:23:53.439128 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:23:53.439113 2566 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 08 19:23:53.441173 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.441158 2566 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 08 19:23:53.441173 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.441168 2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 08 19:23:53.441601 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:23:53.441417 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-195.ec2.internal\" not found" Jul 08 19:23:53.441601 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.441564 2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fqxqn" Jul 08 19:23:53.441971 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.441952 2566 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 08 19:23:53.441971 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.441954 2566 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 08 19:23:53.442103 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.441978 2566 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 08 19:23:53.442103 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.442092 2566 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 08 19:23:53.442195 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.442108 2566 factory.go:55] Registering systemd factory Jul 08 19:23:53.442195 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.442115 2566 factory.go:223] Registration of the systemd container factory successfully Jul 08 19:23:53.442195 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.442127 2566 reconstruct.go:97] "Volume reconstruction finished" Jul 08 19:23:53.442195 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.442137 2566 reconciler.go:29] "Reconciler: start to sync state" Jul 08 19:23:53.442414 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.442398 2566 factory.go:153] Registering CRI-O factory Jul 08 19:23:53.442414 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.442417 2566 factory.go:223] Registration of the crio container factory successfully Jul 08 19:23:53.442566 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.442440 2566 factory.go:103] Registering Raw factory Jul 08 19:23:53.442566 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.442452 2566 manager.go:1196] Started watching for new ooms in manager Jul 08 19:23:53.442979 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.442965 2566 manager.go:349] Starting recovery of all containers Jul 08 19:23:53.447926 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.447904 2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fqxqn" Jul 08 19:23:53.448974 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:23:53.448950 2566 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 08 19:23:53.450017 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:23:53.448996 2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-195.ec2.internal.18c066f001018f9f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-195.ec2.internal,UID:ip-10-0-142-195.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-195.ec2.internal,},FirstTimestamp:2026-07-08 19:23:53.433780127 +0000 UTC m=+0.351356789,LastTimestamp:2026-07-08 19:23:53.433780127 +0000 UTC m=+0.351356789,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-195.ec2.internal,}" Jul 08 19:23:53.450141 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:23:53.450104 2566 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-195.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 08 19:23:53.453648 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.453625 2566 manager.go:354] Recovery completed Jul 08 19:23:53.456030 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:23:53.456005 2566 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Jul 08 19:23:53.459072 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.459060 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 19:23:53.460952 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.460936 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-195.ec2.internal" event="NodeHasSufficientMemory" Jul 08 19:23:53.461070 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.461059 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-195.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 19:23:53.461149 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.461140 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-195.ec2.internal" event="NodeHasSufficientPID" Jul 08 19:23:53.462421 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.462407 2566 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 08 19:23:53.462421 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.462420 2566 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 08 19:23:53.462421 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.462423 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 19:23:53.462658 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.462437 2566 state_mem.go:36] "Initialized new in-memory state store" Jul 08 19:23:53.463183 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.463164 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-195.ec2.internal" event="NodeHasSufficientMemory" Jul 08 19:23:53.463248 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.463197 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-195.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 19:23:53.463248 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.463210 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-195.ec2.internal" event="NodeHasSufficientPID" Jul 08 19:23:53.463539 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.463526 2566 policy_none.go:49] "None policy: Start" Jul 08 19:23:53.463590 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.463545 2566 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 08 19:23:53.463590 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.463559 2566 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 08 19:23:53.464555 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.464543 2566 policy_none.go:47] "Start" Jul 08 19:23:53.507128 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.506999 2566 manager.go:337] "Starting Device Plugin manager" Jul 08 19:23:53.507215 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:23:53.507144 2566 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 08 19:23:53.507215 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.507159 2566 server.go:87] "Starting device plugin registration server" Jul 08 19:23:53.507386 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.507375 2566 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 08 19:23:53.507422 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.507386 2566 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 08 19:23:53.507512 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.507496 2566 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 08 19:23:53.507594 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.507573 2566 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 08 19:23:53.507594 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.507587 2566 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 08 19:23:53.508085 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:23:53.508066 2566 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 08 19:23:53.508160 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:23:53.508111 2566 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-195.ec2.internal\" not found" Jul 08 19:23:53.584441 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.584417 2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 08 19:23:53.585496 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.585468 2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 08 19:23:53.585552 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.585515 2566 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 08 19:23:53.585552 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.585537 2566 kubelet.go:2448] "Starting kubelet main sync loop" Jul 08 19:23:53.585644 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:23:53.585568 2566 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 08 19:23:53.587991 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.587974 2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 19:23:53.608031 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.608020 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 19:23:53.608772 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.608752 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-195.ec2.internal" event="NodeHasSufficientMemory" Jul 08 19:23:53.608849 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.608786 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-195.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 19:23:53.608849 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.608801 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-195.ec2.internal" event="NodeHasSufficientPID" Jul 08 19:23:53.608849 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:53.608828 2566 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-195.ec2.internal" Jul 08 19:23:54.649382 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:54.649342 2566 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-195.ec2.internal" Jul 08 19:23:54.649382 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:23:54.649368 2566 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-195.ec2.internal\": node \"ip-10-0-142-195.ec2.internal\" not found" Jul 08 19:23:54.702969 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:23:54.702932 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-195.ec2.internal\" not found" Jul 08 19:23:54.718021 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:54.717993 2566 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-195.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-195.ec2.internal"] Jul 08 19:23:54.718075 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:54.718063 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 19:23:54.719835 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:54.719824 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-195.ec2.internal" event="NodeHasSufficientMemory" Jul 08 19:23:54.719898 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:54.719852 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-195.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 19:23:54.719898 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:54.719876 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-195.ec2.internal" event="NodeHasSufficientPID" Jul 08 19:23:54.721317 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:54.721306 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 19:23:54.721477 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:54.721465 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-195.ec2.internal" Jul 08 19:23:54.721512 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:54.721492 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 19:23:54.723054 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:54.723040 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-195.ec2.internal" event="NodeHasSufficientMemory" Jul 08 19:23:54.723125 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:54.723054 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-195.ec2.internal" event="NodeHasSufficientMemory" Jul 08 19:23:54.723125 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:54.723066 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-195.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 19:23:54.723125 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:54.723076 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-195.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 19:23:54.723125 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:54.723094 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-195.ec2.internal" event="NodeHasSufficientPID" Jul 08 19:23:54.723239 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:54.723080 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-195.ec2.internal" event="NodeHasSufficientPID" Jul 08 19:23:54.724341 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:54.724327 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-195.ec2.internal" Jul 08 19:23:54.724399 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:54.724350 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 19:23:54.725006 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:54.724991 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-195.ec2.internal" event="NodeHasSufficientMemory" Jul 08 19:23:54.725077 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:54.725020 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-195.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 19:23:54.725077 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:54.725037 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-195.ec2.internal" event="NodeHasSufficientPID" Jul 08 19:23:54.750409 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:23:54.750386 2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-195.ec2.internal\" not found" node="ip-10-0-142-195.ec2.internal" Jul 08 19:23:54.754755 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:23:54.754739 2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-195.ec2.internal\" not found" node="ip-10-0-142-195.ec2.internal" Jul 08 19:23:54.775304 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:54.775283 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3dd85bb084b2f8eec4b6dcb61f820279-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-195.ec2.internal\" (UID: \"3dd85bb084b2f8eec4b6dcb61f820279\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-195.ec2.internal" Jul 08 19:23:54.775379 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:54.775313 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3dd85bb084b2f8eec4b6dcb61f820279-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-195.ec2.internal\" (UID: \"3dd85bb084b2f8eec4b6dcb61f820279\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-195.ec2.internal" Jul 08 19:23:54.775379 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:54.775339 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/47fb7cb7c700e5809cbc3bdae3a4659d-config\") pod \"kube-apiserver-proxy-ip-10-0-142-195.ec2.internal\" (UID: \"47fb7cb7c700e5809cbc3bdae3a4659d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-195.ec2.internal" Jul 08 19:23:54.803903 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:23:54.803885 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-195.ec2.internal\" not found" Jul 08 19:23:54.875573 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:54.875555 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3dd85bb084b2f8eec4b6dcb61f820279-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-195.ec2.internal\" (UID: \"3dd85bb084b2f8eec4b6dcb61f820279\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-195.ec2.internal" Jul 08 19:23:54.875670 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:54.875599 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3dd85bb084b2f8eec4b6dcb61f820279-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-195.ec2.internal\" (UID: \"3dd85bb084b2f8eec4b6dcb61f820279\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-195.ec2.internal" Jul 08 19:23:54.875670 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:54.875645 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3dd85bb084b2f8eec4b6dcb61f820279-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-195.ec2.internal\" (UID: \"3dd85bb084b2f8eec4b6dcb61f820279\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-195.ec2.internal" Jul 08 19:23:54.875670 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:54.875668 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/47fb7cb7c700e5809cbc3bdae3a4659d-config\") pod \"kube-apiserver-proxy-ip-10-0-142-195.ec2.internal\" (UID: \"47fb7cb7c700e5809cbc3bdae3a4659d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-195.ec2.internal" Jul 08 19:23:54.875805 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:54.875693 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/47fb7cb7c700e5809cbc3bdae3a4659d-config\") pod \"kube-apiserver-proxy-ip-10-0-142-195.ec2.internal\" (UID: \"47fb7cb7c700e5809cbc3bdae3a4659d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-195.ec2.internal" Jul 08 19:23:54.875805 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:54.875719 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3dd85bb084b2f8eec4b6dcb61f820279-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-195.ec2.internal\" (UID: \"3dd85bb084b2f8eec4b6dcb61f820279\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-195.ec2.internal" Jul 08 19:23:54.904709 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:23:54.904663 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-195.ec2.internal\" not found" Jul 08 19:23:55.005436 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:23:55.005403 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-195.ec2.internal\" not found" Jul 08 19:23:55.053572 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:55.053544 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-195.ec2.internal" Jul 08 19:23:55.058504 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:55.058487 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-195.ec2.internal" Jul 08 19:23:55.105761 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:23:55.105734 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-195.ec2.internal\" not found" Jul 08 19:23:55.206314 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:23:55.206243 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-195.ec2.internal\" not found" Jul 08 19:23:55.306740 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:23:55.306708 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-195.ec2.internal\" not found" Jul 08 19:23:55.314097 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:55.314077 2566 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 19:23:55.407347 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:23:55.407320 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-195.ec2.internal\" not found" Jul 08 19:23:55.420482 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:55.420462 2566 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 08 19:23:55.420599 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:55.420583 2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 08 19:23:55.420644 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:55.420617 2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 08 19:23:55.481529 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:55.481452 2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-07 19:18:54 +0000 UTC" deadline="2028-01-22 17:42:57.448389001 +0000 UTC" Jul 08 19:23:55.481529 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:55.481481 2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13510h19m1.966911014s" Jul 08 19:23:55.508291 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:23:55.508261 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-195.ec2.internal\" not found" Jul 08 19:23:55.601551 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:55.601455 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dd85bb084b2f8eec4b6dcb61f820279.slice/crio-e8170f27e700a65b9d6bdd033e70495b457746d0db88d51fb5c93c2eb0e448fb WatchSource:0}: Error finding container e8170f27e700a65b9d6bdd033e70495b457746d0db88d51fb5c93c2eb0e448fb: Status 404 returned error can't find the container with id e8170f27e700a65b9d6bdd033e70495b457746d0db88d51fb5c93c2eb0e448fb Jul 08 19:23:55.601756 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:55.601736 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47fb7cb7c700e5809cbc3bdae3a4659d.slice/crio-4e81a6a0588f32fb01d8b0d7126bd7a001f08ef82b9866dcc213877bcff862dd WatchSource:0}: Error finding container 4e81a6a0588f32fb01d8b0d7126bd7a001f08ef82b9866dcc213877bcff862dd: Status 404 returned error can't find the container with id 4e81a6a0588f32fb01d8b0d7126bd7a001f08ef82b9866dcc213877bcff862dd Jul 08 19:23:55.606518 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:55.606503 2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 19:23:55.609136 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:23:55.609118 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-195.ec2.internal\" not found" Jul 08 19:23:55.620071 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:55.620027 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-195.ec2.internal" event={"ID":"3dd85bb084b2f8eec4b6dcb61f820279","Type":"ContainerStarted","Data":"e8170f27e700a65b9d6bdd033e70495b457746d0db88d51fb5c93c2eb0e448fb"} Jul 08 19:23:55.621016 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:55.620997 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-195.ec2.internal" event={"ID":"47fb7cb7c700e5809cbc3bdae3a4659d","Type":"ContainerStarted","Data":"4e81a6a0588f32fb01d8b0d7126bd7a001f08ef82b9866dcc213877bcff862dd"} Jul 08 19:23:55.709516 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:23:55.709484 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-195.ec2.internal\" not found" Jul 08 19:23:55.722773 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:55.722752 2566 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 19:23:55.810447 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:23:55.810364 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-195.ec2.internal\" not found" Jul 08 19:23:55.910854 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:23:55.910823 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-195.ec2.internal\" not found" Jul 08 19:23:56.011718 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:23:56.011685 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-195.ec2.internal\" not found" Jul 08 19:23:56.028252 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.028222 2566 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 19:23:56.073347 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.073273 2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-195.ec2.internal" Jul 08 19:23:56.082712 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.082688 2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 08 19:23:56.083679 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.083657 2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-195.ec2.internal" Jul 08 19:23:56.093018 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.092996 2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 08 19:23:56.455850 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.455818 2566 apiserver.go:52] "Watching apiserver" Jul 08 19:23:56.464786 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.464579 2566 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 08 19:23:56.465765 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.465731 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cwcd","openshift-cluster-node-tuning-operator/tuned-xb4w9","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-195.ec2.internal","openshift-multus/multus-additional-cni-plugins-srmfq","openshift-multus/multus-gz54s","openshift-multus/network-metrics-daemon-rfbjf","kube-system/kube-apiserver-proxy-ip-10-0-142-195.ec2.internal","openshift-dns/node-resolver-dghh8","openshift-image-registry/node-ca-6w52p","openshift-network-diagnostics/network-check-target-rs9hh","openshift-network-operator/iptables-alerter-bkjrq","openshift-ovn-kubernetes/ovnkube-node-cdp8n","kube-system/global-pull-secret-syncer-qdfqb","kube-system/konnectivity-agent-5kt6t"] Jul 08 19:23:56.468456 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.468372 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rs9hh" Jul 08 19:23:56.468572 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:23:56.468456 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rs9hh" podUID="60e1939e-8740-43f5-8d7b-056635c9f877" Jul 08 19:23:56.470543 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.470471 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdfqb" Jul 08 19:23:56.470690 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:23:56.470555 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdfqb" podUID="24b6882e-3f1e-4c28-b128-2b16f3478940" Jul 08 19:23:56.472645 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.472622 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5kt6t" Jul 08 19:23:56.473590 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.473569 2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 08 19:23:56.475222 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.474911 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-srmfq" Jul 08 19:23:56.475321 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.475298 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 08 19:23:56.475413 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.475386 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 08 19:23:56.475779 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.475757 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-hjlnm\"" Jul 08 19:23:56.476947 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.476926 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7nsz9\"" Jul 08 19:23:56.477159 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.477142 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 08 19:23:56.477630 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.477405 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 08 19:23:56.477630 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.477446 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 08 19:23:56.477630 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.477474 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 08 19:23:56.477630 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.477412 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 08 19:23:56.477630 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.477627 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gz54s" Jul 08 19:23:56.479563 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.479330 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfbjf" Jul 08 19:23:56.479563 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:23:56.479393 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rfbjf" podUID="55818509-538d-4733-9e10-eaaf841ef071" Jul 08 19:23:56.479850 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.479788 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 08 19:23:56.479977 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.479882 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-7cf7x\"" Jul 08 19:23:56.481952 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.481905 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6w52p" Jul 08 19:23:56.484023 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.484004 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 08 19:23:56.484163 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.484011 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dghh8" Jul 08 19:23:56.484566 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.484436 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vc5r\" (UniqueName: \"kubernetes.io/projected/60e1939e-8740-43f5-8d7b-056635c9f877-kube-api-access-5vc5r\") pod \"network-check-target-rs9hh\" (UID: \"60e1939e-8740-43f5-8d7b-056635c9f877\") " pod="openshift-network-diagnostics/network-check-target-rs9hh" Jul 08 19:23:56.484566 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.484472 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/24b6882e-3f1e-4c28-b128-2b16f3478940-dbus\") pod \"global-pull-secret-syncer-qdfqb\" (UID: \"24b6882e-3f1e-4c28-b128-2b16f3478940\") " pod="kube-system/global-pull-secret-syncer-qdfqb" Jul 08 19:23:56.484566 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.484502 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkjvq\" (UniqueName: \"kubernetes.io/projected/dbe0d11c-c9c5-4cdb-ad9b-08b142e7335f-kube-api-access-bkjvq\") pod \"multus-additional-cni-plugins-srmfq\" (UID: \"dbe0d11c-c9c5-4cdb-ad9b-08b142e7335f\") " pod="openshift-multus/multus-additional-cni-plugins-srmfq" Jul 08 19:23:56.484566 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.484529 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dbe0d11c-c9c5-4cdb-ad9b-08b142e7335f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-srmfq\" (UID: \"dbe0d11c-c9c5-4cdb-ad9b-08b142e7335f\") " pod="openshift-multus/multus-additional-cni-plugins-srmfq" Jul 08 19:23:56.484566 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.484554 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/42578ec4-91d1-4d4d-b9a1-12a559fee154-agent-certs\") pod \"konnectivity-agent-5kt6t\" (UID: \"42578ec4-91d1-4d4d-b9a1-12a559fee154\") " pod="kube-system/konnectivity-agent-5kt6t" Jul 08 19:23:56.484852 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.484577 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dbe0d11c-c9c5-4cdb-ad9b-08b142e7335f-system-cni-dir\") pod \"multus-additional-cni-plugins-srmfq\" (UID: \"dbe0d11c-c9c5-4cdb-ad9b-08b142e7335f\") " pod="openshift-multus/multus-additional-cni-plugins-srmfq" Jul 08 19:23:56.484852 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.484605 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dbe0d11c-c9c5-4cdb-ad9b-08b142e7335f-cnibin\") pod \"multus-additional-cni-plugins-srmfq\" (UID: \"dbe0d11c-c9c5-4cdb-ad9b-08b142e7335f\") " pod="openshift-multus/multus-additional-cni-plugins-srmfq" Jul 08 19:23:56.484852 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.484629 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dbe0d11c-c9c5-4cdb-ad9b-08b142e7335f-cni-binary-copy\") pod \"multus-additional-cni-plugins-srmfq\" (UID: \"dbe0d11c-c9c5-4cdb-ad9b-08b142e7335f\") " pod="openshift-multus/multus-additional-cni-plugins-srmfq" Jul 08 19:23:56.484852 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.484655 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dbe0d11c-c9c5-4cdb-ad9b-08b142e7335f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-srmfq\" (UID: \"dbe0d11c-c9c5-4cdb-ad9b-08b142e7335f\") " pod="openshift-multus/multus-additional-cni-plugins-srmfq" Jul 08 19:23:56.484852 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.484720 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24b6882e-3f1e-4c28-b128-2b16f3478940-original-pull-secret\") pod \"global-pull-secret-syncer-qdfqb\" (UID: \"24b6882e-3f1e-4c28-b128-2b16f3478940\") " pod="kube-system/global-pull-secret-syncer-qdfqb" Jul 08 19:23:56.484852 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.484728 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 08 19:23:56.484852 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.484767 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-jmcm6\"" Jul 08 19:23:56.484852 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.484771 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 08 19:23:56.485238 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.484940 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/42578ec4-91d1-4d4d-b9a1-12a559fee154-konnectivity-ca\") pod \"konnectivity-agent-5kt6t\" (UID: \"42578ec4-91d1-4d4d-b9a1-12a559fee154\") " pod="kube-system/konnectivity-agent-5kt6t" Jul 08 19:23:56.485238 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.484982 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dbe0d11c-c9c5-4cdb-ad9b-08b142e7335f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-srmfq\" (UID: \"dbe0d11c-c9c5-4cdb-ad9b-08b142e7335f\") " pod="openshift-multus/multus-additional-cni-plugins-srmfq" Jul 08 19:23:56.485238 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.485024 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/24b6882e-3f1e-4c28-b128-2b16f3478940-kubelet-config\") pod \"global-pull-secret-syncer-qdfqb\" (UID: \"24b6882e-3f1e-4c28-b128-2b16f3478940\") " pod="kube-system/global-pull-secret-syncer-qdfqb" Jul 08 19:23:56.485238 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.485077 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dbe0d11c-c9c5-4cdb-ad9b-08b142e7335f-os-release\") pod \"multus-additional-cni-plugins-srmfq\" (UID: \"dbe0d11c-c9c5-4cdb-ad9b-08b142e7335f\") " pod="openshift-multus/multus-additional-cni-plugins-srmfq" Jul 08 19:23:56.486504 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.486480 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ltrq6\"" Jul 08 19:23:56.486593 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.486551 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 08 19:23:56.486788 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.486769 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 08 19:23:56.486915 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.486880 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xb4w9" Jul 08 19:23:56.489003 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.488894 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cwcd" Jul 08 19:23:56.489003 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.488972 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-4rjmq\"" Jul 08 19:23:56.489142 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.489015 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 08 19:23:56.489493 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.489413 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 08 19:23:56.489768 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.489588 2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 08 19:23:56.491182 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.491165 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-nngx2\"" Jul 08 19:23:56.491287 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.491226 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 08 19:23:56.491287 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.491236 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 08 19:23:56.491457 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.491441 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 08 19:23:56.492350 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.491850 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bkjrq" Jul 08 19:23:56.494123 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.494103 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 08 19:23:56.494308 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.494291 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-kctj9\"" Jul 08 19:23:56.494372 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.494351 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 08 19:23:56.494443 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.494392 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 08 19:23:56.494731 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.494712 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-cdp8n" Jul 08 19:23:56.497002 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.496985 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 08 19:23:56.497411 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.497394 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 08 19:23:56.498458 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.498240 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 08 19:23:56.498458 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.498250 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 08 19:23:56.498458 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.498304 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 08 19:23:56.498458 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.498389 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-sblpj\"" Jul 08 19:23:56.498458 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.498242 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 08 19:23:56.507710 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.507689 2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-l2vjg" Jul 08 19:23:56.514958 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.514935 2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-l2vjg" Jul 08 19:23:56.574905 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.574873 2566 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 08 19:23:56.585723 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.585631 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/01c82975-c67d-4e46-b16a-f558032f5dc3-tmp\") pod \"tuned-xb4w9\" (UID: \"01c82975-c67d-4e46-b16a-f558032f5dc3\") " pod="openshift-cluster-node-tuning-operator/tuned-xb4w9" Jul 08 19:23:56.585723 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.585670 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/30762972-ab72-4cdd-a693-584f9c7cdde5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2cwcd\" (UID: \"30762972-ab72-4cdd-a693-584f9c7cdde5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cwcd" Jul 08 19:23:56.585723 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.585696 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/30762972-ab72-4cdd-a693-584f9c7cdde5-socket-dir\") pod \"aws-ebs-csi-driver-node-2cwcd\" (UID: \"30762972-ab72-4cdd-a693-584f9c7cdde5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cwcd" Jul 08 19:23:56.585723 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.585718 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/30762972-ab72-4cdd-a693-584f9c7cdde5-device-dir\") pod \"aws-ebs-csi-driver-node-2cwcd\" (UID: \"30762972-ab72-4cdd-a693-584f9c7cdde5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cwcd" Jul 08 19:23:56.586019 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.585741 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3af16262-f54e-4683-b76c-e1ccfdff4c3c-host-kubelet\") pod \"ovnkube-node-cdp8n\" (UID: \"3af16262-f54e-4683-b76c-e1ccfdff4c3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdp8n" Jul 08 19:23:56.586019 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.585789 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3af16262-f54e-4683-b76c-e1ccfdff4c3c-ovnkube-script-lib\") pod \"ovnkube-node-cdp8n\" (UID: \"3af16262-f54e-4683-b76c-e1ccfdff4c3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdp8n" Jul 08 19:23:56.586019 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.585833 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dbe0d11c-c9c5-4cdb-ad9b-08b142e7335f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-srmfq\" (UID: \"dbe0d11c-c9c5-4cdb-ad9b-08b142e7335f\") " pod="openshift-multus/multus-additional-cni-plugins-srmfq" Jul 08 19:23:56.586019 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.585903 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/01c82975-c67d-4e46-b16a-f558032f5dc3-etc-tuned\") pod \"tuned-xb4w9\" (UID: \"01c82975-c67d-4e46-b16a-f558032f5dc3\") " pod="openshift-cluster-node-tuning-operator/tuned-xb4w9" Jul 08 19:23:56.586019 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.585941 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bhrl\" (UniqueName: \"kubernetes.io/projected/30762972-ab72-4cdd-a693-584f9c7cdde5-kube-api-access-9bhrl\") pod \"aws-ebs-csi-driver-node-2cwcd\" (UID: \"30762972-ab72-4cdd-a693-584f9c7cdde5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cwcd" Jul 08 19:23:56.586019 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.585978 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3af16262-f54e-4683-b76c-e1ccfdff4c3c-var-lib-openvswitch\") pod \"ovnkube-node-cdp8n\" (UID: \"3af16262-f54e-4683-b76c-e1ccfdff4c3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdp8n" Jul 08 19:23:56.586019 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.586008 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3a7da2be-d237-4bfa-887d-0be7632bfdc0-os-release\") pod \"multus-gz54s\" (UID: \"3a7da2be-d237-4bfa-887d-0be7632bfdc0\") " pod="openshift-multus/multus-gz54s" Jul 08 19:23:56.586324 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.586033 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3a7da2be-d237-4bfa-887d-0be7632bfdc0-host-run-k8s-cni-cncf-io\") pod \"multus-gz54s\" (UID: \"3a7da2be-d237-4bfa-887d-0be7632bfdc0\") " pod="openshift-multus/multus-gz54s" Jul 08 19:23:56.586324 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.586055 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3a7da2be-d237-4bfa-887d-0be7632bfdc0-host-run-multus-certs\") pod \"multus-gz54s\" (UID: \"3a7da2be-d237-4bfa-887d-0be7632bfdc0\") " pod="openshift-multus/multus-gz54s" Jul 08 19:23:56.586324 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.586084 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24b6882e-3f1e-4c28-b128-2b16f3478940-original-pull-secret\") pod \"global-pull-secret-syncer-qdfqb\" (UID: \"24b6882e-3f1e-4c28-b128-2b16f3478940\") " pod="kube-system/global-pull-secret-syncer-qdfqb" Jul 08 19:23:56.586324 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.586108 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/01c82975-c67d-4e46-b16a-f558032f5dc3-etc-sysconfig\") pod \"tuned-xb4w9\" (UID: \"01c82975-c67d-4e46-b16a-f558032f5dc3\") " pod="openshift-cluster-node-tuning-operator/tuned-xb4w9" Jul 08 19:23:56.586324 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.586130 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/01c82975-c67d-4e46-b16a-f558032f5dc3-etc-systemd\") pod \"tuned-xb4w9\" (UID: \"01c82975-c67d-4e46-b16a-f558032f5dc3\") " pod="openshift-cluster-node-tuning-operator/tuned-xb4w9" Jul 08 19:23:56.586324 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.586151 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sksw2\" (UniqueName: \"kubernetes.io/projected/01c82975-c67d-4e46-b16a-f558032f5dc3-kube-api-access-sksw2\") pod \"tuned-xb4w9\" (UID: \"01c82975-c67d-4e46-b16a-f558032f5dc3\") " pod="openshift-cluster-node-tuning-operator/tuned-xb4w9" Jul 08 19:23:56.586324 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.586171 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3af16262-f54e-4683-b76c-e1ccfdff4c3c-host-cni-netd\") pod \"ovnkube-node-cdp8n\" (UID: \"3af16262-f54e-4683-b76c-e1ccfdff4c3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdp8n" Jul 08 19:23:56.586324 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.586192 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3af16262-f54e-4683-b76c-e1ccfdff4c3c-ovn-node-metrics-cert\") pod \"ovnkube-node-cdp8n\" (UID: \"3af16262-f54e-4683-b76c-e1ccfdff4c3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdp8n" Jul 08 19:23:56.586324 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:23:56.586202 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 19:23:56.586324 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.586220 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8b78ed73-731b-49fa-9220-4b1e45787806-host\") pod \"node-ca-6w52p\" (UID: \"8b78ed73-731b-49fa-9220-4b1e45787806\") " pod="openshift-image-registry/node-ca-6w52p" Jul 08 19:23:56.586324 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.586244 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/474e5893-2b36-4b57-b154-15b0f8b4f300-hosts-file\") pod \"node-resolver-dghh8\" (UID: \"474e5893-2b36-4b57-b154-15b0f8b4f300\") " pod="openshift-dns/node-resolver-dghh8" Jul 08 19:23:56.586324 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:23:56.586278 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24b6882e-3f1e-4c28-b128-2b16f3478940-original-pull-secret podName:24b6882e-3f1e-4c28-b128-2b16f3478940 nodeName:}" failed. No retries permitted until 2026-07-08 19:23:57.086257628 +0000 UTC m=+2.972447658 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/24b6882e-3f1e-4c28-b128-2b16f3478940-original-pull-secret") pod "global-pull-secret-syncer-qdfqb" (UID: "24b6882e-3f1e-4c28-b128-2b16f3478940") : object "kube-system"/"original-pull-secret" not registered Jul 08 19:23:56.586324 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.586301 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/01c82975-c67d-4e46-b16a-f558032f5dc3-etc-sysctl-d\") pod \"tuned-xb4w9\" (UID: \"01c82975-c67d-4e46-b16a-f558032f5dc3\") " pod="openshift-cluster-node-tuning-operator/tuned-xb4w9" Jul 08 19:23:56.586324 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.586329 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3af16262-f54e-4683-b76c-e1ccfdff4c3c-host-run-netns\") pod \"ovnkube-node-cdp8n\" (UID: \"3af16262-f54e-4683-b76c-e1ccfdff4c3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdp8n" Jul 08 19:23:56.586945 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.586353 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3af16262-f54e-4683-b76c-e1ccfdff4c3c-run-ovn\") pod \"ovnkube-node-cdp8n\" (UID: \"3af16262-f54e-4683-b76c-e1ccfdff4c3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdp8n" Jul 08 19:23:56.586945 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.586378 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a7da2be-d237-4bfa-887d-0be7632bfdc0-system-cni-dir\") pod \"multus-gz54s\" (UID: \"3a7da2be-d237-4bfa-887d-0be7632bfdc0\") " pod="openshift-multus/multus-gz54s" Jul 08 19:23:56.586945 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.586398 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8b78ed73-731b-49fa-9220-4b1e45787806-serviceca\") pod \"node-ca-6w52p\" (UID: \"8b78ed73-731b-49fa-9220-4b1e45787806\") " pod="openshift-image-registry/node-ca-6w52p" Jul 08 19:23:56.586945 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.586444 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/01c82975-c67d-4e46-b16a-f558032f5dc3-lib-modules\") pod \"tuned-xb4w9\" (UID: \"01c82975-c67d-4e46-b16a-f558032f5dc3\") " pod="openshift-cluster-node-tuning-operator/tuned-xb4w9" Jul 08 19:23:56.586945 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.586475 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/30762972-ab72-4cdd-a693-584f9c7cdde5-etc-selinux\") pod \"aws-ebs-csi-driver-node-2cwcd\" (UID: \"30762972-ab72-4cdd-a693-584f9c7cdde5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cwcd" Jul 08 19:23:56.586945 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.586442 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dbe0d11c-c9c5-4cdb-ad9b-08b142e7335f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-srmfq\" (UID: \"dbe0d11c-c9c5-4cdb-ad9b-08b142e7335f\") " pod="openshift-multus/multus-additional-cni-plugins-srmfq" Jul 08 19:23:56.586945 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.586510 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/30762972-ab72-4cdd-a693-584f9c7cdde5-sys-fs\") pod \"aws-ebs-csi-driver-node-2cwcd\" (UID: \"30762972-ab72-4cdd-a693-584f9c7cdde5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cwcd" Jul 08 19:23:56.586945 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.586538 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3af16262-f54e-4683-b76c-e1ccfdff4c3c-log-socket\") pod \"ovnkube-node-cdp8n\" (UID: \"3af16262-f54e-4683-b76c-e1ccfdff4c3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdp8n" Jul 08 19:23:56.586945 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.586580 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3a7da2be-d237-4bfa-887d-0be7632bfdc0-host-var-lib-cni-bin\") pod \"multus-gz54s\" (UID: \"3a7da2be-d237-4bfa-887d-0be7632bfdc0\") " pod="openshift-multus/multus-gz54s" Jul 08 19:23:56.586945 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.586605 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a7da2be-d237-4bfa-887d-0be7632bfdc0-host-var-lib-kubelet\") pod \"multus-gz54s\" (UID: \"3a7da2be-d237-4bfa-887d-0be7632bfdc0\") " pod="openshift-multus/multus-gz54s" Jul 08 19:23:56.586945 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.586635 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dbe0d11c-c9c5-4cdb-ad9b-08b142e7335f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-srmfq\" (UID: \"dbe0d11c-c9c5-4cdb-ad9b-08b142e7335f\") " pod="openshift-multus/multus-additional-cni-plugins-srmfq" Jul 08 19:23:56.586945 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.586657 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/01c82975-c67d-4e46-b16a-f558032f5dc3-etc-kubernetes\") pod \"tuned-xb4w9\" (UID: \"01c82975-c67d-4e46-b16a-f558032f5dc3\") " pod="openshift-cluster-node-tuning-operator/tuned-xb4w9" Jul 08 19:23:56.586945 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.586675 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/01c82975-c67d-4e46-b16a-f558032f5dc3-host\") pod \"tuned-xb4w9\" (UID: \"01c82975-c67d-4e46-b16a-f558032f5dc3\") " pod="openshift-cluster-node-tuning-operator/tuned-xb4w9" Jul 08 19:23:56.586945 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.586728 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3af16262-f54e-4683-b76c-e1ccfdff4c3c-run-openvswitch\") pod \"ovnkube-node-cdp8n\" (UID: \"3af16262-f54e-4683-b76c-e1ccfdff4c3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdp8n" Jul 08 19:23:56.586945 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.586753 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7g4k\" (UniqueName: \"kubernetes.io/projected/3af16262-f54e-4683-b76c-e1ccfdff4c3c-kube-api-access-x7g4k\") pod \"ovnkube-node-cdp8n\" (UID: \"3af16262-f54e-4683-b76c-e1ccfdff4c3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdp8n" Jul 08 19:23:56.586945 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.586777 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3a7da2be-d237-4bfa-887d-0be7632bfdc0-host-run-netns\") pod \"multus-gz54s\" (UID: \"3a7da2be-d237-4bfa-887d-0be7632bfdc0\") " pod="openshift-multus/multus-gz54s" Jul 08 19:23:56.587750 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.586803 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvggl\" (UniqueName: \"kubernetes.io/projected/3a7da2be-d237-4bfa-887d-0be7632bfdc0-kube-api-access-qvggl\") pod \"multus-gz54s\" (UID: \"3a7da2be-d237-4bfa-887d-0be7632bfdc0\") " pod="openshift-multus/multus-gz54s" Jul 08 19:23:56.587750 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.586833 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkjvq\" (UniqueName: \"kubernetes.io/projected/dbe0d11c-c9c5-4cdb-ad9b-08b142e7335f-kube-api-access-bkjvq\") pod \"multus-additional-cni-plugins-srmfq\" (UID: \"dbe0d11c-c9c5-4cdb-ad9b-08b142e7335f\") " pod="openshift-multus/multus-additional-cni-plugins-srmfq" Jul 08 19:23:56.587750 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.586902 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3af16262-f54e-4683-b76c-e1ccfdff4c3c-etc-openvswitch\") pod \"ovnkube-node-cdp8n\" (UID: \"3af16262-f54e-4683-b76c-e1ccfdff4c3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdp8n" Jul 08 19:23:56.587750 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.586929 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3a7da2be-d237-4bfa-887d-0be7632bfdc0-multus-socket-dir-parent\") pod \"multus-gz54s\" (UID: \"3a7da2be-d237-4bfa-887d-0be7632bfdc0\") " pod="openshift-multus/multus-gz54s" Jul 08 19:23:56.587750 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.586956 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/42578ec4-91d1-4d4d-b9a1-12a559fee154-agent-certs\") pod \"konnectivity-agent-5kt6t\" (UID: \"42578ec4-91d1-4d4d-b9a1-12a559fee154\") " pod="kube-system/konnectivity-agent-5kt6t" Jul 08 19:23:56.587750 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.586984 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dbe0d11c-c9c5-4cdb-ad9b-08b142e7335f-system-cni-dir\") pod \"multus-additional-cni-plugins-srmfq\" (UID: \"dbe0d11c-c9c5-4cdb-ad9b-08b142e7335f\") " pod="openshift-multus/multus-additional-cni-plugins-srmfq" Jul 08 19:23:56.587750 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.587012 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dbe0d11c-c9c5-4cdb-ad9b-08b142e7335f-cnibin\") pod \"multus-additional-cni-plugins-srmfq\" (UID: \"dbe0d11c-c9c5-4cdb-ad9b-08b142e7335f\") " pod="openshift-multus/multus-additional-cni-plugins-srmfq" Jul 08 19:23:56.587750 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.587037 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/01c82975-c67d-4e46-b16a-f558032f5dc3-var-lib-kubelet\") pod \"tuned-xb4w9\" (UID: \"01c82975-c67d-4e46-b16a-f558032f5dc3\") " pod="openshift-cluster-node-tuning-operator/tuned-xb4w9" Jul 08 19:23:56.587750 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.587061 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3af16262-f54e-4683-b76c-e1ccfdff4c3c-node-log\") pod \"ovnkube-node-cdp8n\" (UID: \"3af16262-f54e-4683-b76c-e1ccfdff4c3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdp8n" Jul 08 19:23:56.587750 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.587090 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3a7da2be-d237-4bfa-887d-0be7632bfdc0-cni-binary-copy\") pod \"multus-gz54s\" (UID: \"3a7da2be-d237-4bfa-887d-0be7632bfdc0\") " pod="openshift-multus/multus-gz54s" Jul 08 19:23:56.587750 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.587118 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3a7da2be-d237-4bfa-887d-0be7632bfdc0-host-var-lib-cni-multus\") pod \"multus-gz54s\" (UID: \"3a7da2be-d237-4bfa-887d-0be7632bfdc0\") " pod="openshift-multus/multus-gz54s" Jul 08 19:23:56.587750 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.587139 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dbe0d11c-c9c5-4cdb-ad9b-08b142e7335f-system-cni-dir\") pod \"multus-additional-cni-plugins-srmfq\" (UID: \"dbe0d11c-c9c5-4cdb-ad9b-08b142e7335f\") " pod="openshift-multus/multus-additional-cni-plugins-srmfq" Jul 08 19:23:56.587750 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.587154 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dbe0d11c-c9c5-4cdb-ad9b-08b142e7335f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-srmfq\" (UID: \"dbe0d11c-c9c5-4cdb-ad9b-08b142e7335f\") " pod="openshift-multus/multus-additional-cni-plugins-srmfq" Jul 08 19:23:56.587750 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.587208 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dbe0d11c-c9c5-4cdb-ad9b-08b142e7335f-cnibin\") pod \"multus-additional-cni-plugins-srmfq\" (UID: \"dbe0d11c-c9c5-4cdb-ad9b-08b142e7335f\") " pod="openshift-multus/multus-additional-cni-plugins-srmfq" Jul 08 19:23:56.587750 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.587156 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3a7da2be-d237-4bfa-887d-0be7632bfdc0-multus-daemon-config\") pod \"multus-gz54s\" (UID: \"3a7da2be-d237-4bfa-887d-0be7632bfdc0\") " pod="openshift-multus/multus-gz54s" Jul 08 19:23:56.587750 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.587244 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3af16262-f54e-4683-b76c-e1ccfdff4c3c-run-systemd\") pod \"ovnkube-node-cdp8n\" (UID: \"3af16262-f54e-4683-b76c-e1ccfdff4c3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdp8n" Jul 08 19:23:56.587750 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.587271 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3af16262-f54e-4683-b76c-e1ccfdff4c3c-env-overrides\") pod \"ovnkube-node-cdp8n\" (UID: \"3af16262-f54e-4683-b76c-e1ccfdff4c3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdp8n" Jul 08 19:23:56.588516 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.587300 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a7da2be-d237-4bfa-887d-0be7632bfdc0-multus-cni-dir\") pod \"multus-gz54s\" (UID: \"3a7da2be-d237-4bfa-887d-0be7632bfdc0\") " pod="openshift-multus/multus-gz54s" Jul 08 19:23:56.588516 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.587323 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a7da2be-d237-4bfa-887d-0be7632bfdc0-etc-kubernetes\") pod \"multus-gz54s\" (UID: \"3a7da2be-d237-4bfa-887d-0be7632bfdc0\") " pod="openshift-multus/multus-gz54s" Jul 08 19:23:56.588516 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.587347 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65h2n\" (UniqueName: \"kubernetes.io/projected/8b78ed73-731b-49fa-9220-4b1e45787806-kube-api-access-65h2n\") pod \"node-ca-6w52p\" (UID: \"8b78ed73-731b-49fa-9220-4b1e45787806\") " pod="openshift-image-registry/node-ca-6w52p" Jul 08 19:23:56.588516 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.587370 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz2zp\" (UniqueName: \"kubernetes.io/projected/bbe4cd13-7412-42f1-9035-865ce20ba987-kube-api-access-xz2zp\") pod \"iptables-alerter-bkjrq\" (UID: \"bbe4cd13-7412-42f1-9035-865ce20ba987\") " pod="openshift-network-operator/iptables-alerter-bkjrq" Jul 08 19:23:56.588516 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.587397 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5psp\" (UniqueName: \"kubernetes.io/projected/474e5893-2b36-4b57-b154-15b0f8b4f300-kube-api-access-v5psp\") pod \"node-resolver-dghh8\" (UID: \"474e5893-2b36-4b57-b154-15b0f8b4f300\") " pod="openshift-dns/node-resolver-dghh8" Jul 08 19:23:56.588516 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.587448 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3af16262-f54e-4683-b76c-e1ccfdff4c3c-host-run-ovn-kubernetes\") pod \"ovnkube-node-cdp8n\" (UID: \"3af16262-f54e-4683-b76c-e1ccfdff4c3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdp8n" Jul 08 19:23:56.588516 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.587495 2566 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 08 19:23:56.588516 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.587503 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55818509-538d-4733-9e10-eaaf841ef071-metrics-certs\") pod \"network-metrics-daemon-rfbjf\" (UID: \"55818509-538d-4733-9e10-eaaf841ef071\") " pod="openshift-multus/network-metrics-daemon-rfbjf" Jul 08 19:23:56.588516 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.587536 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bbe4cd13-7412-42f1-9035-865ce20ba987-iptables-alerter-script\") pod \"iptables-alerter-bkjrq\" (UID: \"bbe4cd13-7412-42f1-9035-865ce20ba987\") " pod="openshift-network-operator/iptables-alerter-bkjrq" Jul 08 19:23:56.588516 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.587590 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vc5r\" (UniqueName: \"kubernetes.io/projected/60e1939e-8740-43f5-8d7b-056635c9f877-kube-api-access-5vc5r\") pod \"network-check-target-rs9hh\" (UID: \"60e1939e-8740-43f5-8d7b-056635c9f877\") " pod="openshift-network-diagnostics/network-check-target-rs9hh" Jul 08 19:23:56.588516 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.587654 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/24b6882e-3f1e-4c28-b128-2b16f3478940-dbus\") pod \"global-pull-secret-syncer-qdfqb\" (UID: \"24b6882e-3f1e-4c28-b128-2b16f3478940\") " pod="kube-system/global-pull-secret-syncer-qdfqb" Jul 08 19:23:56.588516 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.587681 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/01c82975-c67d-4e46-b16a-f558032f5dc3-etc-sysctl-conf\") pod \"tuned-xb4w9\" (UID: \"01c82975-c67d-4e46-b16a-f558032f5dc3\") " pod="openshift-cluster-node-tuning-operator/tuned-xb4w9" Jul 08 19:23:56.588516 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.587725 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/01c82975-c67d-4e46-b16a-f558032f5dc3-sys\") pod \"tuned-xb4w9\" (UID: \"01c82975-c67d-4e46-b16a-f558032f5dc3\") " pod="openshift-cluster-node-tuning-operator/tuned-xb4w9" Jul 08 19:23:56.588516 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.587749 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/30762972-ab72-4cdd-a693-584f9c7cdde5-registration-dir\") pod \"aws-ebs-csi-driver-node-2cwcd\" (UID: \"30762972-ab72-4cdd-a693-584f9c7cdde5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cwcd" Jul 08 19:23:56.588516 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.587796 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3af16262-f54e-4683-b76c-e1ccfdff4c3c-host-slash\") pod \"ovnkube-node-cdp8n\" (UID: \"3af16262-f54e-4683-b76c-e1ccfdff4c3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdp8n" Jul 08 19:23:56.588516 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.587825 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3a7da2be-d237-4bfa-887d-0be7632bfdc0-hostroot\") pod \"multus-gz54s\" (UID: \"3a7da2be-d237-4bfa-887d-0be7632bfdc0\") " pod="openshift-multus/multus-gz54s" Jul 08 19:23:56.588516 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.587848 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bbe4cd13-7412-42f1-9035-865ce20ba987-host-slash\") pod \"iptables-alerter-bkjrq\" (UID: \"bbe4cd13-7412-42f1-9035-865ce20ba987\") " pod="openshift-network-operator/iptables-alerter-bkjrq" Jul 08 19:23:56.589158 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.587931 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dbe0d11c-c9c5-4cdb-ad9b-08b142e7335f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-srmfq\" (UID: \"dbe0d11c-c9c5-4cdb-ad9b-08b142e7335f\") " pod="openshift-multus/multus-additional-cni-plugins-srmfq" Jul 08 19:23:56.589158 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.587973 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dbe0d11c-c9c5-4cdb-ad9b-08b142e7335f-cni-binary-copy\") pod \"multus-additional-cni-plugins-srmfq\" (UID: \"dbe0d11c-c9c5-4cdb-ad9b-08b142e7335f\") " pod="openshift-multus/multus-additional-cni-plugins-srmfq" Jul 08 19:23:56.589158 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.588022 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3a7da2be-d237-4bfa-887d-0be7632bfdc0-multus-conf-dir\") pod \"multus-gz54s\" (UID: \"3a7da2be-d237-4bfa-887d-0be7632bfdc0\") " pod="openshift-multus/multus-gz54s" Jul 08 19:23:56.589158 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.588036 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/24b6882e-3f1e-4c28-b128-2b16f3478940-dbus\") pod \"global-pull-secret-syncer-qdfqb\" (UID: \"24b6882e-3f1e-4c28-b128-2b16f3478940\") " pod="kube-system/global-pull-secret-syncer-qdfqb" Jul 08 19:23:56.589158 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.588053 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/42578ec4-91d1-4d4d-b9a1-12a559fee154-konnectivity-ca\") pod \"konnectivity-agent-5kt6t\" (UID: \"42578ec4-91d1-4d4d-b9a1-12a559fee154\") " pod="kube-system/konnectivity-agent-5kt6t" Jul 08 19:23:56.589158 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.588081 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dbe0d11c-c9c5-4cdb-ad9b-08b142e7335f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-srmfq\" (UID: \"dbe0d11c-c9c5-4cdb-ad9b-08b142e7335f\") " pod="openshift-multus/multus-additional-cni-plugins-srmfq" Jul 08 19:23:56.589158 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.588088 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/01c82975-c67d-4e46-b16a-f558032f5dc3-etc-modprobe-d\") pod \"tuned-xb4w9\" (UID: \"01c82975-c67d-4e46-b16a-f558032f5dc3\") " pod="openshift-cluster-node-tuning-operator/tuned-xb4w9" Jul 08 19:23:56.589158 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.588121 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3af16262-f54e-4683-b76c-e1ccfdff4c3c-host-cni-bin\") pod \"ovnkube-node-cdp8n\" (UID: \"3af16262-f54e-4683-b76c-e1ccfdff4c3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdp8n" Jul 08 19:23:56.589158 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.588145 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3af16262-f54e-4683-b76c-e1ccfdff4c3c-ovnkube-config\") pod \"ovnkube-node-cdp8n\" (UID: \"3af16262-f54e-4683-b76c-e1ccfdff4c3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdp8n" Jul 08 19:23:56.589158 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.588183 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/24b6882e-3f1e-4c28-b128-2b16f3478940-kubelet-config\") pod \"global-pull-secret-syncer-qdfqb\" (UID: \"24b6882e-3f1e-4c28-b128-2b16f3478940\") " pod="kube-system/global-pull-secret-syncer-qdfqb" Jul 08 19:23:56.589158 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.588209 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dbe0d11c-c9c5-4cdb-ad9b-08b142e7335f-os-release\") pod \"multus-additional-cni-plugins-srmfq\" (UID: \"dbe0d11c-c9c5-4cdb-ad9b-08b142e7335f\") " pod="openshift-multus/multus-additional-cni-plugins-srmfq" Jul 08 19:23:56.589158 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.588271 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dbe0d11c-c9c5-4cdb-ad9b-08b142e7335f-os-release\") pod \"multus-additional-cni-plugins-srmfq\" (UID: \"dbe0d11c-c9c5-4cdb-ad9b-08b142e7335f\") " pod="openshift-multus/multus-additional-cni-plugins-srmfq" Jul 08 19:23:56.589158 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.588266 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/24b6882e-3f1e-4c28-b128-2b16f3478940-kubelet-config\") pod \"global-pull-secret-syncer-qdfqb\" (UID: \"24b6882e-3f1e-4c28-b128-2b16f3478940\") " pod="kube-system/global-pull-secret-syncer-qdfqb" Jul 08 19:23:56.589158 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.588309 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/01c82975-c67d-4e46-b16a-f558032f5dc3-run\") pod \"tuned-xb4w9\" (UID: \"01c82975-c67d-4e46-b16a-f558032f5dc3\") " pod="openshift-cluster-node-tuning-operator/tuned-xb4w9" Jul 08 19:23:56.589158 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.588337 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3af16262-f54e-4683-b76c-e1ccfdff4c3c-systemd-units\") pod \"ovnkube-node-cdp8n\" (UID: \"3af16262-f54e-4683-b76c-e1ccfdff4c3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdp8n" Jul 08 19:23:56.589158 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.588369 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3af16262-f54e-4683-b76c-e1ccfdff4c3c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cdp8n\" (UID: \"3af16262-f54e-4683-b76c-e1ccfdff4c3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdp8n" Jul 08 19:23:56.589158 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.588393 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3a7da2be-d237-4bfa-887d-0be7632bfdc0-cnibin\") pod \"multus-gz54s\" (UID: \"3a7da2be-d237-4bfa-887d-0be7632bfdc0\") " pod="openshift-multus/multus-gz54s" Jul 08 19:23:56.589845 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.588425 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw5sx\" (UniqueName: \"kubernetes.io/projected/55818509-538d-4733-9e10-eaaf841ef071-kube-api-access-qw5sx\") pod \"network-metrics-daemon-rfbjf\" (UID: \"55818509-538d-4733-9e10-eaaf841ef071\") " pod="openshift-multus/network-metrics-daemon-rfbjf" Jul 08 19:23:56.589845 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.588448 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/474e5893-2b36-4b57-b154-15b0f8b4f300-tmp-dir\") pod \"node-resolver-dghh8\" (UID: \"474e5893-2b36-4b57-b154-15b0f8b4f300\") " pod="openshift-dns/node-resolver-dghh8" Jul 08 19:23:56.589845 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.588525 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dbe0d11c-c9c5-4cdb-ad9b-08b142e7335f-cni-binary-copy\") pod \"multus-additional-cni-plugins-srmfq\" (UID: \"dbe0d11c-c9c5-4cdb-ad9b-08b142e7335f\") " pod="openshift-multus/multus-additional-cni-plugins-srmfq" Jul 08 19:23:56.589845 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.588736 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/42578ec4-91d1-4d4d-b9a1-12a559fee154-konnectivity-ca\") pod \"konnectivity-agent-5kt6t\" (UID: \"42578ec4-91d1-4d4d-b9a1-12a559fee154\") " pod="kube-system/konnectivity-agent-5kt6t" Jul 08 19:23:56.592193 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.592166 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/42578ec4-91d1-4d4d-b9a1-12a559fee154-agent-certs\") pod \"konnectivity-agent-5kt6t\" (UID: \"42578ec4-91d1-4d4d-b9a1-12a559fee154\") " pod="kube-system/konnectivity-agent-5kt6t" Jul 08 19:23:56.593305 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:23:56.593286 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 19:23:56.593393 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:23:56.593312 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 19:23:56.593393 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:23:56.593326 2566 projected.go:196] Error preparing data for projected volume kube-api-access-5vc5r for pod openshift-network-diagnostics/network-check-target-rs9hh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 19:23:56.593496 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:23:56.593420 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60e1939e-8740-43f5-8d7b-056635c9f877-kube-api-access-5vc5r podName:60e1939e-8740-43f5-8d7b-056635c9f877 nodeName:}" failed. No retries permitted until 2026-07-08 19:23:57.093403419 +0000 UTC m=+2.979593461 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-5vc5r" (UniqueName: "kubernetes.io/projected/60e1939e-8740-43f5-8d7b-056635c9f877-kube-api-access-5vc5r") pod "network-check-target-rs9hh" (UID: "60e1939e-8740-43f5-8d7b-056635c9f877") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 19:23:56.596068 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.596036 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkjvq\" (UniqueName: \"kubernetes.io/projected/dbe0d11c-c9c5-4cdb-ad9b-08b142e7335f-kube-api-access-bkjvq\") pod \"multus-additional-cni-plugins-srmfq\" (UID: \"dbe0d11c-c9c5-4cdb-ad9b-08b142e7335f\") " pod="openshift-multus/multus-additional-cni-plugins-srmfq" Jul 08 19:23:56.689556 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.689512 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3af16262-f54e-4683-b76c-e1ccfdff4c3c-systemd-units\") pod \"ovnkube-node-cdp8n\" (UID: \"3af16262-f54e-4683-b76c-e1ccfdff4c3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdp8n" Jul 08 19:23:56.689556 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.689552 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3af16262-f54e-4683-b76c-e1ccfdff4c3c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cdp8n\" (UID: \"3af16262-f54e-4683-b76c-e1ccfdff4c3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdp8n" Jul 08 19:23:56.689794 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.689577 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3a7da2be-d237-4bfa-887d-0be7632bfdc0-cnibin\") pod \"multus-gz54s\" (UID: \"3a7da2be-d237-4bfa-887d-0be7632bfdc0\") " pod="openshift-multus/multus-gz54s" Jul 08 19:23:56.689794 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.689594 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw5sx\" (UniqueName: \"kubernetes.io/projected/55818509-538d-4733-9e10-eaaf841ef071-kube-api-access-qw5sx\") pod \"network-metrics-daemon-rfbjf\" (UID: \"55818509-538d-4733-9e10-eaaf841ef071\") " pod="openshift-multus/network-metrics-daemon-rfbjf" Jul 08 19:23:56.689794 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.689619 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3af16262-f54e-4683-b76c-e1ccfdff4c3c-systemd-units\") pod \"ovnkube-node-cdp8n\" (UID: \"3af16262-f54e-4683-b76c-e1ccfdff4c3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdp8n" Jul 08 19:23:56.689794 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.689646 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3af16262-f54e-4683-b76c-e1ccfdff4c3c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cdp8n\" (UID: \"3af16262-f54e-4683-b76c-e1ccfdff4c3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdp8n" Jul 08 19:23:56.689794 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.689677 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3a7da2be-d237-4bfa-887d-0be7632bfdc0-cnibin\") pod \"multus-gz54s\" (UID: \"3a7da2be-d237-4bfa-887d-0be7632bfdc0\") " pod="openshift-multus/multus-gz54s" Jul 08 19:23:56.689794 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.689697 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/474e5893-2b36-4b57-b154-15b0f8b4f300-tmp-dir\") pod \"node-resolver-dghh8\" (UID: \"474e5893-2b36-4b57-b154-15b0f8b4f300\") " pod="openshift-dns/node-resolver-dghh8" Jul 08 19:23:56.689794 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.689729 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/01c82975-c67d-4e46-b16a-f558032f5dc3-tmp\") pod \"tuned-xb4w9\" (UID: \"01c82975-c67d-4e46-b16a-f558032f5dc3\") " pod="openshift-cluster-node-tuning-operator/tuned-xb4w9" Jul 08 19:23:56.689794 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.689756 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/30762972-ab72-4cdd-a693-584f9c7cdde5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2cwcd\" (UID: \"30762972-ab72-4cdd-a693-584f9c7cdde5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cwcd" Jul 08 19:23:56.689794 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.689783 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/30762972-ab72-4cdd-a693-584f9c7cdde5-socket-dir\") pod \"aws-ebs-csi-driver-node-2cwcd\" (UID: \"30762972-ab72-4cdd-a693-584f9c7cdde5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cwcd" Jul 08 19:23:56.690269 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.689807 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/30762972-ab72-4cdd-a693-584f9c7cdde5-device-dir\") pod \"aws-ebs-csi-driver-node-2cwcd\" (UID: \"30762972-ab72-4cdd-a693-584f9c7cdde5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cwcd" Jul 08 19:23:56.690269 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.689889 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/30762972-ab72-4cdd-a693-584f9c7cdde5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2cwcd\" (UID: \"30762972-ab72-4cdd-a693-584f9c7cdde5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cwcd" Jul 08 19:23:56.690269 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.689919 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/30762972-ab72-4cdd-a693-584f9c7cdde5-device-dir\") pod \"aws-ebs-csi-driver-node-2cwcd\" (UID: \"30762972-ab72-4cdd-a693-584f9c7cdde5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cwcd" Jul 08 19:23:56.690269 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.689833 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3af16262-f54e-4683-b76c-e1ccfdff4c3c-host-kubelet\") pod \"ovnkube-node-cdp8n\" (UID: \"3af16262-f54e-4683-b76c-e1ccfdff4c3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdp8n" Jul 08 19:23:56.690269 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.689945 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/30762972-ab72-4cdd-a693-584f9c7cdde5-socket-dir\") pod \"aws-ebs-csi-driver-node-2cwcd\" (UID: \"30762972-ab72-4cdd-a693-584f9c7cdde5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cwcd" Jul 08 19:23:56.690269 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.689976 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3af16262-f54e-4683-b76c-e1ccfdff4c3c-ovnkube-script-lib\") pod \"ovnkube-node-cdp8n\" (UID: \"3af16262-f54e-4683-b76c-e1ccfdff4c3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdp8n" Jul 08 19:23:56.690269 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.689996 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3af16262-f54e-4683-b76c-e1ccfdff4c3c-host-kubelet\") pod \"ovnkube-node-cdp8n\" (UID: \"3af16262-f54e-4683-b76c-e1ccfdff4c3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdp8n" Jul 08 19:23:56.690269 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.690008 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/01c82975-c67d-4e46-b16a-f558032f5dc3-etc-tuned\") pod \"tuned-xb4w9\" (UID: \"01c82975-c67d-4e46-b16a-f558032f5dc3\") " pod="openshift-cluster-node-tuning-operator/tuned-xb4w9" Jul 08 19:23:56.690269 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.690035 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bhrl\" (UniqueName: \"kubernetes.io/projected/30762972-ab72-4cdd-a693-584f9c7cdde5-kube-api-access-9bhrl\") pod \"aws-ebs-csi-driver-node-2cwcd\" (UID: \"30762972-ab72-4cdd-a693-584f9c7cdde5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cwcd" Jul 08 19:23:56.690269 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.690061 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3af16262-f54e-4683-b76c-e1ccfdff4c3c-var-lib-openvswitch\") pod \"ovnkube-node-cdp8n\" (UID: \"3af16262-f54e-4683-b76c-e1ccfdff4c3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdp8n" Jul 08 19:23:56.690269 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.690087 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3a7da2be-d237-4bfa-887d-0be7632bfdc0-os-release\") pod \"multus-gz54s\" (UID: \"3a7da2be-d237-4bfa-887d-0be7632bfdc0\") " pod="openshift-multus/multus-gz54s" Jul 08 19:23:56.690269 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.690111 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3a7da2be-d237-4bfa-887d-0be7632bfdc0-host-run-k8s-cni-cncf-io\") pod \"multus-gz54s\" (UID: \"3a7da2be-d237-4bfa-887d-0be7632bfdc0\") " pod="openshift-multus/multus-gz54s" Jul 08 19:23:56.690269 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.690110 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/474e5893-2b36-4b57-b154-15b0f8b4f300-tmp-dir\") pod \"node-resolver-dghh8\" (UID: \"474e5893-2b36-4b57-b154-15b0f8b4f300\") " pod="openshift-dns/node-resolver-dghh8" Jul 08 19:23:56.690269 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.690139 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3a7da2be-d237-4bfa-887d-0be7632bfdc0-host-run-multus-certs\") pod \"multus-gz54s\" (UID: \"3a7da2be-d237-4bfa-887d-0be7632bfdc0\") " pod="openshift-multus/multus-gz54s" Jul 08 19:23:56.690269 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.690153 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3af16262-f54e-4683-b76c-e1ccfdff4c3c-var-lib-openvswitch\") pod \"ovnkube-node-cdp8n\" (UID: \"3af16262-f54e-4683-b76c-e1ccfdff4c3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdp8n" Jul 08 19:23:56.690269 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.690182 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/01c82975-c67d-4e46-b16a-f558032f5dc3-etc-sysconfig\") pod \"tuned-xb4w9\" (UID: \"01c82975-c67d-4e46-b16a-f558032f5dc3\") " pod="openshift-cluster-node-tuning-operator/tuned-xb4w9" Jul 08 19:23:56.690269 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.690205 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3a7da2be-d237-4bfa-887d-0be7632bfdc0-host-run-k8s-cni-cncf-io\") pod \"multus-gz54s\" (UID: \"3a7da2be-d237-4bfa-887d-0be7632bfdc0\") " pod="openshift-multus/multus-gz54s" Jul 08 19:23:56.691087 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.690208 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/01c82975-c67d-4e46-b16a-f558032f5dc3-etc-systemd\") pod \"tuned-xb4w9\" (UID: \"01c82975-c67d-4e46-b16a-f558032f5dc3\") " pod="openshift-cluster-node-tuning-operator/tuned-xb4w9" Jul 08 19:23:56.691087 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.690241 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3a7da2be-d237-4bfa-887d-0be7632bfdc0-os-release\") pod \"multus-gz54s\" (UID: \"3a7da2be-d237-4bfa-887d-0be7632bfdc0\") " pod="openshift-multus/multus-gz54s" Jul 08 19:23:56.691087 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.690246 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sksw2\" (UniqueName: \"kubernetes.io/projected/01c82975-c67d-4e46-b16a-f558032f5dc3-kube-api-access-sksw2\") pod \"tuned-xb4w9\" (UID: \"01c82975-c67d-4e46-b16a-f558032f5dc3\") " pod="openshift-cluster-node-tuning-operator/tuned-xb4w9" Jul 08 19:23:56.691087 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.690262 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/01c82975-c67d-4e46-b16a-f558032f5dc3-etc-systemd\") pod \"tuned-xb4w9\" (UID: \"01c82975-c67d-4e46-b16a-f558032f5dc3\") " pod="openshift-cluster-node-tuning-operator/tuned-xb4w9" Jul 08 19:23:56.691087 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.690271 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3af16262-f54e-4683-b76c-e1ccfdff4c3c-host-cni-netd\") pod \"ovnkube-node-cdp8n\" (UID: \"3af16262-f54e-4683-b76c-e1ccfdff4c3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdp8n" Jul 08 19:23:56.691087 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.690292 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3a7da2be-d237-4bfa-887d-0be7632bfdc0-host-run-multus-certs\") pod \"multus-gz54s\" (UID: \"3a7da2be-d237-4bfa-887d-0be7632bfdc0\") " pod="openshift-multus/multus-gz54s" Jul 08 19:23:56.691087 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.690321 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3af16262-f54e-4683-b76c-e1ccfdff4c3c-host-cni-netd\") pod \"ovnkube-node-cdp8n\" (UID: \"3af16262-f54e-4683-b76c-e1ccfdff4c3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdp8n" Jul 08 19:23:56.691087 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.690335 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/01c82975-c67d-4e46-b16a-f558032f5dc3-etc-sysconfig\") pod \"tuned-xb4w9\" (UID: \"01c82975-c67d-4e46-b16a-f558032f5dc3\") " pod="openshift-cluster-node-tuning-operator/tuned-xb4w9" Jul 08 19:23:56.691087 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.690397 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3af16262-f54e-4683-b76c-e1ccfdff4c3c-ovn-node-metrics-cert\") pod \"ovnkube-node-cdp8n\" (UID: \"3af16262-f54e-4683-b76c-e1ccfdff4c3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdp8n" Jul 08 19:23:56.691087 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.690432 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8b78ed73-731b-49fa-9220-4b1e45787806-host\") pod \"node-ca-6w52p\" (UID: \"8b78ed73-731b-49fa-9220-4b1e45787806\") " pod="openshift-image-registry/node-ca-6w52p" Jul 08 19:23:56.691087 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.690455 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/474e5893-2b36-4b57-b154-15b0f8b4f300-hosts-file\") pod \"node-resolver-dghh8\" (UID: \"474e5893-2b36-4b57-b154-15b0f8b4f300\") " pod="openshift-dns/node-resolver-dghh8" Jul 08 19:23:56.691087 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.690478 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/01c82975-c67d-4e46-b16a-f558032f5dc3-etc-sysctl-d\") pod \"tuned-xb4w9\" (UID: \"01c82975-c67d-4e46-b16a-f558032f5dc3\") " pod="openshift-cluster-node-tuning-operator/tuned-xb4w9" Jul 08 19:23:56.691087 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.690510 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3af16262-f54e-4683-b76c-e1ccfdff4c3c-host-run-netns\") pod \"ovnkube-node-cdp8n\" (UID: \"3af16262-f54e-4683-b76c-e1ccfdff4c3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdp8n" Jul 08 19:23:56.691087 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.690533 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3af16262-f54e-4683-b76c-e1ccfdff4c3c-run-ovn\") pod \"ovnkube-node-cdp8n\" (UID: \"3af16262-f54e-4683-b76c-e1ccfdff4c3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdp8n" Jul 08 19:23:56.691087 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.690543 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3af16262-f54e-4683-b76c-e1ccfdff4c3c-ovnkube-script-lib\") pod \"ovnkube-node-cdp8n\" (UID: \"3af16262-f54e-4683-b76c-e1ccfdff4c3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdp8n" Jul 08 19:23:56.691087 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.690544 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/474e5893-2b36-4b57-b154-15b0f8b4f300-hosts-file\") pod \"node-resolver-dghh8\" (UID: \"474e5893-2b36-4b57-b154-15b0f8b4f300\") " pod="openshift-dns/node-resolver-dghh8" Jul 08 19:23:56.691087 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.690555 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a7da2be-d237-4bfa-887d-0be7632bfdc0-system-cni-dir\") pod \"multus-gz54s\" (UID: \"3a7da2be-d237-4bfa-887d-0be7632bfdc0\") " pod="openshift-multus/multus-gz54s" Jul 08 19:23:56.691087 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.690579 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8b78ed73-731b-49fa-9220-4b1e45787806-host\") pod \"node-ca-6w52p\" (UID: \"8b78ed73-731b-49fa-9220-4b1e45787806\") " pod="openshift-image-registry/node-ca-6w52p" Jul 08 19:23:56.692005 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.690589 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8b78ed73-731b-49fa-9220-4b1e45787806-serviceca\") pod \"node-ca-6w52p\" (UID: \"8b78ed73-731b-49fa-9220-4b1e45787806\") " pod="openshift-image-registry/node-ca-6w52p" Jul 08 19:23:56.692005 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.690617 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/01c82975-c67d-4e46-b16a-f558032f5dc3-lib-modules\") pod \"tuned-xb4w9\" (UID: \"01c82975-c67d-4e46-b16a-f558032f5dc3\") " pod="openshift-cluster-node-tuning-operator/tuned-xb4w9" Jul 08 19:23:56.692005 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.690628 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3af16262-f54e-4683-b76c-e1ccfdff4c3c-host-run-netns\") pod \"ovnkube-node-cdp8n\" (UID: \"3af16262-f54e-4683-b76c-e1ccfdff4c3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdp8n" Jul 08 19:23:56.692005 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.690643 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/30762972-ab72-4cdd-a693-584f9c7cdde5-etc-selinux\") pod \"aws-ebs-csi-driver-node-2cwcd\" (UID: \"30762972-ab72-4cdd-a693-584f9c7cdde5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cwcd" Jul 08 19:23:56.692005 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.690671 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/30762972-ab72-4cdd-a693-584f9c7cdde5-sys-fs\") pod \"aws-ebs-csi-driver-node-2cwcd\" (UID: \"30762972-ab72-4cdd-a693-584f9c7cdde5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cwcd" Jul 08 19:23:56.692005 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.690673 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a7da2be-d237-4bfa-887d-0be7632bfdc0-system-cni-dir\") pod \"multus-gz54s\" (UID: \"3a7da2be-d237-4bfa-887d-0be7632bfdc0\") " pod="openshift-multus/multus-gz54s" Jul 08 19:23:56.692005 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.690672 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/01c82975-c67d-4e46-b16a-f558032f5dc3-etc-sysctl-d\") pod \"tuned-xb4w9\" (UID: \"01c82975-c67d-4e46-b16a-f558032f5dc3\") " pod="openshift-cluster-node-tuning-operator/tuned-xb4w9" Jul 08 19:23:56.692005 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.690593 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3af16262-f54e-4683-b76c-e1ccfdff4c3c-run-ovn\") pod \"ovnkube-node-cdp8n\" (UID: \"3af16262-f54e-4683-b76c-e1ccfdff4c3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdp8n" Jul 08 19:23:56.692005 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.690694 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3af16262-f54e-4683-b76c-e1ccfdff4c3c-log-socket\") pod \"ovnkube-node-cdp8n\" (UID: \"3af16262-f54e-4683-b76c-e1ccfdff4c3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdp8n" Jul 08 19:23:56.692005 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.690716 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3a7da2be-d237-4bfa-887d-0be7632bfdc0-host-var-lib-cni-bin\") pod \"multus-gz54s\" (UID: \"3a7da2be-d237-4bfa-887d-0be7632bfdc0\") " pod="openshift-multus/multus-gz54s" Jul 08 19:23:56.692005 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.690736 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a7da2be-d237-4bfa-887d-0be7632bfdc0-host-var-lib-kubelet\") pod \"multus-gz54s\" (UID: \"3a7da2be-d237-4bfa-887d-0be7632bfdc0\") " pod="openshift-multus/multus-gz54s" Jul 08 19:23:56.692005 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.690743 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/30762972-ab72-4cdd-a693-584f9c7cdde5-etc-selinux\") pod \"aws-ebs-csi-driver-node-2cwcd\" (UID: \"30762972-ab72-4cdd-a693-584f9c7cdde5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cwcd" Jul 08 19:23:56.692005 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.690759 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/01c82975-c67d-4e46-b16a-f558032f5dc3-etc-kubernetes\") pod \"tuned-xb4w9\" (UID: \"01c82975-c67d-4e46-b16a-f558032f5dc3\") " pod="openshift-cluster-node-tuning-operator/tuned-xb4w9" Jul 08 19:23:56.692005 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.690781 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/01c82975-c67d-4e46-b16a-f558032f5dc3-lib-modules\") pod \"tuned-xb4w9\" (UID: \"01c82975-c67d-4e46-b16a-f558032f5dc3\") " pod="openshift-cluster-node-tuning-operator/tuned-xb4w9" Jul 08 19:23:56.692005 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.690785 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3af16262-f54e-4683-b76c-e1ccfdff4c3c-log-socket\") pod \"ovnkube-node-cdp8n\" (UID: \"3af16262-f54e-4683-b76c-e1ccfdff4c3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdp8n" Jul 08 19:23:56.692005 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.690783 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/01c82975-c67d-4e46-b16a-f558032f5dc3-host\") pod \"tuned-xb4w9\" (UID: \"01c82975-c67d-4e46-b16a-f558032f5dc3\") " pod="openshift-cluster-node-tuning-operator/tuned-xb4w9" Jul 08 19:23:56.692005 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.690818 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3af16262-f54e-4683-b76c-e1ccfdff4c3c-run-openvswitch\") pod \"ovnkube-node-cdp8n\" (UID: \"3af16262-f54e-4683-b76c-e1ccfdff4c3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdp8n" Jul 08 19:23:56.692005 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.690822 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a7da2be-d237-4bfa-887d-0be7632bfdc0-host-var-lib-kubelet\") pod \"multus-gz54s\" (UID: \"3a7da2be-d237-4bfa-887d-0be7632bfdc0\") " pod="openshift-multus/multus-gz54s" Jul 08 19:23:56.692839 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.690820 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3a7da2be-d237-4bfa-887d-0be7632bfdc0-host-var-lib-cni-bin\") pod \"multus-gz54s\" (UID: \"3a7da2be-d237-4bfa-887d-0be7632bfdc0\") " pod="openshift-multus/multus-gz54s" Jul 08 19:23:56.692839 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.690839 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/30762972-ab72-4cdd-a693-584f9c7cdde5-sys-fs\") pod \"aws-ebs-csi-driver-node-2cwcd\" (UID: \"30762972-ab72-4cdd-a693-584f9c7cdde5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cwcd" Jul 08 19:23:56.692839 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.690842 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7g4k\" (UniqueName: \"kubernetes.io/projected/3af16262-f54e-4683-b76c-e1ccfdff4c3c-kube-api-access-x7g4k\") pod \"ovnkube-node-cdp8n\" (UID: \"3af16262-f54e-4683-b76c-e1ccfdff4c3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdp8n" Jul 08 19:23:56.692839 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.690884 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3a7da2be-d237-4bfa-887d-0be7632bfdc0-host-run-netns\") pod \"multus-gz54s\" (UID: \"3a7da2be-d237-4bfa-887d-0be7632bfdc0\") " pod="openshift-multus/multus-gz54s" Jul 08 19:23:56.692839 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.690893 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3af16262-f54e-4683-b76c-e1ccfdff4c3c-run-openvswitch\") pod \"ovnkube-node-cdp8n\" (UID: \"3af16262-f54e-4683-b76c-e1ccfdff4c3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdp8n" Jul 08 19:23:56.692839 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.690909 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvggl\" (UniqueName: \"kubernetes.io/projected/3a7da2be-d237-4bfa-887d-0be7632bfdc0-kube-api-access-qvggl\") pod \"multus-gz54s\" (UID: \"3a7da2be-d237-4bfa-887d-0be7632bfdc0\") " pod="openshift-multus/multus-gz54s" Jul 08 19:23:56.692839 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.690920 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/01c82975-c67d-4e46-b16a-f558032f5dc3-host\") pod \"tuned-xb4w9\" (UID: \"01c82975-c67d-4e46-b16a-f558032f5dc3\") " pod="openshift-cluster-node-tuning-operator/tuned-xb4w9" Jul 08 19:23:56.692839 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.690929 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3a7da2be-d237-4bfa-887d-0be7632bfdc0-host-run-netns\") pod \"multus-gz54s\" (UID: \"3a7da2be-d237-4bfa-887d-0be7632bfdc0\") " pod="openshift-multus/multus-gz54s" Jul 08 19:23:56.692839 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.690966 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3af16262-f54e-4683-b76c-e1ccfdff4c3c-etc-openvswitch\") pod \"ovnkube-node-cdp8n\" (UID: \"3af16262-f54e-4683-b76c-e1ccfdff4c3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdp8n" Jul 08 19:23:56.692839 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.690966 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8b78ed73-731b-49fa-9220-4b1e45787806-serviceca\") pod \"node-ca-6w52p\" (UID: \"8b78ed73-731b-49fa-9220-4b1e45787806\") " pod="openshift-image-registry/node-ca-6w52p" Jul 08 19:23:56.692839 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.691079 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3a7da2be-d237-4bfa-887d-0be7632bfdc0-multus-socket-dir-parent\") pod \"multus-gz54s\" (UID: \"3a7da2be-d237-4bfa-887d-0be7632bfdc0\") " pod="openshift-multus/multus-gz54s" Jul 08 19:23:56.692839 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.691085 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3af16262-f54e-4683-b76c-e1ccfdff4c3c-etc-openvswitch\") pod \"ovnkube-node-cdp8n\" (UID: \"3af16262-f54e-4683-b76c-e1ccfdff4c3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdp8n" Jul 08 19:23:56.692839 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.691122 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/01c82975-c67d-4e46-b16a-f558032f5dc3-etc-kubernetes\") pod \"tuned-xb4w9\" (UID: \"01c82975-c67d-4e46-b16a-f558032f5dc3\") " pod="openshift-cluster-node-tuning-operator/tuned-xb4w9" Jul 08 19:23:56.692839 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.691136 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/01c82975-c67d-4e46-b16a-f558032f5dc3-var-lib-kubelet\") pod \"tuned-xb4w9\" (UID: \"01c82975-c67d-4e46-b16a-f558032f5dc3\") " pod="openshift-cluster-node-tuning-operator/tuned-xb4w9" Jul 08 19:23:56.692839 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.691141 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3a7da2be-d237-4bfa-887d-0be7632bfdc0-multus-socket-dir-parent\") pod \"multus-gz54s\" (UID: \"3a7da2be-d237-4bfa-887d-0be7632bfdc0\") " pod="openshift-multus/multus-gz54s" Jul 08 19:23:56.692839 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.691155 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3af16262-f54e-4683-b76c-e1ccfdff4c3c-node-log\") pod \"ovnkube-node-cdp8n\" (UID: \"3af16262-f54e-4683-b76c-e1ccfdff4c3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdp8n" Jul 08 19:23:56.692839 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.691169 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3a7da2be-d237-4bfa-887d-0be7632bfdc0-cni-binary-copy\") pod \"multus-gz54s\" (UID: \"3a7da2be-d237-4bfa-887d-0be7632bfdc0\") " pod="openshift-multus/multus-gz54s" Jul 08 19:23:56.692839 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.691185 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3a7da2be-d237-4bfa-887d-0be7632bfdc0-host-var-lib-cni-multus\") pod \"multus-gz54s\" (UID: \"3a7da2be-d237-4bfa-887d-0be7632bfdc0\") " pod="openshift-multus/multus-gz54s" Jul 08 19:23:56.693565 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.691206 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3a7da2be-d237-4bfa-887d-0be7632bfdc0-multus-daemon-config\") pod \"multus-gz54s\" (UID: \"3a7da2be-d237-4bfa-887d-0be7632bfdc0\") " pod="openshift-multus/multus-gz54s" Jul 08 19:23:56.693565 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.691196 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/01c82975-c67d-4e46-b16a-f558032f5dc3-var-lib-kubelet\") pod \"tuned-xb4w9\" (UID: \"01c82975-c67d-4e46-b16a-f558032f5dc3\") " pod="openshift-cluster-node-tuning-operator/tuned-xb4w9" Jul 08 19:23:56.693565 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.691243 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3af16262-f54e-4683-b76c-e1ccfdff4c3c-node-log\") pod \"ovnkube-node-cdp8n\" (UID: \"3af16262-f54e-4683-b76c-e1ccfdff4c3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdp8n" Jul 08 19:23:56.693565 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.691256 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3a7da2be-d237-4bfa-887d-0be7632bfdc0-host-var-lib-cni-multus\") pod \"multus-gz54s\" (UID: \"3a7da2be-d237-4bfa-887d-0be7632bfdc0\") " pod="openshift-multus/multus-gz54s" Jul 08 19:23:56.693565 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.691283 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3af16262-f54e-4683-b76c-e1ccfdff4c3c-run-systemd\") pod \"ovnkube-node-cdp8n\" (UID: \"3af16262-f54e-4683-b76c-e1ccfdff4c3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdp8n" Jul 08 19:23:56.693565 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.691327 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3af16262-f54e-4683-b76c-e1ccfdff4c3c-env-overrides\") pod \"ovnkube-node-cdp8n\" (UID: \"3af16262-f54e-4683-b76c-e1ccfdff4c3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdp8n" Jul 08 19:23:56.693565 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.691352 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3af16262-f54e-4683-b76c-e1ccfdff4c3c-run-systemd\") pod \"ovnkube-node-cdp8n\" (UID: \"3af16262-f54e-4683-b76c-e1ccfdff4c3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdp8n" Jul 08 19:23:56.693565 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.691351 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a7da2be-d237-4bfa-887d-0be7632bfdc0-multus-cni-dir\") pod \"multus-gz54s\" (UID: \"3a7da2be-d237-4bfa-887d-0be7632bfdc0\") " pod="openshift-multus/multus-gz54s" Jul 08 19:23:56.693565 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.691383 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a7da2be-d237-4bfa-887d-0be7632bfdc0-etc-kubernetes\") pod \"multus-gz54s\" (UID: \"3a7da2be-d237-4bfa-887d-0be7632bfdc0\") " pod="openshift-multus/multus-gz54s" Jul 08 19:23:56.693565 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.691406 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65h2n\" (UniqueName: \"kubernetes.io/projected/8b78ed73-731b-49fa-9220-4b1e45787806-kube-api-access-65h2n\") pod \"node-ca-6w52p\" (UID: \"8b78ed73-731b-49fa-9220-4b1e45787806\") " pod="openshift-image-registry/node-ca-6w52p" Jul 08 19:23:56.693565 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.691428 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz2zp\" (UniqueName: \"kubernetes.io/projected/bbe4cd13-7412-42f1-9035-865ce20ba987-kube-api-access-xz2zp\") pod \"iptables-alerter-bkjrq\" (UID: \"bbe4cd13-7412-42f1-9035-865ce20ba987\") " pod="openshift-network-operator/iptables-alerter-bkjrq" Jul 08 19:23:56.693565 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.691436 2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 19:23:56.693565 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.691452 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5psp\" (UniqueName: \"kubernetes.io/projected/474e5893-2b36-4b57-b154-15b0f8b4f300-kube-api-access-v5psp\") pod \"node-resolver-dghh8\" (UID: \"474e5893-2b36-4b57-b154-15b0f8b4f300\") " pod="openshift-dns/node-resolver-dghh8" Jul 08 19:23:56.693565 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.691478 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3af16262-f54e-4683-b76c-e1ccfdff4c3c-host-run-ovn-kubernetes\") pod \"ovnkube-node-cdp8n\" (UID: \"3af16262-f54e-4683-b76c-e1ccfdff4c3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdp8n" Jul 08 19:23:56.693565 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.691501 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55818509-538d-4733-9e10-eaaf841ef071-metrics-certs\") pod \"network-metrics-daemon-rfbjf\" (UID: \"55818509-538d-4733-9e10-eaaf841ef071\") " pod="openshift-multus/network-metrics-daemon-rfbjf" Jul 08 19:23:56.693565 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.691523 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bbe4cd13-7412-42f1-9035-865ce20ba987-iptables-alerter-script\") pod \"iptables-alerter-bkjrq\" (UID: \"bbe4cd13-7412-42f1-9035-865ce20ba987\") " pod="openshift-network-operator/iptables-alerter-bkjrq" Jul 08 19:23:56.693565 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.691562 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/01c82975-c67d-4e46-b16a-f558032f5dc3-etc-sysctl-conf\") pod \"tuned-xb4w9\" (UID: \"01c82975-c67d-4e46-b16a-f558032f5dc3\") " pod="openshift-cluster-node-tuning-operator/tuned-xb4w9" Jul 08 19:23:56.693565 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.691584 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/01c82975-c67d-4e46-b16a-f558032f5dc3-sys\") pod \"tuned-xb4w9\" (UID: \"01c82975-c67d-4e46-b16a-f558032f5dc3\") " pod="openshift-cluster-node-tuning-operator/tuned-xb4w9" Jul 08 19:23:56.694300 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.691608 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/30762972-ab72-4cdd-a693-584f9c7cdde5-registration-dir\") pod \"aws-ebs-csi-driver-node-2cwcd\" (UID: \"30762972-ab72-4cdd-a693-584f9c7cdde5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cwcd" Jul 08 19:23:56.694300 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.691631 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3af16262-f54e-4683-b76c-e1ccfdff4c3c-host-slash\") pod \"ovnkube-node-cdp8n\" (UID: \"3af16262-f54e-4683-b76c-e1ccfdff4c3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdp8n" Jul 08 19:23:56.694300 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.691655 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3a7da2be-d237-4bfa-887d-0be7632bfdc0-hostroot\") pod \"multus-gz54s\" (UID: \"3a7da2be-d237-4bfa-887d-0be7632bfdc0\") " pod="openshift-multus/multus-gz54s" Jul 08 19:23:56.694300 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.691678 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bbe4cd13-7412-42f1-9035-865ce20ba987-host-slash\") pod \"iptables-alerter-bkjrq\" (UID: \"bbe4cd13-7412-42f1-9035-865ce20ba987\") " pod="openshift-network-operator/iptables-alerter-bkjrq" Jul 08 19:23:56.694300 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.691704 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3a7da2be-d237-4bfa-887d-0be7632bfdc0-multus-conf-dir\") pod \"multus-gz54s\" (UID: \"3a7da2be-d237-4bfa-887d-0be7632bfdc0\") " pod="openshift-multus/multus-gz54s" Jul 08 19:23:56.694300 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.691721 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3a7da2be-d237-4bfa-887d-0be7632bfdc0-cni-binary-copy\") pod \"multus-gz54s\" (UID: \"3a7da2be-d237-4bfa-887d-0be7632bfdc0\") " pod="openshift-multus/multus-gz54s" Jul 08 19:23:56.694300 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.691730 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/01c82975-c67d-4e46-b16a-f558032f5dc3-etc-modprobe-d\") pod \"tuned-xb4w9\" (UID: \"01c82975-c67d-4e46-b16a-f558032f5dc3\") " pod="openshift-cluster-node-tuning-operator/tuned-xb4w9" Jul 08 19:23:56.694300 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.691758 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3af16262-f54e-4683-b76c-e1ccfdff4c3c-host-cni-bin\") pod \"ovnkube-node-cdp8n\" (UID: \"3af16262-f54e-4683-b76c-e1ccfdff4c3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdp8n" Jul 08 19:23:56.694300 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.691785 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3af16262-f54e-4683-b76c-e1ccfdff4c3c-ovnkube-config\") pod \"ovnkube-node-cdp8n\" (UID: \"3af16262-f54e-4683-b76c-e1ccfdff4c3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdp8n" Jul 08 19:23:56.694300 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.691820 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/01c82975-c67d-4e46-b16a-f558032f5dc3-run\") pod \"tuned-xb4w9\" (UID: \"01c82975-c67d-4e46-b16a-f558032f5dc3\") " pod="openshift-cluster-node-tuning-operator/tuned-xb4w9" Jul 08 19:23:56.694300 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.691918 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/01c82975-c67d-4e46-b16a-f558032f5dc3-run\") pod \"tuned-xb4w9\" (UID: \"01c82975-c67d-4e46-b16a-f558032f5dc3\") " pod="openshift-cluster-node-tuning-operator/tuned-xb4w9" Jul 08 19:23:56.694300 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.691962 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3af16262-f54e-4683-b76c-e1ccfdff4c3c-env-overrides\") pod \"ovnkube-node-cdp8n\" (UID: \"3af16262-f54e-4683-b76c-e1ccfdff4c3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdp8n" Jul 08 19:23:56.694300 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.691977 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/30762972-ab72-4cdd-a693-584f9c7cdde5-registration-dir\") pod \"aws-ebs-csi-driver-node-2cwcd\" (UID: \"30762972-ab72-4cdd-a693-584f9c7cdde5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cwcd" Jul 08 19:23:56.694300 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.692024 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a7da2be-d237-4bfa-887d-0be7632bfdc0-multus-cni-dir\") pod \"multus-gz54s\" (UID: \"3a7da2be-d237-4bfa-887d-0be7632bfdc0\") " pod="openshift-multus/multus-gz54s" Jul 08 19:23:56.694300 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.692063 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3a7da2be-d237-4bfa-887d-0be7632bfdc0-hostroot\") pod \"multus-gz54s\" (UID: \"3a7da2be-d237-4bfa-887d-0be7632bfdc0\") " pod="openshift-multus/multus-gz54s" Jul 08 19:23:56.694300 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.692067 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3a7da2be-d237-4bfa-887d-0be7632bfdc0-multus-conf-dir\") pod \"multus-gz54s\" (UID: \"3a7da2be-d237-4bfa-887d-0be7632bfdc0\") " pod="openshift-multus/multus-gz54s" Jul 08 19:23:56.694300 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.692099 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bbe4cd13-7412-42f1-9035-865ce20ba987-host-slash\") pod \"iptables-alerter-bkjrq\" (UID: \"bbe4cd13-7412-42f1-9035-865ce20ba987\") " pod="openshift-network-operator/iptables-alerter-bkjrq" Jul 08 19:23:56.694300 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.692026 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3af16262-f54e-4683-b76c-e1ccfdff4c3c-host-slash\") pod \"ovnkube-node-cdp8n\" (UID: \"3af16262-f54e-4683-b76c-e1ccfdff4c3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdp8n" Jul 08 19:23:56.695089 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.692161 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/01c82975-c67d-4e46-b16a-f558032f5dc3-etc-modprobe-d\") pod \"tuned-xb4w9\" (UID: \"01c82975-c67d-4e46-b16a-f558032f5dc3\") " pod="openshift-cluster-node-tuning-operator/tuned-xb4w9" Jul 08 19:23:56.695089 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.692170 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3af16262-f54e-4683-b76c-e1ccfdff4c3c-host-run-ovn-kubernetes\") pod \"ovnkube-node-cdp8n\" (UID: \"3af16262-f54e-4683-b76c-e1ccfdff4c3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdp8n" Jul 08 19:23:56.695089 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.692179 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3af16262-f54e-4683-b76c-e1ccfdff4c3c-host-cni-bin\") pod \"ovnkube-node-cdp8n\" (UID: \"3af16262-f54e-4683-b76c-e1ccfdff4c3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdp8n" Jul 08 19:23:56.695089 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.692247 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/01c82975-c67d-4e46-b16a-f558032f5dc3-sys\") pod \"tuned-xb4w9\" (UID: \"01c82975-c67d-4e46-b16a-f558032f5dc3\") " pod="openshift-cluster-node-tuning-operator/tuned-xb4w9" Jul 08 19:23:56.695089 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.692288 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a7da2be-d237-4bfa-887d-0be7632bfdc0-etc-kubernetes\") pod \"multus-gz54s\" (UID: \"3a7da2be-d237-4bfa-887d-0be7632bfdc0\") " pod="openshift-multus/multus-gz54s" Jul 08 19:23:56.695089 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.692289 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/01c82975-c67d-4e46-b16a-f558032f5dc3-etc-sysctl-conf\") pod \"tuned-xb4w9\" (UID: \"01c82975-c67d-4e46-b16a-f558032f5dc3\") " pod="openshift-cluster-node-tuning-operator/tuned-xb4w9" Jul 08 19:23:56.695089 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:23:56.692357 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 19:23:56.695089 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:23:56.692409 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55818509-538d-4733-9e10-eaaf841ef071-metrics-certs podName:55818509-538d-4733-9e10-eaaf841ef071 nodeName:}" failed. No retries permitted until 2026-07-08 19:23:57.19239315 +0000 UTC m=+3.078583188 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55818509-538d-4733-9e10-eaaf841ef071-metrics-certs") pod "network-metrics-daemon-rfbjf" (UID: "55818509-538d-4733-9e10-eaaf841ef071") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 19:23:56.695089 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.692519 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/01c82975-c67d-4e46-b16a-f558032f5dc3-tmp\") pod \"tuned-xb4w9\" (UID: \"01c82975-c67d-4e46-b16a-f558032f5dc3\") " pod="openshift-cluster-node-tuning-operator/tuned-xb4w9" Jul 08 19:23:56.695089 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.692610 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3af16262-f54e-4683-b76c-e1ccfdff4c3c-ovnkube-config\") pod \"ovnkube-node-cdp8n\" (UID: \"3af16262-f54e-4683-b76c-e1ccfdff4c3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdp8n" Jul 08 19:23:56.695089 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.692619 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/01c82975-c67d-4e46-b16a-f558032f5dc3-etc-tuned\") pod \"tuned-xb4w9\" (UID: \"01c82975-c67d-4e46-b16a-f558032f5dc3\") " pod="openshift-cluster-node-tuning-operator/tuned-xb4w9" Jul 08 19:23:56.695089 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.692985 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3a7da2be-d237-4bfa-887d-0be7632bfdc0-multus-daemon-config\") pod \"multus-gz54s\" (UID: \"3a7da2be-d237-4bfa-887d-0be7632bfdc0\") " pod="openshift-multus/multus-gz54s" Jul 08 19:23:56.695089 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.693040 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bbe4cd13-7412-42f1-9035-865ce20ba987-iptables-alerter-script\") pod \"iptables-alerter-bkjrq\" (UID: \"bbe4cd13-7412-42f1-9035-865ce20ba987\") " pod="openshift-network-operator/iptables-alerter-bkjrq" Jul 08 19:23:56.695089 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.694636 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3af16262-f54e-4683-b76c-e1ccfdff4c3c-ovn-node-metrics-cert\") pod \"ovnkube-node-cdp8n\" (UID: \"3af16262-f54e-4683-b76c-e1ccfdff4c3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdp8n" Jul 08 19:23:56.700754 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.700706 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7g4k\" (UniqueName: \"kubernetes.io/projected/3af16262-f54e-4683-b76c-e1ccfdff4c3c-kube-api-access-x7g4k\") pod \"ovnkube-node-cdp8n\" (UID: \"3af16262-f54e-4683-b76c-e1ccfdff4c3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdp8n" Jul 08 19:23:56.700944 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.700784 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvggl\" (UniqueName: \"kubernetes.io/projected/3a7da2be-d237-4bfa-887d-0be7632bfdc0-kube-api-access-qvggl\") pod \"multus-gz54s\" (UID: \"3a7da2be-d237-4bfa-887d-0be7632bfdc0\") " pod="openshift-multus/multus-gz54s" Jul 08 19:23:56.701374 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.701278 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65h2n\" (UniqueName: \"kubernetes.io/projected/8b78ed73-731b-49fa-9220-4b1e45787806-kube-api-access-65h2n\") pod \"node-ca-6w52p\" (UID: \"8b78ed73-731b-49fa-9220-4b1e45787806\") " pod="openshift-image-registry/node-ca-6w52p" Jul 08 19:23:56.701768 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.701748 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5psp\" (UniqueName: \"kubernetes.io/projected/474e5893-2b36-4b57-b154-15b0f8b4f300-kube-api-access-v5psp\") pod \"node-resolver-dghh8\" (UID: \"474e5893-2b36-4b57-b154-15b0f8b4f300\") " pod="openshift-dns/node-resolver-dghh8" Jul 08 19:23:56.702543 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.702521 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz2zp\" (UniqueName: \"kubernetes.io/projected/bbe4cd13-7412-42f1-9035-865ce20ba987-kube-api-access-xz2zp\") pod \"iptables-alerter-bkjrq\" (UID: \"bbe4cd13-7412-42f1-9035-865ce20ba987\") " pod="openshift-network-operator/iptables-alerter-bkjrq" Jul 08 19:23:56.702777 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.702754 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bhrl\" (UniqueName: \"kubernetes.io/projected/30762972-ab72-4cdd-a693-584f9c7cdde5-kube-api-access-9bhrl\") pod \"aws-ebs-csi-driver-node-2cwcd\" (UID: \"30762972-ab72-4cdd-a693-584f9c7cdde5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cwcd" Jul 08 19:23:56.703211 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.703192 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sksw2\" (UniqueName: \"kubernetes.io/projected/01c82975-c67d-4e46-b16a-f558032f5dc3-kube-api-access-sksw2\") pod \"tuned-xb4w9\" (UID: \"01c82975-c67d-4e46-b16a-f558032f5dc3\") " pod="openshift-cluster-node-tuning-operator/tuned-xb4w9" Jul 08 19:23:56.703291 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.703274 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw5sx\" (UniqueName: \"kubernetes.io/projected/55818509-538d-4733-9e10-eaaf841ef071-kube-api-access-qw5sx\") pod \"network-metrics-daemon-rfbjf\" (UID: \"55818509-538d-4733-9e10-eaaf841ef071\") " pod="openshift-multus/network-metrics-daemon-rfbjf" Jul 08 19:23:56.786359 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.786278 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5kt6t" Jul 08 19:23:56.792658 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.792637 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-srmfq" Jul 08 19:23:56.808371 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.808331 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gz54s" Jul 08 19:23:56.812372 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.812350 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6w52p" Jul 08 19:23:56.818961 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.818937 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dghh8" Jul 08 19:23:56.825472 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.825449 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xb4w9" Jul 08 19:23:56.833272 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.833231 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cwcd" Jul 08 19:23:56.840069 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.840046 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bkjrq" Jul 08 19:23:56.845893 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:56.845874 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-cdp8n" Jul 08 19:23:57.094067 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:57.093970 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vc5r\" (UniqueName: \"kubernetes.io/projected/60e1939e-8740-43f5-8d7b-056635c9f877-kube-api-access-5vc5r\") pod \"network-check-target-rs9hh\" (UID: \"60e1939e-8740-43f5-8d7b-056635c9f877\") " pod="openshift-network-diagnostics/network-check-target-rs9hh" Jul 08 19:23:57.094067 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:57.094048 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24b6882e-3f1e-4c28-b128-2b16f3478940-original-pull-secret\") pod \"global-pull-secret-syncer-qdfqb\" (UID: \"24b6882e-3f1e-4c28-b128-2b16f3478940\") " pod="kube-system/global-pull-secret-syncer-qdfqb" Jul 08 19:23:57.094247 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:23:57.094158 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 19:23:57.094247 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:23:57.094173 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 19:23:57.094247 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:23:57.094183 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 19:23:57.094247 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:23:57.094195 2566 projected.go:196] Error preparing data for projected volume kube-api-access-5vc5r for pod openshift-network-diagnostics/network-check-target-rs9hh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 19:23:57.094247 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:23:57.094239 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24b6882e-3f1e-4c28-b128-2b16f3478940-original-pull-secret podName:24b6882e-3f1e-4c28-b128-2b16f3478940 nodeName:}" failed. No retries permitted until 2026-07-08 19:23:58.09422118 +0000 UTC m=+3.980411215 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/24b6882e-3f1e-4c28-b128-2b16f3478940-original-pull-secret") pod "global-pull-secret-syncer-qdfqb" (UID: "24b6882e-3f1e-4c28-b128-2b16f3478940") : object "kube-system"/"original-pull-secret" not registered Jul 08 19:23:57.094437 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:23:57.094253 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60e1939e-8740-43f5-8d7b-056635c9f877-kube-api-access-5vc5r podName:60e1939e-8740-43f5-8d7b-056635c9f877 nodeName:}" failed. No retries permitted until 2026-07-08 19:23:58.09424754 +0000 UTC m=+3.980437570 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-5vc5r" (UniqueName: "kubernetes.io/projected/60e1939e-8740-43f5-8d7b-056635c9f877-kube-api-access-5vc5r") pod "network-check-target-rs9hh" (UID: "60e1939e-8740-43f5-8d7b-056635c9f877") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 19:23:57.195272 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:57.195241 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55818509-538d-4733-9e10-eaaf841ef071-metrics-certs\") pod \"network-metrics-daemon-rfbjf\" (UID: \"55818509-538d-4733-9e10-eaaf841ef071\") " pod="openshift-multus/network-metrics-daemon-rfbjf" Jul 08 19:23:57.195456 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:23:57.195433 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 19:23:57.195528 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:23:57.195517 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55818509-538d-4733-9e10-eaaf841ef071-metrics-certs podName:55818509-538d-4733-9e10-eaaf841ef071 nodeName:}" failed. No retries permitted until 2026-07-08 19:23:58.195497002 +0000 UTC m=+4.081687033 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55818509-538d-4733-9e10-eaaf841ef071-metrics-certs") pod "network-metrics-daemon-rfbjf" (UID: "55818509-538d-4733-9e10-eaaf841ef071") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 19:23:57.511276 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:57.511251 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a7da2be_d237_4bfa_887d_0be7632bfdc0.slice/crio-10d488b95163f9edbed4ebc5a1fd7715ccdf5ca9add9bb70edaa585fc683caff WatchSource:0}: Error finding container 10d488b95163f9edbed4ebc5a1fd7715ccdf5ca9add9bb70edaa585fc683caff: Status 404 returned error can't find the container with id 10d488b95163f9edbed4ebc5a1fd7715ccdf5ca9add9bb70edaa585fc683caff Jul 08 19:23:57.515897 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:57.515854 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3af16262_f54e_4683_b76c_e1ccfdff4c3c.slice/crio-12e9a05da363a330d2916cde5f607656fee627a5197019e427d937d0d4393d41 WatchSource:0}: Error finding container 12e9a05da363a330d2916cde5f607656fee627a5197019e427d937d0d4393d41: Status 404 returned error can't find the container with id 12e9a05da363a330d2916cde5f607656fee627a5197019e427d937d0d4393d41 Jul 08 19:23:57.516443 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:57.516422 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod474e5893_2b36_4b57_b154_15b0f8b4f300.slice/crio-a4c99bf55fc234aa76e420106dd645e58e8b430fc23d0699b55116a21c0cdc32 WatchSource:0}: Error finding container a4c99bf55fc234aa76e420106dd645e58e8b430fc23d0699b55116a21c0cdc32: Status 404 returned error can't find the container with id a4c99bf55fc234aa76e420106dd645e58e8b430fc23d0699b55116a21c0cdc32 Jul 08 19:23:57.517425 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:57.517355 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30762972_ab72_4cdd_a693_584f9c7cdde5.slice/crio-0444fc3e8c7a33f18cc3878ddddb18d28d13d7593e52c699da5133c86d1b9cc3 WatchSource:0}: Error finding container 0444fc3e8c7a33f18cc3878ddddb18d28d13d7593e52c699da5133c86d1b9cc3: Status 404 returned error can't find the container with id 0444fc3e8c7a33f18cc3878ddddb18d28d13d7593e52c699da5133c86d1b9cc3 Jul 08 19:23:57.518028 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:57.517997 2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-07 19:18:56 +0000 UTC" deadline="2028-01-29 09:16:44.089866163 +0000 UTC" Jul 08 19:23:57.518028 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:57.518028 2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13669h52m46.571842384s" Jul 08 19:23:57.523356 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:57.522730 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbe0d11c_c9c5_4cdb_ad9b_08b142e7335f.slice/crio-a562954e4508a2cc095c3f48c588e65d075b2fe1a61c1b2c96cc0fb948969c2d WatchSource:0}: Error finding container a562954e4508a2cc095c3f48c588e65d075b2fe1a61c1b2c96cc0fb948969c2d: Status 404 returned error can't find the container with id a562954e4508a2cc095c3f48c588e65d075b2fe1a61c1b2c96cc0fb948969c2d Jul 08 19:23:57.523796 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:57.523704 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b78ed73_731b_49fa_9220_4b1e45787806.slice/crio-5d7deb230cd7197b2ca517f3fd15393b89708c04a142b8562d38ee2d9b21c97b WatchSource:0}: Error finding container 5d7deb230cd7197b2ca517f3fd15393b89708c04a142b8562d38ee2d9b21c97b: Status 404 returned error can't find the container with id 5d7deb230cd7197b2ca517f3fd15393b89708c04a142b8562d38ee2d9b21c97b Jul 08 19:23:57.526163 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:23:57.526054 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbe4cd13_7412_42f1_9035_865ce20ba987.slice/crio-5cbc47018f069b5e2ae0cfd5edc0e8ab98ec201f6f41e4cf70bab9d694cf2c2c WatchSource:0}: Error finding container 5cbc47018f069b5e2ae0cfd5edc0e8ab98ec201f6f41e4cf70bab9d694cf2c2c: Status 404 returned error can't find the container with id 5cbc47018f069b5e2ae0cfd5edc0e8ab98ec201f6f41e4cf70bab9d694cf2c2c Jul 08 19:23:57.625143 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:57.625003 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-srmfq" event={"ID":"dbe0d11c-c9c5-4cdb-ad9b-08b142e7335f","Type":"ContainerStarted","Data":"a562954e4508a2cc095c3f48c588e65d075b2fe1a61c1b2c96cc0fb948969c2d"} Jul 08 19:23:57.629528 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:57.629500 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cdp8n" event={"ID":"3af16262-f54e-4683-b76c-e1ccfdff4c3c","Type":"ContainerStarted","Data":"12e9a05da363a330d2916cde5f607656fee627a5197019e427d937d0d4393d41"} Jul 08 19:23:57.630611 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:57.630590 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bkjrq" event={"ID":"bbe4cd13-7412-42f1-9035-865ce20ba987","Type":"ContainerStarted","Data":"5cbc47018f069b5e2ae0cfd5edc0e8ab98ec201f6f41e4cf70bab9d694cf2c2c"} Jul 08 19:23:57.631600 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:57.631580 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6w52p" event={"ID":"8b78ed73-731b-49fa-9220-4b1e45787806","Type":"ContainerStarted","Data":"5d7deb230cd7197b2ca517f3fd15393b89708c04a142b8562d38ee2d9b21c97b"} Jul 08 19:23:57.637149 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:57.637008 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5kt6t" event={"ID":"42578ec4-91d1-4d4d-b9a1-12a559fee154","Type":"ContainerStarted","Data":"b0dd0414c13ced432da2fc546c04304b9c611ae24120ffa0146860d13a9fbb14"} Jul 08 19:23:57.643349 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:57.643325 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xb4w9" event={"ID":"01c82975-c67d-4e46-b16a-f558032f5dc3","Type":"ContainerStarted","Data":"c61ff8a4190d73f39ed6c481b07c54bf8ea94f2d1af619d5d0f43b7ff1f83d5f"} Jul 08 19:23:57.646367 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:57.646344 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cwcd" event={"ID":"30762972-ab72-4cdd-a693-584f9c7cdde5","Type":"ContainerStarted","Data":"0444fc3e8c7a33f18cc3878ddddb18d28d13d7593e52c699da5133c86d1b9cc3"} Jul 08 19:23:57.649404 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:57.649381 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dghh8" event={"ID":"474e5893-2b36-4b57-b154-15b0f8b4f300","Type":"ContainerStarted","Data":"a4c99bf55fc234aa76e420106dd645e58e8b430fc23d0699b55116a21c0cdc32"} Jul 08 19:23:57.650353 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:57.650334 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gz54s" event={"ID":"3a7da2be-d237-4bfa-887d-0be7632bfdc0","Type":"ContainerStarted","Data":"10d488b95163f9edbed4ebc5a1fd7715ccdf5ca9add9bb70edaa585fc683caff"} Jul 08 19:23:57.651935 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:57.651913 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-195.ec2.internal" event={"ID":"47fb7cb7c700e5809cbc3bdae3a4659d","Type":"ContainerStarted","Data":"d1cd92e81c59e9d194fea44e7205c8df596325aad38b1cc31840a0f200c1856c"} Jul 08 19:23:57.668218 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:57.668174 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-195.ec2.internal" podStartSLOduration=1.668162948 podStartE2EDuration="1.668162948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 19:23:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:23:57.667790411 +0000 UTC m=+3.553980459" watchObservedRunningTime="2026-07-08 19:23:57.668162948 +0000 UTC m=+3.554352995" Jul 08 19:23:57.758986 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:57.758956 2566 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 19:23:58.101830 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:58.101602 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vc5r\" (UniqueName: \"kubernetes.io/projected/60e1939e-8740-43f5-8d7b-056635c9f877-kube-api-access-5vc5r\") pod \"network-check-target-rs9hh\" (UID: \"60e1939e-8740-43f5-8d7b-056635c9f877\") " pod="openshift-network-diagnostics/network-check-target-rs9hh" Jul 08 19:23:58.101830 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:58.101674 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24b6882e-3f1e-4c28-b128-2b16f3478940-original-pull-secret\") pod \"global-pull-secret-syncer-qdfqb\" (UID: \"24b6882e-3f1e-4c28-b128-2b16f3478940\") " pod="kube-system/global-pull-secret-syncer-qdfqb" Jul 08 19:23:58.101830 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:23:58.101768 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 19:23:58.101830 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:23:58.101792 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 19:23:58.101830 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:23:58.101804 2566 projected.go:196] Error preparing data for projected volume kube-api-access-5vc5r for pod openshift-network-diagnostics/network-check-target-rs9hh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 19:23:58.102180 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:23:58.101875 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60e1939e-8740-43f5-8d7b-056635c9f877-kube-api-access-5vc5r podName:60e1939e-8740-43f5-8d7b-056635c9f877 nodeName:}" failed. No retries permitted until 2026-07-08 19:24:00.101842848 +0000 UTC m=+5.988032892 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-5vc5r" (UniqueName: "kubernetes.io/projected/60e1939e-8740-43f5-8d7b-056635c9f877-kube-api-access-5vc5r") pod "network-check-target-rs9hh" (UID: "60e1939e-8740-43f5-8d7b-056635c9f877") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 19:23:58.102250 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:23:58.101766 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 19:23:58.102250 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:23:58.102243 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24b6882e-3f1e-4c28-b128-2b16f3478940-original-pull-secret podName:24b6882e-3f1e-4c28-b128-2b16f3478940 nodeName:}" failed. No retries permitted until 2026-07-08 19:24:00.102226179 +0000 UTC m=+5.988416206 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/24b6882e-3f1e-4c28-b128-2b16f3478940-original-pull-secret") pod "global-pull-secret-syncer-qdfqb" (UID: "24b6882e-3f1e-4c28-b128-2b16f3478940") : object "kube-system"/"original-pull-secret" not registered Jul 08 19:23:58.202751 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:58.202708 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55818509-538d-4733-9e10-eaaf841ef071-metrics-certs\") pod \"network-metrics-daemon-rfbjf\" (UID: \"55818509-538d-4733-9e10-eaaf841ef071\") " pod="openshift-multus/network-metrics-daemon-rfbjf" Jul 08 19:23:58.202949 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:23:58.202881 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 19:23:58.202949 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:23:58.202949 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55818509-538d-4733-9e10-eaaf841ef071-metrics-certs podName:55818509-538d-4733-9e10-eaaf841ef071 nodeName:}" failed. No retries permitted until 2026-07-08 19:24:00.20292838 +0000 UTC m=+6.089118405 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55818509-538d-4733-9e10-eaaf841ef071-metrics-certs") pod "network-metrics-daemon-rfbjf" (UID: "55818509-538d-4733-9e10-eaaf841ef071") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 19:23:58.518442 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:58.518354 2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-07 19:18:56 +0000 UTC" deadline="2027-12-14 02:51:24.552009375 +0000 UTC" Jul 08 19:23:58.518442 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:58.518412 2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12559h27m26.033601775s" Jul 08 19:23:58.619779 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:58.619747 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rs9hh" Jul 08 19:23:58.620038 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:23:58.619898 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rs9hh" podUID="60e1939e-8740-43f5-8d7b-056635c9f877" Jul 08 19:23:58.620542 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:58.620521 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdfqb" Jul 08 19:23:58.620643 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:23:58.620623 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdfqb" podUID="24b6882e-3f1e-4c28-b128-2b16f3478940" Jul 08 19:23:58.620715 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:58.620698 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfbjf" Jul 08 19:23:58.620811 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:23:58.620795 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rfbjf" podUID="55818509-538d-4733-9e10-eaaf841ef071" Jul 08 19:23:58.671324 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:58.670569 2566 generic.go:356] "Generic (PLEG): container finished" podID="3dd85bb084b2f8eec4b6dcb61f820279" containerID="3f107107c4531039b9ad56c9a0c547e249d3177e1fd068999bbdadc9beec59af" exitCode=0 Jul 08 19:23:58.671324 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:58.671106 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-195.ec2.internal" event={"ID":"3dd85bb084b2f8eec4b6dcb61f820279","Type":"ContainerDied","Data":"3f107107c4531039b9ad56c9a0c547e249d3177e1fd068999bbdadc9beec59af"} Jul 08 19:23:59.707889 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:23:59.707224 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-195.ec2.internal" event={"ID":"3dd85bb084b2f8eec4b6dcb61f820279","Type":"ContainerStarted","Data":"746bc0135be31f88d3fb05626da7dfcea5a6d2d1ee31ba18bb530716674ae4fa"} Jul 08 19:24:00.124829 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:00.124742 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vc5r\" (UniqueName: \"kubernetes.io/projected/60e1939e-8740-43f5-8d7b-056635c9f877-kube-api-access-5vc5r\") pod \"network-check-target-rs9hh\" (UID: \"60e1939e-8740-43f5-8d7b-056635c9f877\") " pod="openshift-network-diagnostics/network-check-target-rs9hh" Jul 08 19:24:00.124829 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:00.124808 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24b6882e-3f1e-4c28-b128-2b16f3478940-original-pull-secret\") pod \"global-pull-secret-syncer-qdfqb\" (UID: \"24b6882e-3f1e-4c28-b128-2b16f3478940\") " pod="kube-system/global-pull-secret-syncer-qdfqb" Jul 08 19:24:00.125069 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:00.124957 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 19:24:00.125069 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:00.125026 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24b6882e-3f1e-4c28-b128-2b16f3478940-original-pull-secret podName:24b6882e-3f1e-4c28-b128-2b16f3478940 nodeName:}" failed. No retries permitted until 2026-07-08 19:24:04.125004881 +0000 UTC m=+10.011194920 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/24b6882e-3f1e-4c28-b128-2b16f3478940-original-pull-secret") pod "global-pull-secret-syncer-qdfqb" (UID: "24b6882e-3f1e-4c28-b128-2b16f3478940") : object "kube-system"/"original-pull-secret" not registered Jul 08 19:24:00.125470 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:00.125445 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 19:24:00.125581 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:00.125474 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 19:24:00.125581 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:00.125487 2566 projected.go:196] Error preparing data for projected volume kube-api-access-5vc5r for pod openshift-network-diagnostics/network-check-target-rs9hh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 19:24:00.125581 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:00.125535 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60e1939e-8740-43f5-8d7b-056635c9f877-kube-api-access-5vc5r podName:60e1939e-8740-43f5-8d7b-056635c9f877 nodeName:}" failed. No retries permitted until 2026-07-08 19:24:04.125519954 +0000 UTC m=+10.011709992 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-5vc5r" (UniqueName: "kubernetes.io/projected/60e1939e-8740-43f5-8d7b-056635c9f877-kube-api-access-5vc5r") pod "network-check-target-rs9hh" (UID: "60e1939e-8740-43f5-8d7b-056635c9f877") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 19:24:00.225444 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:00.225407 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55818509-538d-4733-9e10-eaaf841ef071-metrics-certs\") pod \"network-metrics-daemon-rfbjf\" (UID: \"55818509-538d-4733-9e10-eaaf841ef071\") " pod="openshift-multus/network-metrics-daemon-rfbjf" Jul 08 19:24:00.225631 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:00.225619 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 19:24:00.225689 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:00.225679 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55818509-538d-4733-9e10-eaaf841ef071-metrics-certs podName:55818509-538d-4733-9e10-eaaf841ef071 nodeName:}" failed. No retries permitted until 2026-07-08 19:24:04.225661707 +0000 UTC m=+10.111851735 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55818509-538d-4733-9e10-eaaf841ef071-metrics-certs") pod "network-metrics-daemon-rfbjf" (UID: "55818509-538d-4733-9e10-eaaf841ef071") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 19:24:00.618700 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:00.618049 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rs9hh" Jul 08 19:24:00.618700 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:00.618171 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rs9hh" podUID="60e1939e-8740-43f5-8d7b-056635c9f877" Jul 08 19:24:00.618700 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:00.618554 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdfqb" Jul 08 19:24:00.618700 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:00.618647 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdfqb" podUID="24b6882e-3f1e-4c28-b128-2b16f3478940" Jul 08 19:24:00.619321 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:00.619092 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfbjf" Jul 08 19:24:00.619321 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:00.619195 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rfbjf" podUID="55818509-538d-4733-9e10-eaaf841ef071" Jul 08 19:24:02.618231 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:02.618195 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfbjf" Jul 08 19:24:02.618231 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:02.618218 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rs9hh" Jul 08 19:24:02.618741 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:02.618354 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdfqb" Jul 08 19:24:02.618741 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:02.618351 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rfbjf" podUID="55818509-538d-4733-9e10-eaaf841ef071" Jul 08 19:24:02.618741 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:02.618512 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rs9hh" podUID="60e1939e-8740-43f5-8d7b-056635c9f877" Jul 08 19:24:02.618741 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:02.618614 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdfqb" podUID="24b6882e-3f1e-4c28-b128-2b16f3478940" Jul 08 19:24:04.157629 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:04.157593 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vc5r\" (UniqueName: \"kubernetes.io/projected/60e1939e-8740-43f5-8d7b-056635c9f877-kube-api-access-5vc5r\") pod \"network-check-target-rs9hh\" (UID: \"60e1939e-8740-43f5-8d7b-056635c9f877\") " pod="openshift-network-diagnostics/network-check-target-rs9hh" Jul 08 19:24:04.158054 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:04.157667 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24b6882e-3f1e-4c28-b128-2b16f3478940-original-pull-secret\") pod \"global-pull-secret-syncer-qdfqb\" (UID: \"24b6882e-3f1e-4c28-b128-2b16f3478940\") " pod="kube-system/global-pull-secret-syncer-qdfqb" Jul 08 19:24:04.158054 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:04.157775 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 19:24:04.158054 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:04.157786 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 19:24:04.158054 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:04.157795 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 19:24:04.158054 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:04.157805 2566 projected.go:196] Error preparing data for projected volume kube-api-access-5vc5r for pod openshift-network-diagnostics/network-check-target-rs9hh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 19:24:04.158054 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:04.157845 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24b6882e-3f1e-4c28-b128-2b16f3478940-original-pull-secret podName:24b6882e-3f1e-4c28-b128-2b16f3478940 nodeName:}" failed. No retries permitted until 2026-07-08 19:24:12.157826637 +0000 UTC m=+18.044016664 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/24b6882e-3f1e-4c28-b128-2b16f3478940-original-pull-secret") pod "global-pull-secret-syncer-qdfqb" (UID: "24b6882e-3f1e-4c28-b128-2b16f3478940") : object "kube-system"/"original-pull-secret" not registered Jul 08 19:24:04.158054 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:04.157880 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60e1939e-8740-43f5-8d7b-056635c9f877-kube-api-access-5vc5r podName:60e1939e-8740-43f5-8d7b-056635c9f877 nodeName:}" failed. No retries permitted until 2026-07-08 19:24:12.157854732 +0000 UTC m=+18.044044758 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-5vc5r" (UniqueName: "kubernetes.io/projected/60e1939e-8740-43f5-8d7b-056635c9f877-kube-api-access-5vc5r") pod "network-check-target-rs9hh" (UID: "60e1939e-8740-43f5-8d7b-056635c9f877") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 19:24:04.258892 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:04.258837 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55818509-538d-4733-9e10-eaaf841ef071-metrics-certs\") pod \"network-metrics-daemon-rfbjf\" (UID: \"55818509-538d-4733-9e10-eaaf841ef071\") " pod="openshift-multus/network-metrics-daemon-rfbjf" Jul 08 19:24:04.259082 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:04.259067 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 19:24:04.259157 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:04.259135 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55818509-538d-4733-9e10-eaaf841ef071-metrics-certs podName:55818509-538d-4733-9e10-eaaf841ef071 nodeName:}" failed. No retries permitted until 2026-07-08 19:24:12.259116111 +0000 UTC m=+18.145306138 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55818509-538d-4733-9e10-eaaf841ef071-metrics-certs") pod "network-metrics-daemon-rfbjf" (UID: "55818509-538d-4733-9e10-eaaf841ef071") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 19:24:04.619592 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:04.618834 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rs9hh" Jul 08 19:24:04.619592 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:04.618961 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rs9hh" podUID="60e1939e-8740-43f5-8d7b-056635c9f877" Jul 08 19:24:04.619592 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:04.619308 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdfqb" Jul 08 19:24:04.619592 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:04.619393 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdfqb" podUID="24b6882e-3f1e-4c28-b128-2b16f3478940" Jul 08 19:24:04.619592 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:04.619436 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfbjf" Jul 08 19:24:04.619592 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:04.619514 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rfbjf" podUID="55818509-538d-4733-9e10-eaaf841ef071" Jul 08 19:24:06.617983 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:06.617954 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rs9hh" Jul 08 19:24:06.618438 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:06.618071 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rs9hh" podUID="60e1939e-8740-43f5-8d7b-056635c9f877" Jul 08 19:24:06.618523 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:06.618498 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdfqb" Jul 08 19:24:06.618620 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:06.618594 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdfqb" podUID="24b6882e-3f1e-4c28-b128-2b16f3478940" Jul 08 19:24:06.618792 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:06.618682 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfbjf" Jul 08 19:24:06.618895 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:06.618814 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rfbjf" podUID="55818509-538d-4733-9e10-eaaf841ef071" Jul 08 19:24:08.617976 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:08.617941 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rs9hh" Jul 08 19:24:08.617976 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:08.617977 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdfqb" Jul 08 19:24:08.618497 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:08.617941 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfbjf" Jul 08 19:24:08.618497 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:08.618083 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rs9hh" podUID="60e1939e-8740-43f5-8d7b-056635c9f877" Jul 08 19:24:08.618497 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:08.618137 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdfqb" podUID="24b6882e-3f1e-4c28-b128-2b16f3478940" Jul 08 19:24:08.618497 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:08.618218 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rfbjf" podUID="55818509-538d-4733-9e10-eaaf841ef071" Jul 08 19:24:10.617321 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:10.617283 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdfqb" Jul 08 19:24:10.617769 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:10.617287 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfbjf" Jul 08 19:24:10.617769 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:10.617417 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdfqb" podUID="24b6882e-3f1e-4c28-b128-2b16f3478940" Jul 08 19:24:10.617769 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:10.617431 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rs9hh" Jul 08 19:24:10.617769 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:10.617522 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rfbjf" podUID="55818509-538d-4733-9e10-eaaf841ef071" Jul 08 19:24:10.617769 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:10.617598 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rs9hh" podUID="60e1939e-8740-43f5-8d7b-056635c9f877" Jul 08 19:24:12.217561 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:12.217528 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vc5r\" (UniqueName: \"kubernetes.io/projected/60e1939e-8740-43f5-8d7b-056635c9f877-kube-api-access-5vc5r\") pod \"network-check-target-rs9hh\" (UID: \"60e1939e-8740-43f5-8d7b-056635c9f877\") " pod="openshift-network-diagnostics/network-check-target-rs9hh" Jul 08 19:24:12.218043 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:12.217580 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24b6882e-3f1e-4c28-b128-2b16f3478940-original-pull-secret\") pod \"global-pull-secret-syncer-qdfqb\" (UID: \"24b6882e-3f1e-4c28-b128-2b16f3478940\") " pod="kube-system/global-pull-secret-syncer-qdfqb" Jul 08 19:24:12.218043 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:12.217690 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 19:24:12.218043 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:12.217710 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 19:24:12.218043 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:12.217733 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 19:24:12.218043 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:12.217744 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24b6882e-3f1e-4c28-b128-2b16f3478940-original-pull-secret podName:24b6882e-3f1e-4c28-b128-2b16f3478940 nodeName:}" failed. No retries permitted until 2026-07-08 19:24:28.217730032 +0000 UTC m=+34.103920057 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/24b6882e-3f1e-4c28-b128-2b16f3478940-original-pull-secret") pod "global-pull-secret-syncer-qdfqb" (UID: "24b6882e-3f1e-4c28-b128-2b16f3478940") : object "kube-system"/"original-pull-secret" not registered Jul 08 19:24:12.218043 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:12.217744 2566 projected.go:196] Error preparing data for projected volume kube-api-access-5vc5r for pod openshift-network-diagnostics/network-check-target-rs9hh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 19:24:12.218043 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:12.217772 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60e1939e-8740-43f5-8d7b-056635c9f877-kube-api-access-5vc5r podName:60e1939e-8740-43f5-8d7b-056635c9f877 nodeName:}" failed. No retries permitted until 2026-07-08 19:24:28.217766991 +0000 UTC m=+34.103957015 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-5vc5r" (UniqueName: "kubernetes.io/projected/60e1939e-8740-43f5-8d7b-056635c9f877-kube-api-access-5vc5r") pod "network-check-target-rs9hh" (UID: "60e1939e-8740-43f5-8d7b-056635c9f877") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 19:24:12.317923 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:12.317883 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55818509-538d-4733-9e10-eaaf841ef071-metrics-certs\") pod \"network-metrics-daemon-rfbjf\" (UID: \"55818509-538d-4733-9e10-eaaf841ef071\") " pod="openshift-multus/network-metrics-daemon-rfbjf" Jul 08 19:24:12.318070 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:12.318026 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 19:24:12.318136 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:12.318081 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55818509-538d-4733-9e10-eaaf841ef071-metrics-certs podName:55818509-538d-4733-9e10-eaaf841ef071 nodeName:}" failed. No retries permitted until 2026-07-08 19:24:28.318064042 +0000 UTC m=+34.204254088 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55818509-538d-4733-9e10-eaaf841ef071-metrics-certs") pod "network-metrics-daemon-rfbjf" (UID: "55818509-538d-4733-9e10-eaaf841ef071") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 19:24:12.617223 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:12.617154 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdfqb" Jul 08 19:24:12.617371 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:12.617158 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfbjf" Jul 08 19:24:12.617371 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:12.617274 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdfqb" podUID="24b6882e-3f1e-4c28-b128-2b16f3478940" Jul 08 19:24:12.617519 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:12.617369 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rfbjf" podUID="55818509-538d-4733-9e10-eaaf841ef071" Jul 08 19:24:12.617519 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:12.617149 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rs9hh" Jul 08 19:24:12.617519 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:12.617481 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rs9hh" podUID="60e1939e-8740-43f5-8d7b-056635c9f877" Jul 08 19:24:14.618795 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:14.618752 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfbjf" Jul 08 19:24:14.619251 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:14.618911 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rs9hh" Jul 08 19:24:14.619251 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:14.618933 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdfqb" Jul 08 19:24:14.619251 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:14.618913 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rfbjf" podUID="55818509-538d-4733-9e10-eaaf841ef071" Jul 08 19:24:14.619251 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:14.619025 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rs9hh" podUID="60e1939e-8740-43f5-8d7b-056635c9f877" Jul 08 19:24:14.619251 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:14.619109 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdfqb" podUID="24b6882e-3f1e-4c28-b128-2b16f3478940" Jul 08 19:24:16.617727 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:16.617455 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rs9hh" Jul 08 19:24:16.618383 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:16.617461 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdfqb" Jul 08 19:24:16.618383 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:16.617804 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rs9hh" podUID="60e1939e-8740-43f5-8d7b-056635c9f877" Jul 08 19:24:16.618383 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:16.617529 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfbjf" Jul 08 19:24:16.618383 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:16.617952 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdfqb" podUID="24b6882e-3f1e-4c28-b128-2b16f3478940" Jul 08 19:24:16.618383 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:16.618059 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rfbjf" podUID="55818509-538d-4733-9e10-eaaf841ef071" Jul 08 19:24:16.743244 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:16.743215 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gz54s" event={"ID":"3a7da2be-d237-4bfa-887d-0be7632bfdc0","Type":"ContainerStarted","Data":"3a8d53af96a10cbc56f26a8f0322d590ab65319048549506c32bbff573358e71"} Jul 08 19:24:16.744607 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:16.744586 2566 generic.go:356] "Generic (PLEG): container finished" podID="dbe0d11c-c9c5-4cdb-ad9b-08b142e7335f" containerID="267f564314ab7c7c6d6ec3cc8417937ad6b239ff7aa8865d733b42ea4f39df17" exitCode=0 Jul 08 19:24:16.744712 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:16.744652 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-srmfq" event={"ID":"dbe0d11c-c9c5-4cdb-ad9b-08b142e7335f","Type":"ContainerDied","Data":"267f564314ab7c7c6d6ec3cc8417937ad6b239ff7aa8865d733b42ea4f39df17"} Jul 08 19:24:16.747841 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:16.747813 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cdp8n" event={"ID":"3af16262-f54e-4683-b76c-e1ccfdff4c3c","Type":"ContainerStarted","Data":"a5f2eceff5d57a5e2e015ee1efe824fd99fa9d0b682c481086ac34616c36b3be"} Jul 08 19:24:16.747950 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:16.747924 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cdp8n" event={"ID":"3af16262-f54e-4683-b76c-e1ccfdff4c3c","Type":"ContainerStarted","Data":"1dd0ee6cfccc5dd8f25f4917619a30cc0e360d46a72ef43e84ba1a508e357ccb"} Jul 08 19:24:16.748009 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:16.747985 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cdp8n" event={"ID":"3af16262-f54e-4683-b76c-e1ccfdff4c3c","Type":"ContainerStarted","Data":"7275be271df369b2c1bf2aa59642a976443b09d6188018a1267eeab7668282d6"} Jul 08 19:24:16.748071 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:16.748039 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cdp8n" event={"ID":"3af16262-f54e-4683-b76c-e1ccfdff4c3c","Type":"ContainerStarted","Data":"81449949a6a145d8cc72d33355b5f3d4f37dea4df7d53c6cd9409ced4c967c0c"} Jul 08 19:24:16.748116 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:16.748089 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cdp8n" event={"ID":"3af16262-f54e-4683-b76c-e1ccfdff4c3c","Type":"ContainerStarted","Data":"80e0c64b8289de2630699358c6b5deea182e478c7cd7b2b40dfe1515fa117a7d"} Jul 08 19:24:16.748207 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:16.748138 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cdp8n" event={"ID":"3af16262-f54e-4683-b76c-e1ccfdff4c3c","Type":"ContainerStarted","Data":"8bba10d38df1bd6e5d8d09dea69660855873caa9bb5a383506bf2b8ea3577af0"} Jul 08 19:24:16.749295 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:16.749274 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6w52p" event={"ID":"8b78ed73-731b-49fa-9220-4b1e45787806","Type":"ContainerStarted","Data":"deb816b8e5bd6590c4b58d159433a2106e48ac0a42978228fa9ca094d31a5895"} Jul 08 19:24:16.750752 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:16.750722 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5kt6t" event={"ID":"42578ec4-91d1-4d4d-b9a1-12a559fee154","Type":"ContainerStarted","Data":"e801f1bf8774d5fc46e0c2f8ee5153fb5fc1d142c2d3de6226c5850186da4e87"} Jul 08 19:24:16.752768 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:16.752750 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xb4w9" event={"ID":"01c82975-c67d-4e46-b16a-f558032f5dc3","Type":"ContainerStarted","Data":"ba0a49fb719176dece3e99e92517f859c6b759dfa50c35169c76d7512439af6f"} Jul 08 19:24:16.754615 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:16.754595 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cwcd" event={"ID":"30762972-ab72-4cdd-a693-584f9c7cdde5","Type":"ContainerStarted","Data":"49a7e5202fedb1e46ac160d377f25981c2cb14f32438ab44a3ffb680383132be"} Jul 08 19:24:16.755966 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:16.755944 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dghh8" event={"ID":"474e5893-2b36-4b57-b154-15b0f8b4f300","Type":"ContainerStarted","Data":"35a6c971aa1f7c99851b392edca44085e5b554bd1c3832718d0d6c5374daafee"} Jul 08 19:24:16.760997 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:16.760960 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-195.ec2.internal" podStartSLOduration=20.760949004 podStartE2EDuration="20.760949004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 19:23:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:23:59.724572338 +0000 UTC m=+5.610762386" watchObservedRunningTime="2026-07-08 19:24:16.760949004 +0000 UTC m=+22.647139051" Jul 08 19:24:16.761099 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:16.761081 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-gz54s" podStartSLOduration=4.22116336 podStartE2EDuration="22.761077737s" totalImagesPullingTime="18.539914377s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 19:23:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:24:16.76071651 +0000 UTC m=+22.646906561" watchObservedRunningTime="2026-07-08 19:24:16.761077737 +0000 UTC m=+22.647267783" Jul 08 19:24:16.791669 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:16.791617 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-dghh8" podStartSLOduration=4.307775859 podStartE2EDuration="22.791600258s" totalImagesPullingTime="18.483824399s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 19:23:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:24:16.774847399 +0000 UTC m=+22.661037448" watchObservedRunningTime="2026-07-08 19:24:16.791600258 +0000 UTC m=+22.677790303" Jul 08 19:24:16.792143 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:16.792098 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-5kt6t" podStartSLOduration=8.50314288 podStartE2EDuration="22.792067616s" totalImagesPullingTime="14.288924736s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:23:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:24:16.791558095 +0000 UTC m=+22.677748146" watchObservedRunningTime="2026-07-08 19:24:16.792067616 +0000 UTC m=+22.678257662" Jul 08 19:24:16.829813 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:16.829770 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-6w52p" podStartSLOduration=4.358890547 podStartE2EDuration="22.829757858s" totalImagesPullingTime="18.470867311s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 19:23:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:24:16.829440617 +0000 UTC m=+22.715630663" watchObservedRunningTime="2026-07-08 19:24:16.829757858 +0000 UTC m=+22.715947905" Jul 08 19:24:16.845645 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:16.845609 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-xb4w9" podStartSLOduration=4.364847083 podStartE2EDuration="22.84559596s" totalImagesPullingTime="18.480748877s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 19:23:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:24:16.84526892 +0000 UTC m=+22.731458968" watchObservedRunningTime="2026-07-08 19:24:16.84559596 +0000 UTC m=+22.731786007" Jul 08 19:24:17.760343 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:17.760308 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bkjrq" event={"ID":"bbe4cd13-7412-42f1-9035-865ce20ba987","Type":"ContainerStarted","Data":"3aadf531257e0219af95ec0d411b2231bd4a6b545f0db97fcdfe0a58a6086846"} Jul 08 19:24:17.775697 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:17.775677 2566 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 08 19:24:17.777215 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:17.777174 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-bkjrq" podStartSLOduration=5.330571742 podStartE2EDuration="23.777158059s" totalImagesPullingTime="18.446586317s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 19:23:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:24:17.776589771 +0000 UTC m=+23.662779818" watchObservedRunningTime="2026-07-08 19:24:17.777158059 +0000 UTC m=+23.663348105" Jul 08 19:24:17.999061 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:17.998982 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-5kt6t" Jul 08 19:24:17.999506 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:17.999487 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-5kt6t" Jul 08 19:24:18.550942 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:18.550834 2566 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-08T19:24:17.77569305Z","UUID":"3af07b0c-fb39-42db-b468-ac1922259082","Handler":null,"Name":"","Endpoint":""} Jul 08 19:24:18.552360 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:18.552337 2566 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 08 19:24:18.552475 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:18.552378 2566 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 08 19:24:18.618180 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:18.618151 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rs9hh" Jul 08 19:24:18.618180 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:18.618152 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdfqb" Jul 08 19:24:18.618391 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:18.618202 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfbjf" Jul 08 19:24:18.618391 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:18.618272 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdfqb" podUID="24b6882e-3f1e-4c28-b128-2b16f3478940" Jul 08 19:24:18.618391 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:18.618315 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rfbjf" podUID="55818509-538d-4733-9e10-eaaf841ef071" Jul 08 19:24:18.618391 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:18.618330 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rs9hh" podUID="60e1939e-8740-43f5-8d7b-056635c9f877" Jul 08 19:24:18.764847 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:18.764815 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cwcd" event={"ID":"30762972-ab72-4cdd-a693-584f9c7cdde5","Type":"ContainerStarted","Data":"67ec3ce59dedd2a97d7f916dbf97e476e369c13cb459b32c9eed21113d4583b7"} Jul 08 19:24:18.765334 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:18.765136 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-5kt6t" Jul 08 19:24:18.765495 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:18.765477 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5kt6t" Jul 08 19:24:19.770240 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:19.770066 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cdp8n" event={"ID":"3af16262-f54e-4683-b76c-e1ccfdff4c3c","Type":"ContainerStarted","Data":"626a716cd618e2fb70d7452dcb09c900227a0f3b8af6c7e72933464afec95ee0"} Jul 08 19:24:19.772030 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:19.771998 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cwcd" event={"ID":"30762972-ab72-4cdd-a693-584f9c7cdde5","Type":"ContainerStarted","Data":"8526e736fbcc6e435c23c1b2169fce5a5733783807053a95c38fefdb298a41ad"} Jul 08 19:24:20.618331 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:20.618295 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rs9hh" Jul 08 19:24:20.618331 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:20.618321 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdfqb" Jul 08 19:24:20.618927 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:20.618431 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfbjf" Jul 08 19:24:20.618927 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:20.618432 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rs9hh" podUID="60e1939e-8740-43f5-8d7b-056635c9f877" Jul 08 19:24:20.618927 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:20.618531 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdfqb" podUID="24b6882e-3f1e-4c28-b128-2b16f3478940" Jul 08 19:24:20.618927 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:20.618608 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rfbjf" podUID="55818509-538d-4733-9e10-eaaf841ef071" Jul 08 19:24:22.618170 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:22.617994 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rs9hh" Jul 08 19:24:22.618958 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:22.617994 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdfqb" Jul 08 19:24:22.618958 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:22.618254 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rs9hh" podUID="60e1939e-8740-43f5-8d7b-056635c9f877" Jul 08 19:24:22.618958 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:22.618045 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfbjf" Jul 08 19:24:22.618958 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:22.618309 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdfqb" podUID="24b6882e-3f1e-4c28-b128-2b16f3478940" Jul 08 19:24:22.618958 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:22.618405 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rfbjf" podUID="55818509-538d-4733-9e10-eaaf841ef071" Jul 08 19:24:22.781026 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:22.780954 2566 generic.go:356] "Generic (PLEG): container finished" podID="dbe0d11c-c9c5-4cdb-ad9b-08b142e7335f" containerID="37c51538f8b1625e68c899475033d45f383c5255a121b8b06bedc2b149e06892" exitCode=0 Jul 08 19:24:22.781153 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:22.781040 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-srmfq" event={"ID":"dbe0d11c-c9c5-4cdb-ad9b-08b142e7335f","Type":"ContainerDied","Data":"37c51538f8b1625e68c899475033d45f383c5255a121b8b06bedc2b149e06892"} Jul 08 19:24:22.784760 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:22.784736 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cdp8n" event={"ID":"3af16262-f54e-4683-b76c-e1ccfdff4c3c","Type":"ContainerStarted","Data":"51bba31dd4416b81367d56641d623be85192fc5f0c160768b98b93cdc62be61d"} Jul 08 19:24:22.785049 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:22.785025 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cdp8n" Jul 08 19:24:22.785216 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:22.785090 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cdp8n" Jul 08 19:24:22.800459 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:22.800438 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cdp8n" Jul 08 19:24:22.809503 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:22.809469 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cwcd" podStartSLOduration=7.427412912 podStartE2EDuration="28.809457218s" totalImagesPullingTime="21.382044306s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 19:23:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:24:19.788163066 +0000 UTC m=+25.674353113" watchObservedRunningTime="2026-07-08 19:24:22.809457218 +0000 UTC m=+28.695647264" Jul 08 19:24:22.835326 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:22.835292 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-cdp8n" podStartSLOduration=10.275590828 podStartE2EDuration="28.835280334s" totalImagesPullingTime="18.559689506s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 19:23:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:24:22.83358436 +0000 UTC m=+28.719774408" watchObservedRunningTime="2026-07-08 19:24:22.835280334 +0000 UTC m=+28.721470380" Jul 08 19:24:23.787590 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:23.787562 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cdp8n" Jul 08 19:24:23.806307 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:23.806284 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cdp8n" Jul 08 19:24:24.190673 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:24.190641 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qdfqb"] Jul 08 19:24:24.190821 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:24.190766 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdfqb" Jul 08 19:24:24.190944 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:24.190918 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdfqb" podUID="24b6882e-3f1e-4c28-b128-2b16f3478940" Jul 08 19:24:24.193809 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:24.193782 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rs9hh"] Jul 08 19:24:24.193943 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:24.193899 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rs9hh" Jul 08 19:24:24.194017 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:24.193996 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rs9hh" podUID="60e1939e-8740-43f5-8d7b-056635c9f877" Jul 08 19:24:24.194387 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:24.194365 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rfbjf"] Jul 08 19:24:24.194468 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:24.194454 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfbjf" Jul 08 19:24:24.194560 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:24.194543 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rfbjf" podUID="55818509-538d-4733-9e10-eaaf841ef071" Jul 08 19:24:24.791383 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:24.791343 2566 generic.go:356] "Generic (PLEG): container finished" podID="dbe0d11c-c9c5-4cdb-ad9b-08b142e7335f" containerID="b1e2ee55f7d3897e86424fb91de27c0c5db6cefbf75463c9c63831410049c50f" exitCode=0 Jul 08 19:24:24.791791 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:24.791437 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-srmfq" event={"ID":"dbe0d11c-c9c5-4cdb-ad9b-08b142e7335f","Type":"ContainerDied","Data":"b1e2ee55f7d3897e86424fb91de27c0c5db6cefbf75463c9c63831410049c50f"} Jul 08 19:24:25.618070 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:25.617902 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfbjf" Jul 08 19:24:25.618216 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:25.617903 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rs9hh" Jul 08 19:24:25.618216 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:25.617910 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdfqb" Jul 08 19:24:25.618216 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:25.618202 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rfbjf" podUID="55818509-538d-4733-9e10-eaaf841ef071" Jul 08 19:24:25.618386 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:25.618264 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rs9hh" podUID="60e1939e-8740-43f5-8d7b-056635c9f877" Jul 08 19:24:25.618386 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:25.618346 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdfqb" podUID="24b6882e-3f1e-4c28-b128-2b16f3478940" Jul 08 19:24:26.798625 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:26.798596 2566 generic.go:356] "Generic (PLEG): container finished" podID="dbe0d11c-c9c5-4cdb-ad9b-08b142e7335f" containerID="685ec0f1862266ae94751388f5d096921c8ca2f75171bbb9ef35e951093e44e8" exitCode=0 Jul 08 19:24:26.799153 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:26.798640 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-srmfq" event={"ID":"dbe0d11c-c9c5-4cdb-ad9b-08b142e7335f","Type":"ContainerDied","Data":"685ec0f1862266ae94751388f5d096921c8ca2f75171bbb9ef35e951093e44e8"} Jul 08 19:24:26.866674 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:26.866637 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:24:26.949140 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:26.949096 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:24:27.617532 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:27.617498 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfbjf" Jul 08 19:24:27.617706 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:27.617498 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdfqb" Jul 08 19:24:27.617706 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:27.617599 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rfbjf" podUID="55818509-538d-4733-9e10-eaaf841ef071" Jul 08 19:24:27.617706 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:27.617623 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rs9hh" Jul 08 19:24:27.617706 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:27.617699 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdfqb" podUID="24b6882e-3f1e-4c28-b128-2b16f3478940" Jul 08 19:24:27.617916 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:27.617775 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rs9hh" podUID="60e1939e-8740-43f5-8d7b-056635c9f877" Jul 08 19:24:27.794045 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:27.794007 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:24:27.857211 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:27.857185 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:24:28.239613 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:28.239572 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vc5r\" (UniqueName: \"kubernetes.io/projected/60e1939e-8740-43f5-8d7b-056635c9f877-kube-api-access-5vc5r\") pod \"network-check-target-rs9hh\" (UID: \"60e1939e-8740-43f5-8d7b-056635c9f877\") " pod="openshift-network-diagnostics/network-check-target-rs9hh" Jul 08 19:24:28.239771 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:28.239633 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24b6882e-3f1e-4c28-b128-2b16f3478940-original-pull-secret\") pod \"global-pull-secret-syncer-qdfqb\" (UID: \"24b6882e-3f1e-4c28-b128-2b16f3478940\") " pod="kube-system/global-pull-secret-syncer-qdfqb" Jul 08 19:24:28.239771 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:28.239736 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 19:24:28.239771 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:28.239766 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 19:24:28.239940 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:28.239791 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 19:24:28.239940 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:28.239806 2566 projected.go:196] Error preparing data for projected volume kube-api-access-5vc5r for pod openshift-network-diagnostics/network-check-target-rs9hh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 19:24:28.239940 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:28.239791 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24b6882e-3f1e-4c28-b128-2b16f3478940-original-pull-secret podName:24b6882e-3f1e-4c28-b128-2b16f3478940 nodeName:}" failed. No retries permitted until 2026-07-08 19:25:00.239775658 +0000 UTC m=+66.125965683 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/24b6882e-3f1e-4c28-b128-2b16f3478940-original-pull-secret") pod "global-pull-secret-syncer-qdfqb" (UID: "24b6882e-3f1e-4c28-b128-2b16f3478940") : object "kube-system"/"original-pull-secret" not registered Jul 08 19:24:28.239940 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:28.239885 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60e1939e-8740-43f5-8d7b-056635c9f877-kube-api-access-5vc5r podName:60e1939e-8740-43f5-8d7b-056635c9f877 nodeName:}" failed. No retries permitted until 2026-07-08 19:25:00.239851994 +0000 UTC m=+66.126042025 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-5vc5r" (UniqueName: "kubernetes.io/projected/60e1939e-8740-43f5-8d7b-056635c9f877-kube-api-access-5vc5r") pod "network-check-target-rs9hh" (UID: "60e1939e-8740-43f5-8d7b-056635c9f877") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 19:24:28.340217 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:28.340184 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55818509-538d-4733-9e10-eaaf841ef071-metrics-certs\") pod \"network-metrics-daemon-rfbjf\" (UID: \"55818509-538d-4733-9e10-eaaf841ef071\") " pod="openshift-multus/network-metrics-daemon-rfbjf" Jul 08 19:24:28.340364 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:28.340309 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 19:24:28.340364 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:28.340356 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55818509-538d-4733-9e10-eaaf841ef071-metrics-certs podName:55818509-538d-4733-9e10-eaaf841ef071 nodeName:}" failed. No retries permitted until 2026-07-08 19:25:00.340343074 +0000 UTC m=+66.226533099 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55818509-538d-4733-9e10-eaaf841ef071-metrics-certs") pod "network-metrics-daemon-rfbjf" (UID: "55818509-538d-4733-9e10-eaaf841ef071") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 19:24:28.854981 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:28.854907 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-195.ec2.internal" event="NodeReady" Jul 08 19:24:28.855144 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:28.855017 2566 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 08 19:24:28.894448 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:28.894419 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-npv6z"] Jul 08 19:24:28.896901 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:28.896855 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-npv6z" Jul 08 19:24:28.897410 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:28.897387 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-5nfnp"] Jul 08 19:24:28.899642 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:28.899147 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5nfnp" Jul 08 19:24:28.899642 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:28.899211 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dclc4\"" Jul 08 19:24:28.899642 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:28.899211 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 08 19:24:28.899642 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:28.899370 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 08 19:24:28.902023 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:28.901855 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 08 19:24:28.902023 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:28.901855 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-qgtw2\"" Jul 08 19:24:28.902023 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:28.901949 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 08 19:24:28.902452 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:28.902435 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 08 19:24:28.908255 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:28.908195 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-npv6z"] Jul 08 19:24:28.909944 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:28.909885 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5nfnp"] Jul 08 19:24:29.046046 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:29.046010 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e2bd191d-a59d-4dbe-811f-44e2c4ca16e1-config-volume\") pod \"dns-default-npv6z\" (UID: \"e2bd191d-a59d-4dbe-811f-44e2c4ca16e1\") " pod="openshift-dns/dns-default-npv6z" Jul 08 19:24:29.046214 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:29.046071 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5plwr\" (UniqueName: \"kubernetes.io/projected/e2bd191d-a59d-4dbe-811f-44e2c4ca16e1-kube-api-access-5plwr\") pod \"dns-default-npv6z\" (UID: \"e2bd191d-a59d-4dbe-811f-44e2c4ca16e1\") " pod="openshift-dns/dns-default-npv6z" Jul 08 19:24:29.046214 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:29.046100 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d36b2cab-e4d2-4a23-85f9-0bd24ccbb497-cert\") pod \"ingress-canary-5nfnp\" (UID: \"d36b2cab-e4d2-4a23-85f9-0bd24ccbb497\") " pod="openshift-ingress-canary/ingress-canary-5nfnp" Jul 08 19:24:29.046214 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:29.046125 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2bd191d-a59d-4dbe-811f-44e2c4ca16e1-metrics-tls\") pod \"dns-default-npv6z\" (UID: \"e2bd191d-a59d-4dbe-811f-44e2c4ca16e1\") " pod="openshift-dns/dns-default-npv6z" Jul 08 19:24:29.046214 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:29.046151 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5247\" (UniqueName: \"kubernetes.io/projected/d36b2cab-e4d2-4a23-85f9-0bd24ccbb497-kube-api-access-x5247\") pod \"ingress-canary-5nfnp\" (UID: \"d36b2cab-e4d2-4a23-85f9-0bd24ccbb497\") " pod="openshift-ingress-canary/ingress-canary-5nfnp" Jul 08 19:24:29.046391 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:29.046241 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2bd191d-a59d-4dbe-811f-44e2c4ca16e1-tmp-dir\") pod \"dns-default-npv6z\" (UID: \"e2bd191d-a59d-4dbe-811f-44e2c4ca16e1\") " pod="openshift-dns/dns-default-npv6z" Jul 08 19:24:29.147349 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:29.147273 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2bd191d-a59d-4dbe-811f-44e2c4ca16e1-tmp-dir\") pod \"dns-default-npv6z\" (UID: \"e2bd191d-a59d-4dbe-811f-44e2c4ca16e1\") " pod="openshift-dns/dns-default-npv6z" Jul 08 19:24:29.147349 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:29.147325 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e2bd191d-a59d-4dbe-811f-44e2c4ca16e1-config-volume\") pod \"dns-default-npv6z\" (UID: \"e2bd191d-a59d-4dbe-811f-44e2c4ca16e1\") " pod="openshift-dns/dns-default-npv6z" Jul 08 19:24:29.147527 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:29.147377 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5plwr\" (UniqueName: \"kubernetes.io/projected/e2bd191d-a59d-4dbe-811f-44e2c4ca16e1-kube-api-access-5plwr\") pod \"dns-default-npv6z\" (UID: \"e2bd191d-a59d-4dbe-811f-44e2c4ca16e1\") " pod="openshift-dns/dns-default-npv6z" Jul 08 19:24:29.147527 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:29.147401 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d36b2cab-e4d2-4a23-85f9-0bd24ccbb497-cert\") pod \"ingress-canary-5nfnp\" (UID: \"d36b2cab-e4d2-4a23-85f9-0bd24ccbb497\") " pod="openshift-ingress-canary/ingress-canary-5nfnp" Jul 08 19:24:29.147527 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:29.147435 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2bd191d-a59d-4dbe-811f-44e2c4ca16e1-metrics-tls\") pod \"dns-default-npv6z\" (UID: \"e2bd191d-a59d-4dbe-811f-44e2c4ca16e1\") " pod="openshift-dns/dns-default-npv6z" Jul 08 19:24:29.147527 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:29.147463 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5247\" (UniqueName: \"kubernetes.io/projected/d36b2cab-e4d2-4a23-85f9-0bd24ccbb497-kube-api-access-x5247\") pod \"ingress-canary-5nfnp\" (UID: \"d36b2cab-e4d2-4a23-85f9-0bd24ccbb497\") " pod="openshift-ingress-canary/ingress-canary-5nfnp" Jul 08 19:24:29.147717 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:29.147535 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 19:24:29.147717 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:29.147608 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2bd191d-a59d-4dbe-811f-44e2c4ca16e1-metrics-tls podName:e2bd191d-a59d-4dbe-811f-44e2c4ca16e1 nodeName:}" failed. No retries permitted until 2026-07-08 19:24:29.647587776 +0000 UTC m=+35.533777806 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e2bd191d-a59d-4dbe-811f-44e2c4ca16e1-metrics-tls") pod "dns-default-npv6z" (UID: "e2bd191d-a59d-4dbe-811f-44e2c4ca16e1") : secret "dns-default-metrics-tls" not found Jul 08 19:24:29.147717 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:29.147693 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2bd191d-a59d-4dbe-811f-44e2c4ca16e1-tmp-dir\") pod \"dns-default-npv6z\" (UID: \"e2bd191d-a59d-4dbe-811f-44e2c4ca16e1\") " pod="openshift-dns/dns-default-npv6z" Jul 08 19:24:29.147940 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:29.147923 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e2bd191d-a59d-4dbe-811f-44e2c4ca16e1-config-volume\") pod \"dns-default-npv6z\" (UID: \"e2bd191d-a59d-4dbe-811f-44e2c4ca16e1\") " pod="openshift-dns/dns-default-npv6z" Jul 08 19:24:29.151524 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:29.151497 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d36b2cab-e4d2-4a23-85f9-0bd24ccbb497-cert\") pod \"ingress-canary-5nfnp\" (UID: \"d36b2cab-e4d2-4a23-85f9-0bd24ccbb497\") " pod="openshift-ingress-canary/ingress-canary-5nfnp" Jul 08 19:24:29.157769 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:29.157747 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5plwr\" (UniqueName: \"kubernetes.io/projected/e2bd191d-a59d-4dbe-811f-44e2c4ca16e1-kube-api-access-5plwr\") pod \"dns-default-npv6z\" (UID: \"e2bd191d-a59d-4dbe-811f-44e2c4ca16e1\") " pod="openshift-dns/dns-default-npv6z" Jul 08 19:24:29.158124 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:29.158106 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5247\" (UniqueName: \"kubernetes.io/projected/d36b2cab-e4d2-4a23-85f9-0bd24ccbb497-kube-api-access-x5247\") pod \"ingress-canary-5nfnp\" (UID: \"d36b2cab-e4d2-4a23-85f9-0bd24ccbb497\") " pod="openshift-ingress-canary/ingress-canary-5nfnp" Jul 08 19:24:29.217256 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:29.217225 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5nfnp" Jul 08 19:24:29.373292 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:29.373081 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5nfnp"] Jul 08 19:24:29.378421 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:24:29.378389 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd36b2cab_e4d2_4a23_85f9_0bd24ccbb497.slice/crio-f33bc00d8d9a57a5217a3e01e6edd35c948b32df54eb3c0d4302c331fe03ad56 WatchSource:0}: Error finding container f33bc00d8d9a57a5217a3e01e6edd35c948b32df54eb3c0d4302c331fe03ad56: Status 404 returned error can't find the container with id f33bc00d8d9a57a5217a3e01e6edd35c948b32df54eb3c0d4302c331fe03ad56 Jul 08 19:24:29.617668 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:29.617637 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rs9hh" Jul 08 19:24:29.617668 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:29.617666 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdfqb" Jul 08 19:24:29.617942 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:29.617638 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfbjf" Jul 08 19:24:29.620386 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:29.620357 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 08 19:24:29.621998 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:29.621810 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hgx5c\"" Jul 08 19:24:29.622108 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:29.621997 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 08 19:24:29.622179 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:29.622143 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 08 19:24:29.623087 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:29.622668 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-98h84\"" Jul 08 19:24:29.623087 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:29.622961 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 08 19:24:29.651544 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:29.651523 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2bd191d-a59d-4dbe-811f-44e2c4ca16e1-metrics-tls\") pod \"dns-default-npv6z\" (UID: \"e2bd191d-a59d-4dbe-811f-44e2c4ca16e1\") " pod="openshift-dns/dns-default-npv6z" Jul 08 19:24:29.651645 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:29.651633 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 19:24:29.651704 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:29.651693 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2bd191d-a59d-4dbe-811f-44e2c4ca16e1-metrics-tls podName:e2bd191d-a59d-4dbe-811f-44e2c4ca16e1 nodeName:}" failed. No retries permitted until 2026-07-08 19:24:30.651676678 +0000 UTC m=+36.537866706 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e2bd191d-a59d-4dbe-811f-44e2c4ca16e1-metrics-tls") pod "dns-default-npv6z" (UID: "e2bd191d-a59d-4dbe-811f-44e2c4ca16e1") : secret "dns-default-metrics-tls" not found Jul 08 19:24:29.806611 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:29.806542 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5nfnp" event={"ID":"d36b2cab-e4d2-4a23-85f9-0bd24ccbb497","Type":"ContainerStarted","Data":"f33bc00d8d9a57a5217a3e01e6edd35c948b32df54eb3c0d4302c331fe03ad56"} Jul 08 19:24:30.659002 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:30.658959 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2bd191d-a59d-4dbe-811f-44e2c4ca16e1-metrics-tls\") pod \"dns-default-npv6z\" (UID: \"e2bd191d-a59d-4dbe-811f-44e2c4ca16e1\") " pod="openshift-dns/dns-default-npv6z" Jul 08 19:24:30.659670 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:30.659105 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 19:24:30.659670 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:30.659169 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2bd191d-a59d-4dbe-811f-44e2c4ca16e1-metrics-tls podName:e2bd191d-a59d-4dbe-811f-44e2c4ca16e1 nodeName:}" failed. No retries permitted until 2026-07-08 19:24:32.659149569 +0000 UTC m=+38.545339611 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e2bd191d-a59d-4dbe-811f-44e2c4ca16e1-metrics-tls") pod "dns-default-npv6z" (UID: "e2bd191d-a59d-4dbe-811f-44e2c4ca16e1") : secret "dns-default-metrics-tls" not found Jul 08 19:24:32.471206 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:32.471175 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:24:32.527078 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:32.527053 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:24:32.675254 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:32.675231 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2bd191d-a59d-4dbe-811f-44e2c4ca16e1-metrics-tls\") pod \"dns-default-npv6z\" (UID: \"e2bd191d-a59d-4dbe-811f-44e2c4ca16e1\") " pod="openshift-dns/dns-default-npv6z" Jul 08 19:24:32.675411 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:32.675388 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 19:24:32.675483 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:32.675471 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2bd191d-a59d-4dbe-811f-44e2c4ca16e1-metrics-tls podName:e2bd191d-a59d-4dbe-811f-44e2c4ca16e1 nodeName:}" failed. No retries permitted until 2026-07-08 19:24:36.675450024 +0000 UTC m=+42.561640068 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e2bd191d-a59d-4dbe-811f-44e2c4ca16e1-metrics-tls") pod "dns-default-npv6z" (UID: "e2bd191d-a59d-4dbe-811f-44e2c4ca16e1") : secret "dns-default-metrics-tls" not found Jul 08 19:24:32.815924 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:32.815830 2566 generic.go:356] "Generic (PLEG): container finished" podID="dbe0d11c-c9c5-4cdb-ad9b-08b142e7335f" containerID="7bc455b4d33a0660f3c0b4495256956c95799390b1344e1d7dd7cfd683c88bdf" exitCode=0 Jul 08 19:24:32.815924 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:32.815887 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-srmfq" event={"ID":"dbe0d11c-c9c5-4cdb-ad9b-08b142e7335f","Type":"ContainerDied","Data":"7bc455b4d33a0660f3c0b4495256956c95799390b1344e1d7dd7cfd683c88bdf"} Jul 08 19:24:32.817369 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:32.817347 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5nfnp" event={"ID":"d36b2cab-e4d2-4a23-85f9-0bd24ccbb497","Type":"ContainerStarted","Data":"8532e549d310a9859f6d2cbc494661afa0ce6f5d7b641839ae26f1a263742905"} Jul 08 19:24:32.853519 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:32.853471 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-5nfnp" podStartSLOduration=1.796604224 podStartE2EDuration="4.853454204s" totalImagesPullingTime="3.05684998s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:24:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:24:32.852678448 +0000 UTC m=+38.738868496" watchObservedRunningTime="2026-07-08 19:24:32.853454204 +0000 UTC m=+38.739644252" Jul 08 19:24:33.521196 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:33.521151 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:24:33.822460 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:33.822383 2566 generic.go:356] "Generic (PLEG): container finished" podID="dbe0d11c-c9c5-4cdb-ad9b-08b142e7335f" containerID="cef66b3799d67db91609b5bb3a3c070cc5d812366a39516190d528b7e90180c1" exitCode=0 Jul 08 19:24:33.822592 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:33.822472 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-srmfq" event={"ID":"dbe0d11c-c9c5-4cdb-ad9b-08b142e7335f","Type":"ContainerDied","Data":"cef66b3799d67db91609b5bb3a3c070cc5d812366a39516190d528b7e90180c1"} Jul 08 19:24:34.827769 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:34.827732 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-srmfq" event={"ID":"dbe0d11c-c9c5-4cdb-ad9b-08b142e7335f","Type":"ContainerStarted","Data":"797759be3825d6069c70f7b17e3751d405bf99c85413d96f7fe91b9cc5f8b13e"} Jul 08 19:24:34.852568 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:34.852523 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-srmfq" podStartSLOduration=8.993124328 podStartE2EDuration="40.852504345s" totalImagesPullingTime="31.859380017s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 19:23:54 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:24:34.851210662 +0000 UTC m=+40.737400709" watchObservedRunningTime="2026-07-08 19:24:34.852504345 +0000 UTC m=+40.738694391" Jul 08 19:24:34.973207 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:34.973177 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:24:35.031022 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:35.030997 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:24:35.187037 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:35.187000 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:24:35.248833 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:35.248804 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:24:36.702338 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:36.702299 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2bd191d-a59d-4dbe-811f-44e2c4ca16e1-metrics-tls\") pod \"dns-default-npv6z\" (UID: \"e2bd191d-a59d-4dbe-811f-44e2c4ca16e1\") " pod="openshift-dns/dns-default-npv6z" Jul 08 19:24:36.702727 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:36.702415 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 19:24:36.702727 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:36.702471 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2bd191d-a59d-4dbe-811f-44e2c4ca16e1-metrics-tls podName:e2bd191d-a59d-4dbe-811f-44e2c4ca16e1 nodeName:}" failed. No retries permitted until 2026-07-08 19:24:44.702456744 +0000 UTC m=+50.588646769 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e2bd191d-a59d-4dbe-811f-44e2c4ca16e1-metrics-tls") pod "dns-default-npv6z" (UID: "e2bd191d-a59d-4dbe-811f-44e2c4ca16e1") : secret "dns-default-metrics-tls" not found Jul 08 19:24:40.275208 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:40.275167 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:24:41.961492 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:41.961450 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:24:42.031513 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:42.031482 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:24:44.757734 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:44.757692 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2bd191d-a59d-4dbe-811f-44e2c4ca16e1-metrics-tls\") pod \"dns-default-npv6z\" (UID: \"e2bd191d-a59d-4dbe-811f-44e2c4ca16e1\") " pod="openshift-dns/dns-default-npv6z" Jul 08 19:24:44.758132 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:44.757815 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 19:24:44.758132 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:24:44.757895 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2bd191d-a59d-4dbe-811f-44e2c4ca16e1-metrics-tls podName:e2bd191d-a59d-4dbe-811f-44e2c4ca16e1 nodeName:}" failed. No retries permitted until 2026-07-08 19:25:00.757879715 +0000 UTC m=+66.644069755 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e2bd191d-a59d-4dbe-811f-44e2c4ca16e1-metrics-tls") pod "dns-default-npv6z" (UID: "e2bd191d-a59d-4dbe-811f-44e2c4ca16e1") : secret "dns-default-metrics-tls" not found Jul 08 19:24:51.819267 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:51.819225 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:24:51.876996 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:51.876962 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:24:54.531325 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:54.531292 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:24:55.804853 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:55.804826 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cdp8n" Jul 08 19:24:55.956928 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:55.956891 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:24:56.057505 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:24:56.057425 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:25:00.264414 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:25:00.264360 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24b6882e-3f1e-4c28-b128-2b16f3478940-original-pull-secret\") pod \"global-pull-secret-syncer-qdfqb\" (UID: \"24b6882e-3f1e-4c28-b128-2b16f3478940\") " pod="kube-system/global-pull-secret-syncer-qdfqb" Jul 08 19:25:00.265004 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:25:00.264510 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vc5r\" (UniqueName: \"kubernetes.io/projected/60e1939e-8740-43f5-8d7b-056635c9f877-kube-api-access-5vc5r\") pod \"network-check-target-rs9hh\" (UID: \"60e1939e-8740-43f5-8d7b-056635c9f877\") " pod="openshift-network-diagnostics/network-check-target-rs9hh" Jul 08 19:25:00.267104 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:25:00.267085 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 08 19:25:00.267209 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:25:00.267193 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 08 19:25:00.277667 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:25:00.277640 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 08 19:25:00.278872 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:25:00.278827 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24b6882e-3f1e-4c28-b128-2b16f3478940-original-pull-secret\") pod \"global-pull-secret-syncer-qdfqb\" (UID: \"24b6882e-3f1e-4c28-b128-2b16f3478940\") " pod="kube-system/global-pull-secret-syncer-qdfqb" Jul 08 19:25:00.287635 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:25:00.287612 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vc5r\" (UniqueName: \"kubernetes.io/projected/60e1939e-8740-43f5-8d7b-056635c9f877-kube-api-access-5vc5r\") pod \"network-check-target-rs9hh\" (UID: \"60e1939e-8740-43f5-8d7b-056635c9f877\") " pod="openshift-network-diagnostics/network-check-target-rs9hh" Jul 08 19:25:00.364820 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:25:00.364785 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55818509-538d-4733-9e10-eaaf841ef071-metrics-certs\") pod \"network-metrics-daemon-rfbjf\" (UID: \"55818509-538d-4733-9e10-eaaf841ef071\") " pod="openshift-multus/network-metrics-daemon-rfbjf" Jul 08 19:25:00.367713 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:25:00.367696 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 08 19:25:00.375978 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:25:00.375961 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 08 19:25:00.376043 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:25:00.376035 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55818509-538d-4733-9e10-eaaf841ef071-metrics-certs podName:55818509-538d-4733-9e10-eaaf841ef071 nodeName:}" failed. No retries permitted until 2026-07-08 19:26:04.376017554 +0000 UTC m=+130.262207579 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55818509-538d-4733-9e10-eaaf841ef071-metrics-certs") pod "network-metrics-daemon-rfbjf" (UID: "55818509-538d-4733-9e10-eaaf841ef071") : secret "metrics-daemon-secret" not found Jul 08 19:25:00.537465 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:25:00.537387 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-98h84\"" Jul 08 19:25:00.545546 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:25:00.545525 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rs9hh" Jul 08 19:25:00.546765 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:25:00.546748 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdfqb" Jul 08 19:25:00.675567 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:25:00.675443 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rs9hh"] Jul 08 19:25:00.680022 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:25:00.679992 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60e1939e_8740_43f5_8d7b_056635c9f877.slice/crio-7d266342a47d5eb0de6ea3304ac53478f5ba523dac6b84f3c8a23450b6571bae WatchSource:0}: Error finding container 7d266342a47d5eb0de6ea3304ac53478f5ba523dac6b84f3c8a23450b6571bae: Status 404 returned error can't find the container with id 7d266342a47d5eb0de6ea3304ac53478f5ba523dac6b84f3c8a23450b6571bae Jul 08 19:25:00.692643 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:25:00.692617 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qdfqb"] Jul 08 19:25:00.696720 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:25:00.696692 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24b6882e_3f1e_4c28_b128_2b16f3478940.slice/crio-14cc631336b648a1e6c065ed4f31122bcb47d2ff0098496902619f234de73f90 WatchSource:0}: Error finding container 14cc631336b648a1e6c065ed4f31122bcb47d2ff0098496902619f234de73f90: Status 404 returned error can't find the container with id 14cc631336b648a1e6c065ed4f31122bcb47d2ff0098496902619f234de73f90 Jul 08 19:25:00.767294 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:25:00.767257 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2bd191d-a59d-4dbe-811f-44e2c4ca16e1-metrics-tls\") pod \"dns-default-npv6z\" (UID: \"e2bd191d-a59d-4dbe-811f-44e2c4ca16e1\") " pod="openshift-dns/dns-default-npv6z" Jul 08 19:25:00.767474 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:25:00.767426 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 19:25:00.767523 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:25:00.767499 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2bd191d-a59d-4dbe-811f-44e2c4ca16e1-metrics-tls podName:e2bd191d-a59d-4dbe-811f-44e2c4ca16e1 nodeName:}" failed. No retries permitted until 2026-07-08 19:25:32.767481247 +0000 UTC m=+98.653671271 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e2bd191d-a59d-4dbe-811f-44e2c4ca16e1-metrics-tls") pod "dns-default-npv6z" (UID: "e2bd191d-a59d-4dbe-811f-44e2c4ca16e1") : secret "dns-default-metrics-tls" not found Jul 08 19:25:00.888322 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:25:00.888225 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qdfqb" event={"ID":"24b6882e-3f1e-4c28-b128-2b16f3478940","Type":"ContainerStarted","Data":"14cc631336b648a1e6c065ed4f31122bcb47d2ff0098496902619f234de73f90"} Jul 08 19:25:00.889293 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:25:00.889270 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rs9hh" event={"ID":"60e1939e-8740-43f5-8d7b-056635c9f877","Type":"ContainerStarted","Data":"7d266342a47d5eb0de6ea3304ac53478f5ba523dac6b84f3c8a23450b6571bae"} Jul 08 19:25:00.924023 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:25:00.923989 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:25:00.994685 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:25:00.994659 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:25:01.062623 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:25:01.062593 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:25:06.905286 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:25:06.905254 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qdfqb" event={"ID":"24b6882e-3f1e-4c28-b128-2b16f3478940","Type":"ContainerStarted","Data":"a4406a126bd28c2351f1d15db83d25355fffb18e6a2a89598429d7b134129103"} Jul 08 19:25:06.906762 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:25:06.906739 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rs9hh" event={"ID":"60e1939e-8740-43f5-8d7b-056635c9f877","Type":"ContainerStarted","Data":"ea8d690ab0357da16de3c7e3b20318aee16637a6ec6bde7119fe65dae62fc849"} Jul 08 19:25:06.906927 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:25:06.906808 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-rs9hh" Jul 08 19:25:06.919493 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:25:06.919400 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-qdfqb" podStartSLOduration=67.085328116 podStartE2EDuration="1m12.919386211s" totalImagesPullingTime="5.834058095s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 19:23:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:25:06.91921257 +0000 UTC m=+72.805402618" watchObservedRunningTime="2026-07-08 19:25:06.919386211 +0000 UTC m=+72.805576254" Jul 08 19:25:06.924247 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:25:06.924227 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:25:06.933393 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:25:06.933361 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-rs9hh" podStartSLOduration=67.080103182 podStartE2EDuration="1m12.933350106s" totalImagesPullingTime="5.853246924s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:23:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:25:06.933082496 +0000 UTC m=+72.819272547" watchObservedRunningTime="2026-07-08 19:25:06.933350106 +0000 UTC m=+72.819540154" Jul 08 19:25:07.025602 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:25:07.025575 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:25:16.737755 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:25:16.737652 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-67fd4cbc8c-7xbn7"] Jul 08 19:25:16.739624 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:25:16.739607 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67fd4cbc8c-7xbn7" Jul 08 19:25:16.743191 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:25:16.741955 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 08 19:25:16.743191 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:25:16.743081 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 08 19:25:16.743336 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:25:16.743256 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 08 19:25:16.743336 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:25:16.743265 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 08 19:25:16.749954 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:25:16.749926 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-67fd4cbc8c-7xbn7"] Jul 08 19:25:16.769144 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:25:16.769121 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/a7bf1078-d8ed-4c6e-bb79-6f62ac1b6130-klusterlet-config\") pod \"klusterlet-addon-workmgr-67fd4cbc8c-7xbn7\" (UID: \"a7bf1078-d8ed-4c6e-bb79-6f62ac1b6130\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67fd4cbc8c-7xbn7" Jul 08 19:25:16.769247 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:25:16.769155 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9wgf\" (UniqueName: \"kubernetes.io/projected/a7bf1078-d8ed-4c6e-bb79-6f62ac1b6130-kube-api-access-f9wgf\") pod \"klusterlet-addon-workmgr-67fd4cbc8c-7xbn7\" (UID: \"a7bf1078-d8ed-4c6e-bb79-6f62ac1b6130\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67fd4cbc8c-7xbn7" Jul 08 19:25:16.769247 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:25:16.769174 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a7bf1078-d8ed-4c6e-bb79-6f62ac1b6130-tmp\") pod \"klusterlet-addon-workmgr-67fd4cbc8c-7xbn7\" (UID: \"a7bf1078-d8ed-4c6e-bb79-6f62ac1b6130\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67fd4cbc8c-7xbn7" Jul 08 19:25:16.870382 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:25:16.870344 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9wgf\" (UniqueName: \"kubernetes.io/projected/a7bf1078-d8ed-4c6e-bb79-6f62ac1b6130-kube-api-access-f9wgf\") pod \"klusterlet-addon-workmgr-67fd4cbc8c-7xbn7\" (UID: \"a7bf1078-d8ed-4c6e-bb79-6f62ac1b6130\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67fd4cbc8c-7xbn7" Jul 08 19:25:16.870382 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:25:16.870387 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a7bf1078-d8ed-4c6e-bb79-6f62ac1b6130-tmp\") pod \"klusterlet-addon-workmgr-67fd4cbc8c-7xbn7\" (UID: \"a7bf1078-d8ed-4c6e-bb79-6f62ac1b6130\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67fd4cbc8c-7xbn7" Jul 08 19:25:16.870593 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:25:16.870436 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/a7bf1078-d8ed-4c6e-bb79-6f62ac1b6130-klusterlet-config\") pod \"klusterlet-addon-workmgr-67fd4cbc8c-7xbn7\" (UID: \"a7bf1078-d8ed-4c6e-bb79-6f62ac1b6130\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67fd4cbc8c-7xbn7" Jul 08 19:25:16.870877 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:25:16.870839 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a7bf1078-d8ed-4c6e-bb79-6f62ac1b6130-tmp\") pod \"klusterlet-addon-workmgr-67fd4cbc8c-7xbn7\" (UID: \"a7bf1078-d8ed-4c6e-bb79-6f62ac1b6130\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67fd4cbc8c-7xbn7" Jul 08 19:25:16.872808 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:25:16.872791 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/a7bf1078-d8ed-4c6e-bb79-6f62ac1b6130-klusterlet-config\") pod \"klusterlet-addon-workmgr-67fd4cbc8c-7xbn7\" (UID: \"a7bf1078-d8ed-4c6e-bb79-6f62ac1b6130\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67fd4cbc8c-7xbn7" Jul 08 19:25:16.878592 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:25:16.878569 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9wgf\" (UniqueName: \"kubernetes.io/projected/a7bf1078-d8ed-4c6e-bb79-6f62ac1b6130-kube-api-access-f9wgf\") pod \"klusterlet-addon-workmgr-67fd4cbc8c-7xbn7\" (UID: \"a7bf1078-d8ed-4c6e-bb79-6f62ac1b6130\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67fd4cbc8c-7xbn7" Jul 08 19:25:17.051329 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:25:17.051263 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67fd4cbc8c-7xbn7" Jul 08 19:25:17.166028 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:25:17.165996 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-67fd4cbc8c-7xbn7"] Jul 08 19:25:17.169016 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:25:17.168988 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7bf1078_d8ed_4c6e_bb79_6f62ac1b6130.slice/crio-98579c33f84da1728a9d1fa67530f1c6f162ee3f6e3083d40f43af383762c332 WatchSource:0}: Error finding container 98579c33f84da1728a9d1fa67530f1c6f162ee3f6e3083d40f43af383762c332: Status 404 returned error can't find the container with id 98579c33f84da1728a9d1fa67530f1c6f162ee3f6e3083d40f43af383762c332 Jul 08 19:25:17.932901 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:25:17.932850 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67fd4cbc8c-7xbn7" event={"ID":"a7bf1078-d8ed-4c6e-bb79-6f62ac1b6130","Type":"ContainerStarted","Data":"98579c33f84da1728a9d1fa67530f1c6f162ee3f6e3083d40f43af383762c332"} Jul 08 19:25:21.948058 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:25:21.948016 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67fd4cbc8c-7xbn7" event={"ID":"a7bf1078-d8ed-4c6e-bb79-6f62ac1b6130","Type":"ContainerStarted","Data":"cd101e86b6fc6cd10999b69baaded88a85e84368e132d1b5f42b99f26fd85c52"} Jul 08 19:25:21.948542 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:25:21.948242 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67fd4cbc8c-7xbn7" Jul 08 19:25:21.949819 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:25:21.949801 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67fd4cbc8c-7xbn7" Jul 08 19:25:21.963857 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:25:21.963811 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67fd4cbc8c-7xbn7" podStartSLOduration=2.135466308 podStartE2EDuration="5.963799293s" totalImagesPullingTime="3.828332985s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:25:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:25:21.962647561 +0000 UTC m=+87.848837609" watchObservedRunningTime="2026-07-08 19:25:21.963799293 +0000 UTC m=+87.849989337" Jul 08 19:25:27.301184 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:25:27.301149 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:25:27.437219 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:25:27.437176 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:25:32.775600 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:25:32.775556 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2bd191d-a59d-4dbe-811f-44e2c4ca16e1-metrics-tls\") pod \"dns-default-npv6z\" (UID: \"e2bd191d-a59d-4dbe-811f-44e2c4ca16e1\") " pod="openshift-dns/dns-default-npv6z" Jul 08 19:25:32.775992 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:25:32.775665 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 19:25:32.775992 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:25:32.775727 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2bd191d-a59d-4dbe-811f-44e2c4ca16e1-metrics-tls podName:e2bd191d-a59d-4dbe-811f-44e2c4ca16e1 nodeName:}" failed. No retries permitted until 2026-07-08 19:26:36.775713309 +0000 UTC m=+162.661903335 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e2bd191d-a59d-4dbe-811f-44e2c4ca16e1-metrics-tls") pod "dns-default-npv6z" (UID: "e2bd191d-a59d-4dbe-811f-44e2c4ca16e1") : secret "dns-default-metrics-tls" not found Jul 08 19:25:37.911810 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:25:37.911777 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-rs9hh" Jul 08 19:25:46.166744 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:25:46.166703 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:26:04.393974 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:26:04.393926 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55818509-538d-4733-9e10-eaaf841ef071-metrics-certs\") pod \"network-metrics-daemon-rfbjf\" (UID: \"55818509-538d-4733-9e10-eaaf841ef071\") " pod="openshift-multus/network-metrics-daemon-rfbjf" Jul 08 19:26:04.394362 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:26:04.394076 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 08 19:26:04.394362 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:26:04.394155 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55818509-538d-4733-9e10-eaaf841ef071-metrics-certs podName:55818509-538d-4733-9e10-eaaf841ef071 nodeName:}" failed. No retries permitted until 2026-07-08 19:28:06.394139727 +0000 UTC m=+252.280329756 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55818509-538d-4733-9e10-eaaf841ef071-metrics-certs") pod "network-metrics-daemon-rfbjf" (UID: "55818509-538d-4733-9e10-eaaf841ef071") : secret "metrics-daemon-secret" not found Jul 08 19:26:31.911529 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:26:31.911473 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-npv6z" podUID="e2bd191d-a59d-4dbe-811f-44e2c4ca16e1" Jul 08 19:26:32.115001 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:26:32.114974 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-npv6z" Jul 08 19:26:32.641061 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:26:32.641024 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-rfbjf" podUID="55818509-538d-4733-9e10-eaaf841ef071" Jul 08 19:26:36.824931 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:26:36.824858 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2bd191d-a59d-4dbe-811f-44e2c4ca16e1-metrics-tls\") pod \"dns-default-npv6z\" (UID: \"e2bd191d-a59d-4dbe-811f-44e2c4ca16e1\") " pod="openshift-dns/dns-default-npv6z" Jul 08 19:26:36.825359 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:26:36.824972 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 19:26:36.825359 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:26:36.825052 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2bd191d-a59d-4dbe-811f-44e2c4ca16e1-metrics-tls podName:e2bd191d-a59d-4dbe-811f-44e2c4ca16e1 nodeName:}" failed. No retries permitted until 2026-07-08 19:28:38.825035374 +0000 UTC m=+284.711225399 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e2bd191d-a59d-4dbe-811f-44e2c4ca16e1-metrics-tls") pod "dns-default-npv6z" (UID: "e2bd191d-a59d-4dbe-811f-44e2c4ca16e1") : secret "dns-default-metrics-tls" not found Jul 08 19:26:45.617978 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:26:45.617939 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfbjf" Jul 08 19:27:21.233682 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:27:21.233647 2566 generic.go:356] "Generic (PLEG): container finished" podID="a7bf1078-d8ed-4c6e-bb79-6f62ac1b6130" containerID="cd101e86b6fc6cd10999b69baaded88a85e84368e132d1b5f42b99f26fd85c52" exitCode=1 Jul 08 19:27:21.234161 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:27:21.233730 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67fd4cbc8c-7xbn7" event={"ID":"a7bf1078-d8ed-4c6e-bb79-6f62ac1b6130","Type":"ContainerDied","Data":"cd101e86b6fc6cd10999b69baaded88a85e84368e132d1b5f42b99f26fd85c52"} Jul 08 19:27:21.234161 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:27:21.234096 2566 scope.go:117] "RemoveContainer" containerID="cd101e86b6fc6cd10999b69baaded88a85e84368e132d1b5f42b99f26fd85c52" Jul 08 19:27:21.948527 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:27:21.948475 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67fd4cbc8c-7xbn7" Jul 08 19:27:22.238202 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:27:22.238109 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67fd4cbc8c-7xbn7" event={"ID":"a7bf1078-d8ed-4c6e-bb79-6f62ac1b6130","Type":"ContainerStarted","Data":"084297add1a5b6aabf66e50a5959b8ac9bc720c0e540e66e79278e56a7e8be01"} Jul 08 19:27:22.238618 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:27:22.238341 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67fd4cbc8c-7xbn7" Jul 08 19:27:22.238921 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:27:22.238903 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67fd4cbc8c-7xbn7" Jul 08 19:27:40.606525 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:27:40.606474 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:28:06.427714 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:06.427652 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55818509-538d-4733-9e10-eaaf841ef071-metrics-certs\") pod \"network-metrics-daemon-rfbjf\" (UID: \"55818509-538d-4733-9e10-eaaf841ef071\") " pod="openshift-multus/network-metrics-daemon-rfbjf" Jul 08 19:28:06.428182 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:28:06.427814 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 08 19:28:06.428182 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:28:06.427906 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55818509-538d-4733-9e10-eaaf841ef071-metrics-certs podName:55818509-538d-4733-9e10-eaaf841ef071 nodeName:}" failed. No retries permitted until 2026-07-08 19:30:08.427888474 +0000 UTC m=+374.314078503 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55818509-538d-4733-9e10-eaaf841ef071-metrics-certs") pod "network-metrics-daemon-rfbjf" (UID: "55818509-538d-4733-9e10-eaaf841ef071") : secret "metrics-daemon-secret" not found Jul 08 19:28:14.809375 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:14.809340 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-gfkmh"] Jul 08 19:28:14.811318 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:14.811297 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-gfkmh" Jul 08 19:28:14.814303 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:14.814275 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 08 19:28:14.814413 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:14.814337 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 08 19:28:14.814413 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:14.814391 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-2jc9t\"" Jul 08 19:28:14.815381 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:14.815365 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 08 19:28:14.815381 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:14.815387 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 08 19:28:14.823203 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:14.823184 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 08 19:28:14.830349 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:14.830322 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-gfkmh"] Jul 08 19:28:14.884213 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:14.884173 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6c0d602f-7a0a-415c-9716-54ffb6c51e67-serving-cert\") pod \"insights-operator-5c847dcfc4-gfkmh\" (UID: \"6c0d602f-7a0a-415c-9716-54ffb6c51e67\") " pod="openshift-insights/insights-operator-5c847dcfc4-gfkmh" Jul 08 19:28:14.884388 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:14.884227 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c0d602f-7a0a-415c-9716-54ffb6c51e67-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-gfkmh\" (UID: \"6c0d602f-7a0a-415c-9716-54ffb6c51e67\") " pod="openshift-insights/insights-operator-5c847dcfc4-gfkmh" Jul 08 19:28:14.884388 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:14.884244 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c0d602f-7a0a-415c-9716-54ffb6c51e67-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-gfkmh\" (UID: \"6c0d602f-7a0a-415c-9716-54ffb6c51e67\") " pod="openshift-insights/insights-operator-5c847dcfc4-gfkmh" Jul 08 19:28:14.884388 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:14.884343 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhf5f\" (UniqueName: \"kubernetes.io/projected/6c0d602f-7a0a-415c-9716-54ffb6c51e67-kube-api-access-qhf5f\") pod \"insights-operator-5c847dcfc4-gfkmh\" (UID: \"6c0d602f-7a0a-415c-9716-54ffb6c51e67\") " pod="openshift-insights/insights-operator-5c847dcfc4-gfkmh" Jul 08 19:28:14.884388 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:14.884377 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6c0d602f-7a0a-415c-9716-54ffb6c51e67-tmp\") pod \"insights-operator-5c847dcfc4-gfkmh\" (UID: \"6c0d602f-7a0a-415c-9716-54ffb6c51e67\") " pod="openshift-insights/insights-operator-5c847dcfc4-gfkmh" Jul 08 19:28:14.884555 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:14.884395 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/6c0d602f-7a0a-415c-9716-54ffb6c51e67-snapshots\") pod \"insights-operator-5c847dcfc4-gfkmh\" (UID: \"6c0d602f-7a0a-415c-9716-54ffb6c51e67\") " pod="openshift-insights/insights-operator-5c847dcfc4-gfkmh" Jul 08 19:28:14.985345 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:14.985310 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhf5f\" (UniqueName: \"kubernetes.io/projected/6c0d602f-7a0a-415c-9716-54ffb6c51e67-kube-api-access-qhf5f\") pod \"insights-operator-5c847dcfc4-gfkmh\" (UID: \"6c0d602f-7a0a-415c-9716-54ffb6c51e67\") " pod="openshift-insights/insights-operator-5c847dcfc4-gfkmh" Jul 08 19:28:14.985345 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:14.985352 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6c0d602f-7a0a-415c-9716-54ffb6c51e67-tmp\") pod \"insights-operator-5c847dcfc4-gfkmh\" (UID: \"6c0d602f-7a0a-415c-9716-54ffb6c51e67\") " pod="openshift-insights/insights-operator-5c847dcfc4-gfkmh" Jul 08 19:28:14.985565 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:14.985370 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/6c0d602f-7a0a-415c-9716-54ffb6c51e67-snapshots\") pod \"insights-operator-5c847dcfc4-gfkmh\" (UID: \"6c0d602f-7a0a-415c-9716-54ffb6c51e67\") " pod="openshift-insights/insights-operator-5c847dcfc4-gfkmh" Jul 08 19:28:14.985565 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:14.985409 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6c0d602f-7a0a-415c-9716-54ffb6c51e67-serving-cert\") pod \"insights-operator-5c847dcfc4-gfkmh\" (UID: \"6c0d602f-7a0a-415c-9716-54ffb6c51e67\") " pod="openshift-insights/insights-operator-5c847dcfc4-gfkmh" Jul 08 19:28:14.985565 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:14.985450 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c0d602f-7a0a-415c-9716-54ffb6c51e67-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-gfkmh\" (UID: \"6c0d602f-7a0a-415c-9716-54ffb6c51e67\") " pod="openshift-insights/insights-operator-5c847dcfc4-gfkmh" Jul 08 19:28:14.985565 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:14.985476 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c0d602f-7a0a-415c-9716-54ffb6c51e67-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-gfkmh\" (UID: \"6c0d602f-7a0a-415c-9716-54ffb6c51e67\") " pod="openshift-insights/insights-operator-5c847dcfc4-gfkmh" Jul 08 19:28:14.985828 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:14.985804 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6c0d602f-7a0a-415c-9716-54ffb6c51e67-tmp\") pod \"insights-operator-5c847dcfc4-gfkmh\" (UID: \"6c0d602f-7a0a-415c-9716-54ffb6c51e67\") " pod="openshift-insights/insights-operator-5c847dcfc4-gfkmh" Jul 08 19:28:14.986084 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:14.986057 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/6c0d602f-7a0a-415c-9716-54ffb6c51e67-snapshots\") pod \"insights-operator-5c847dcfc4-gfkmh\" (UID: \"6c0d602f-7a0a-415c-9716-54ffb6c51e67\") " pod="openshift-insights/insights-operator-5c847dcfc4-gfkmh" Jul 08 19:28:14.986201 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:14.986066 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c0d602f-7a0a-415c-9716-54ffb6c51e67-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-gfkmh\" (UID: \"6c0d602f-7a0a-415c-9716-54ffb6c51e67\") " pod="openshift-insights/insights-operator-5c847dcfc4-gfkmh" Jul 08 19:28:14.986201 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:14.986195 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c0d602f-7a0a-415c-9716-54ffb6c51e67-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-gfkmh\" (UID: \"6c0d602f-7a0a-415c-9716-54ffb6c51e67\") " pod="openshift-insights/insights-operator-5c847dcfc4-gfkmh" Jul 08 19:28:14.987727 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:14.987699 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6c0d602f-7a0a-415c-9716-54ffb6c51e67-serving-cert\") pod \"insights-operator-5c847dcfc4-gfkmh\" (UID: \"6c0d602f-7a0a-415c-9716-54ffb6c51e67\") " pod="openshift-insights/insights-operator-5c847dcfc4-gfkmh" Jul 08 19:28:15.000767 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.000745 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhf5f\" (UniqueName: \"kubernetes.io/projected/6c0d602f-7a0a-415c-9716-54ffb6c51e67-kube-api-access-qhf5f\") pod \"insights-operator-5c847dcfc4-gfkmh\" (UID: \"6c0d602f-7a0a-415c-9716-54ffb6c51e67\") " pod="openshift-insights/insights-operator-5c847dcfc4-gfkmh" Jul 08 19:28:15.015143 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.015109 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-v4w6v"] Jul 08 19:28:15.017169 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.017144 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-v4w6v" Jul 08 19:28:15.017889 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.017855 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5846dc5c45-fgr6x"] Jul 08 19:28:15.019611 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.019596 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5846dc5c45-fgr6x" Jul 08 19:28:15.020598 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.020582 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 08 19:28:15.023096 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.023078 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-fdvwh\"" Jul 08 19:28:15.023305 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.023292 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 08 19:28:15.023387 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.023368 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 08 19:28:15.024704 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.024684 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 08 19:28:15.029132 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.029105 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 08 19:28:15.029132 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.029125 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-f52fc\"" Jul 08 19:28:15.029286 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.029177 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 08 19:28:15.030169 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.030137 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 08 19:28:15.030398 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.030379 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 08 19:28:15.030487 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.030438 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 08 19:28:15.030487 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.030457 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 08 19:28:15.030797 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.030782 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6d9dd85c4b-s89jc"] Jul 08 19:28:15.036071 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.036047 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-7ntp9"] Jul 08 19:28:15.036495 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.036159 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d9dd85c4b-s89jc" Jul 08 19:28:15.038969 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.038944 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-7ntp9" Jul 08 19:28:15.039759 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.039728 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 08 19:28:15.039937 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.039744 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 08 19:28:15.040654 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.040631 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5846dc5c45-fgr6x"] Jul 08 19:28:15.040742 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.040725 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-phtdc\"" Jul 08 19:28:15.040805 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.040725 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 08 19:28:15.041169 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.041146 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-v4w6v"] Jul 08 19:28:15.042490 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.042457 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 08 19:28:15.042585 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.042570 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 08 19:28:15.042754 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.042734 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 08 19:28:15.042967 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.042741 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 08 19:28:15.043134 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.042854 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-gdnqm\"" Jul 08 19:28:15.046163 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.046146 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 08 19:28:15.053047 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.053026 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-7ntp9"] Jul 08 19:28:15.062211 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.062153 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6d9dd85c4b-s89jc"] Jul 08 19:28:15.086733 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.086699 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/cee815ab-2b57-4584-8b6c-a8748c849be9-default-certificate\") pod \"router-default-5846dc5c45-fgr6x\" (UID: \"cee815ab-2b57-4584-8b6c-a8748c849be9\") " pod="openshift-ingress/router-default-5846dc5c45-fgr6x" Jul 08 19:28:15.086733 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.086732 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ec79a684-b3d4-4942-aed4-41d82124967d-trusted-ca\") pod \"image-registry-6d9dd85c4b-s89jc\" (UID: \"ec79a684-b3d4-4942-aed4-41d82124967d\") " pod="openshift-image-registry/image-registry-6d9dd85c4b-s89jc" Jul 08 19:28:15.086949 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.086748 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cee815ab-2b57-4584-8b6c-a8748c849be9-service-ca-bundle\") pod \"router-default-5846dc5c45-fgr6x\" (UID: \"cee815ab-2b57-4584-8b6c-a8748c849be9\") " pod="openshift-ingress/router-default-5846dc5c45-fgr6x" Jul 08 19:28:15.086949 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.086766 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6e934fe3-7407-4bd1-88cc-21be3143047c-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-v4w6v\" (UID: \"6e934fe3-7407-4bd1-88cc-21be3143047c\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-v4w6v" Jul 08 19:28:15.086949 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.086800 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ec79a684-b3d4-4942-aed4-41d82124967d-registry-tls\") pod \"image-registry-6d9dd85c4b-s89jc\" (UID: \"ec79a684-b3d4-4942-aed4-41d82124967d\") " pod="openshift-image-registry/image-registry-6d9dd85c4b-s89jc" Jul 08 19:28:15.086949 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.086823 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ec79a684-b3d4-4942-aed4-41d82124967d-registry-certificates\") pod \"image-registry-6d9dd85c4b-s89jc\" (UID: \"ec79a684-b3d4-4942-aed4-41d82124967d\") " pod="openshift-image-registry/image-registry-6d9dd85c4b-s89jc" Jul 08 19:28:15.086949 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.086837 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ec79a684-b3d4-4942-aed4-41d82124967d-installation-pull-secrets\") pod \"image-registry-6d9dd85c4b-s89jc\" (UID: \"ec79a684-b3d4-4942-aed4-41d82124967d\") " pod="openshift-image-registry/image-registry-6d9dd85c4b-s89jc" Jul 08 19:28:15.086949 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.086852 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n9kp\" (UniqueName: \"kubernetes.io/projected/ec79a684-b3d4-4942-aed4-41d82124967d-kube-api-access-8n9kp\") pod \"image-registry-6d9dd85c4b-s89jc\" (UID: \"ec79a684-b3d4-4942-aed4-41d82124967d\") " pod="openshift-image-registry/image-registry-6d9dd85c4b-s89jc" Jul 08 19:28:15.086949 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.086908 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/05373d28-0e17-422e-96ec-1add0377dd9d-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-7ntp9\" (UID: \"05373d28-0e17-422e-96ec-1add0377dd9d\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-7ntp9" Jul 08 19:28:15.086949 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.086937 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5bzl\" (UniqueName: \"kubernetes.io/projected/cee815ab-2b57-4584-8b6c-a8748c849be9-kube-api-access-c5bzl\") pod \"router-default-5846dc5c45-fgr6x\" (UID: \"cee815ab-2b57-4584-8b6c-a8748c849be9\") " pod="openshift-ingress/router-default-5846dc5c45-fgr6x" Jul 08 19:28:15.087188 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.086975 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/05373d28-0e17-422e-96ec-1add0377dd9d-config\") pod \"service-ca-operator-5d46d9ffd4-7ntp9\" (UID: \"05373d28-0e17-422e-96ec-1add0377dd9d\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-7ntp9" Jul 08 19:28:15.087188 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.087003 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ec79a684-b3d4-4942-aed4-41d82124967d-ca-trust-extracted\") pod \"image-registry-6d9dd85c4b-s89jc\" (UID: \"ec79a684-b3d4-4942-aed4-41d82124967d\") " pod="openshift-image-registry/image-registry-6d9dd85c4b-s89jc" Jul 08 19:28:15.087188 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.087055 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/cee815ab-2b57-4584-8b6c-a8748c849be9-stats-auth\") pod \"router-default-5846dc5c45-fgr6x\" (UID: \"cee815ab-2b57-4584-8b6c-a8748c849be9\") " pod="openshift-ingress/router-default-5846dc5c45-fgr6x" Jul 08 19:28:15.087188 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.087070 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cee815ab-2b57-4584-8b6c-a8748c849be9-metrics-certs\") pod \"router-default-5846dc5c45-fgr6x\" (UID: \"cee815ab-2b57-4584-8b6c-a8748c849be9\") " pod="openshift-ingress/router-default-5846dc5c45-fgr6x" Jul 08 19:28:15.087188 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.087106 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbwqh\" (UniqueName: \"kubernetes.io/projected/6e934fe3-7407-4bd1-88cc-21be3143047c-kube-api-access-qbwqh\") pod \"cluster-monitoring-operator-64786d9cbb-v4w6v\" (UID: \"6e934fe3-7407-4bd1-88cc-21be3143047c\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-v4w6v" Jul 08 19:28:15.087188 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.087126 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6e934fe3-7407-4bd1-88cc-21be3143047c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-v4w6v\" (UID: \"6e934fe3-7407-4bd1-88cc-21be3143047c\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-v4w6v" Jul 08 19:28:15.087188 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.087144 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ec79a684-b3d4-4942-aed4-41d82124967d-image-registry-private-configuration\") pod \"image-registry-6d9dd85c4b-s89jc\" (UID: \"ec79a684-b3d4-4942-aed4-41d82124967d\") " pod="openshift-image-registry/image-registry-6d9dd85c4b-s89jc" Jul 08 19:28:15.087188 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.087168 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ec79a684-b3d4-4942-aed4-41d82124967d-bound-sa-token\") pod \"image-registry-6d9dd85c4b-s89jc\" (UID: \"ec79a684-b3d4-4942-aed4-41d82124967d\") " pod="openshift-image-registry/image-registry-6d9dd85c4b-s89jc" Jul 08 19:28:15.087422 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.087227 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvms2\" (UniqueName: \"kubernetes.io/projected/05373d28-0e17-422e-96ec-1add0377dd9d-kube-api-access-bvms2\") pod \"service-ca-operator-5d46d9ffd4-7ntp9\" (UID: \"05373d28-0e17-422e-96ec-1add0377dd9d\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-7ntp9" Jul 08 19:28:15.119851 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.119818 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-gfkmh" Jul 08 19:28:15.187696 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.187620 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ec79a684-b3d4-4942-aed4-41d82124967d-trusted-ca\") pod \"image-registry-6d9dd85c4b-s89jc\" (UID: \"ec79a684-b3d4-4942-aed4-41d82124967d\") " pod="openshift-image-registry/image-registry-6d9dd85c4b-s89jc" Jul 08 19:28:15.187696 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.187675 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cee815ab-2b57-4584-8b6c-a8748c849be9-service-ca-bundle\") pod \"router-default-5846dc5c45-fgr6x\" (UID: \"cee815ab-2b57-4584-8b6c-a8748c849be9\") " pod="openshift-ingress/router-default-5846dc5c45-fgr6x" Jul 08 19:28:15.187902 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.187703 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6e934fe3-7407-4bd1-88cc-21be3143047c-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-v4w6v\" (UID: \"6e934fe3-7407-4bd1-88cc-21be3143047c\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-v4w6v" Jul 08 19:28:15.187902 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.187733 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ec79a684-b3d4-4942-aed4-41d82124967d-registry-tls\") pod \"image-registry-6d9dd85c4b-s89jc\" (UID: \"ec79a684-b3d4-4942-aed4-41d82124967d\") " pod="openshift-image-registry/image-registry-6d9dd85c4b-s89jc" Jul 08 19:28:15.187902 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:28:15.187838 2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 08 19:28:15.187902 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:28:15.187850 2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d9dd85c4b-s89jc: secret "image-registry-tls" not found Jul 08 19:28:15.187902 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.187877 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ec79a684-b3d4-4942-aed4-41d82124967d-registry-certificates\") pod \"image-registry-6d9dd85c4b-s89jc\" (UID: \"ec79a684-b3d4-4942-aed4-41d82124967d\") " pod="openshift-image-registry/image-registry-6d9dd85c4b-s89jc" Jul 08 19:28:15.188054 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.187924 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ec79a684-b3d4-4942-aed4-41d82124967d-installation-pull-secrets\") pod \"image-registry-6d9dd85c4b-s89jc\" (UID: \"ec79a684-b3d4-4942-aed4-41d82124967d\") " pod="openshift-image-registry/image-registry-6d9dd85c4b-s89jc" Jul 08 19:28:15.188054 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:28:15.187937 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ec79a684-b3d4-4942-aed4-41d82124967d-registry-tls podName:ec79a684-b3d4-4942-aed4-41d82124967d nodeName:}" failed. No retries permitted until 2026-07-08 19:28:15.68791984 +0000 UTC m=+261.574109865 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ec79a684-b3d4-4942-aed4-41d82124967d-registry-tls") pod "image-registry-6d9dd85c4b-s89jc" (UID: "ec79a684-b3d4-4942-aed4-41d82124967d") : secret "image-registry-tls" not found Jul 08 19:28:15.188054 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.187969 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n9kp\" (UniqueName: \"kubernetes.io/projected/ec79a684-b3d4-4942-aed4-41d82124967d-kube-api-access-8n9kp\") pod \"image-registry-6d9dd85c4b-s89jc\" (UID: \"ec79a684-b3d4-4942-aed4-41d82124967d\") " pod="openshift-image-registry/image-registry-6d9dd85c4b-s89jc" Jul 08 19:28:15.188054 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.188007 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/05373d28-0e17-422e-96ec-1add0377dd9d-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-7ntp9\" (UID: \"05373d28-0e17-422e-96ec-1add0377dd9d\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-7ntp9" Jul 08 19:28:15.188054 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.188032 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5bzl\" (UniqueName: \"kubernetes.io/projected/cee815ab-2b57-4584-8b6c-a8748c849be9-kube-api-access-c5bzl\") pod \"router-default-5846dc5c45-fgr6x\" (UID: \"cee815ab-2b57-4584-8b6c-a8748c849be9\") " pod="openshift-ingress/router-default-5846dc5c45-fgr6x" Jul 08 19:28:15.188258 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.188065 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/05373d28-0e17-422e-96ec-1add0377dd9d-config\") pod \"service-ca-operator-5d46d9ffd4-7ntp9\" (UID: \"05373d28-0e17-422e-96ec-1add0377dd9d\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-7ntp9" Jul 08 19:28:15.188258 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.188091 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ec79a684-b3d4-4942-aed4-41d82124967d-ca-trust-extracted\") pod \"image-registry-6d9dd85c4b-s89jc\" (UID: \"ec79a684-b3d4-4942-aed4-41d82124967d\") " pod="openshift-image-registry/image-registry-6d9dd85c4b-s89jc" Jul 08 19:28:15.188258 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.188130 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/cee815ab-2b57-4584-8b6c-a8748c849be9-stats-auth\") pod \"router-default-5846dc5c45-fgr6x\" (UID: \"cee815ab-2b57-4584-8b6c-a8748c849be9\") " pod="openshift-ingress/router-default-5846dc5c45-fgr6x" Jul 08 19:28:15.188258 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.188164 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cee815ab-2b57-4584-8b6c-a8748c849be9-metrics-certs\") pod \"router-default-5846dc5c45-fgr6x\" (UID: \"cee815ab-2b57-4584-8b6c-a8748c849be9\") " pod="openshift-ingress/router-default-5846dc5c45-fgr6x" Jul 08 19:28:15.188258 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.188200 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbwqh\" (UniqueName: \"kubernetes.io/projected/6e934fe3-7407-4bd1-88cc-21be3143047c-kube-api-access-qbwqh\") pod \"cluster-monitoring-operator-64786d9cbb-v4w6v\" (UID: \"6e934fe3-7407-4bd1-88cc-21be3143047c\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-v4w6v" Jul 08 19:28:15.188258 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.188235 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6e934fe3-7407-4bd1-88cc-21be3143047c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-v4w6v\" (UID: \"6e934fe3-7407-4bd1-88cc-21be3143047c\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-v4w6v" Jul 08 19:28:15.188552 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.188272 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ec79a684-b3d4-4942-aed4-41d82124967d-image-registry-private-configuration\") pod \"image-registry-6d9dd85c4b-s89jc\" (UID: \"ec79a684-b3d4-4942-aed4-41d82124967d\") " pod="openshift-image-registry/image-registry-6d9dd85c4b-s89jc" Jul 08 19:28:15.188552 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.188317 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ec79a684-b3d4-4942-aed4-41d82124967d-bound-sa-token\") pod \"image-registry-6d9dd85c4b-s89jc\" (UID: \"ec79a684-b3d4-4942-aed4-41d82124967d\") " pod="openshift-image-registry/image-registry-6d9dd85c4b-s89jc" Jul 08 19:28:15.188552 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.188368 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvms2\" (UniqueName: \"kubernetes.io/projected/05373d28-0e17-422e-96ec-1add0377dd9d-kube-api-access-bvms2\") pod \"service-ca-operator-5d46d9ffd4-7ntp9\" (UID: \"05373d28-0e17-422e-96ec-1add0377dd9d\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-7ntp9" Jul 08 19:28:15.188552 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.188424 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/cee815ab-2b57-4584-8b6c-a8748c849be9-default-certificate\") pod \"router-default-5846dc5c45-fgr6x\" (UID: \"cee815ab-2b57-4584-8b6c-a8748c849be9\") " pod="openshift-ingress/router-default-5846dc5c45-fgr6x" Jul 08 19:28:15.188552 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:28:15.188543 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cee815ab-2b57-4584-8b6c-a8748c849be9-service-ca-bundle podName:cee815ab-2b57-4584-8b6c-a8748c849be9 nodeName:}" failed. No retries permitted until 2026-07-08 19:28:15.688525182 +0000 UTC m=+261.574715231 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cee815ab-2b57-4584-8b6c-a8748c849be9-service-ca-bundle") pod "router-default-5846dc5c45-fgr6x" (UID: "cee815ab-2b57-4584-8b6c-a8748c849be9") : configmap references non-existent config key: service-ca.crt Jul 08 19:28:15.188786 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.188591 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6e934fe3-7407-4bd1-88cc-21be3143047c-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-v4w6v\" (UID: \"6e934fe3-7407-4bd1-88cc-21be3143047c\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-v4w6v" Jul 08 19:28:15.188786 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:28:15.188687 2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 08 19:28:15.188786 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:28:15.188737 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e934fe3-7407-4bd1-88cc-21be3143047c-cluster-monitoring-operator-tls podName:6e934fe3-7407-4bd1-88cc-21be3143047c nodeName:}" failed. No retries permitted until 2026-07-08 19:28:15.688719427 +0000 UTC m=+261.574909455 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6e934fe3-7407-4bd1-88cc-21be3143047c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-v4w6v" (UID: "6e934fe3-7407-4bd1-88cc-21be3143047c") : secret "cluster-monitoring-operator-tls" not found Jul 08 19:28:15.188786 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.188760 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ec79a684-b3d4-4942-aed4-41d82124967d-registry-certificates\") pod \"image-registry-6d9dd85c4b-s89jc\" (UID: \"ec79a684-b3d4-4942-aed4-41d82124967d\") " pod="openshift-image-registry/image-registry-6d9dd85c4b-s89jc" Jul 08 19:28:15.189014 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.188851 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ec79a684-b3d4-4942-aed4-41d82124967d-trusted-ca\") pod \"image-registry-6d9dd85c4b-s89jc\" (UID: \"ec79a684-b3d4-4942-aed4-41d82124967d\") " pod="openshift-image-registry/image-registry-6d9dd85c4b-s89jc" Jul 08 19:28:15.189014 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.188923 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ec79a684-b3d4-4942-aed4-41d82124967d-ca-trust-extracted\") pod \"image-registry-6d9dd85c4b-s89jc\" (UID: \"ec79a684-b3d4-4942-aed4-41d82124967d\") " pod="openshift-image-registry/image-registry-6d9dd85c4b-s89jc" Jul 08 19:28:15.189014 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:28:15.188947 2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 08 19:28:15.189014 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:28:15.188994 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cee815ab-2b57-4584-8b6c-a8748c849be9-metrics-certs podName:cee815ab-2b57-4584-8b6c-a8748c849be9 nodeName:}" failed. No retries permitted until 2026-07-08 19:28:15.688979295 +0000 UTC m=+261.575169320 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cee815ab-2b57-4584-8b6c-a8748c849be9-metrics-certs") pod "router-default-5846dc5c45-fgr6x" (UID: "cee815ab-2b57-4584-8b6c-a8748c849be9") : secret "router-metrics-certs-default" not found Jul 08 19:28:15.189580 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.189553 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/05373d28-0e17-422e-96ec-1add0377dd9d-config\") pod \"service-ca-operator-5d46d9ffd4-7ntp9\" (UID: \"05373d28-0e17-422e-96ec-1add0377dd9d\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-7ntp9" Jul 08 19:28:15.191262 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.191226 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/cee815ab-2b57-4584-8b6c-a8748c849be9-default-certificate\") pod \"router-default-5846dc5c45-fgr6x\" (UID: \"cee815ab-2b57-4584-8b6c-a8748c849be9\") " pod="openshift-ingress/router-default-5846dc5c45-fgr6x" Jul 08 19:28:15.191573 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.191546 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/cee815ab-2b57-4584-8b6c-a8748c849be9-stats-auth\") pod \"router-default-5846dc5c45-fgr6x\" (UID: \"cee815ab-2b57-4584-8b6c-a8748c849be9\") " pod="openshift-ingress/router-default-5846dc5c45-fgr6x" Jul 08 19:28:15.191616 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.191586 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/05373d28-0e17-422e-96ec-1add0377dd9d-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-7ntp9\" (UID: \"05373d28-0e17-422e-96ec-1add0377dd9d\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-7ntp9" Jul 08 19:28:15.191616 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.191596 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ec79a684-b3d4-4942-aed4-41d82124967d-installation-pull-secrets\") pod \"image-registry-6d9dd85c4b-s89jc\" (UID: \"ec79a684-b3d4-4942-aed4-41d82124967d\") " pod="openshift-image-registry/image-registry-6d9dd85c4b-s89jc" Jul 08 19:28:15.191688 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.191671 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ec79a684-b3d4-4942-aed4-41d82124967d-image-registry-private-configuration\") pod \"image-registry-6d9dd85c4b-s89jc\" (UID: \"ec79a684-b3d4-4942-aed4-41d82124967d\") " pod="openshift-image-registry/image-registry-6d9dd85c4b-s89jc" Jul 08 19:28:15.206046 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.206007 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvms2\" (UniqueName: \"kubernetes.io/projected/05373d28-0e17-422e-96ec-1add0377dd9d-kube-api-access-bvms2\") pod \"service-ca-operator-5d46d9ffd4-7ntp9\" (UID: \"05373d28-0e17-422e-96ec-1add0377dd9d\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-7ntp9" Jul 08 19:28:15.206716 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.206663 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ec79a684-b3d4-4942-aed4-41d82124967d-bound-sa-token\") pod \"image-registry-6d9dd85c4b-s89jc\" (UID: \"ec79a684-b3d4-4942-aed4-41d82124967d\") " pod="openshift-image-registry/image-registry-6d9dd85c4b-s89jc" Jul 08 19:28:15.206827 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.206752 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n9kp\" (UniqueName: \"kubernetes.io/projected/ec79a684-b3d4-4942-aed4-41d82124967d-kube-api-access-8n9kp\") pod \"image-registry-6d9dd85c4b-s89jc\" (UID: \"ec79a684-b3d4-4942-aed4-41d82124967d\") " pod="openshift-image-registry/image-registry-6d9dd85c4b-s89jc" Jul 08 19:28:15.207956 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.207931 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbwqh\" (UniqueName: \"kubernetes.io/projected/6e934fe3-7407-4bd1-88cc-21be3143047c-kube-api-access-qbwqh\") pod \"cluster-monitoring-operator-64786d9cbb-v4w6v\" (UID: \"6e934fe3-7407-4bd1-88cc-21be3143047c\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-v4w6v" Jul 08 19:28:15.208407 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.208386 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5bzl\" (UniqueName: \"kubernetes.io/projected/cee815ab-2b57-4584-8b6c-a8748c849be9-kube-api-access-c5bzl\") pod \"router-default-5846dc5c45-fgr6x\" (UID: \"cee815ab-2b57-4584-8b6c-a8748c849be9\") " pod="openshift-ingress/router-default-5846dc5c45-fgr6x" Jul 08 19:28:15.218163 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.218137 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:28:15.248686 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.248655 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-gfkmh"] Jul 08 19:28:15.251453 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:28:15.251427 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c0d602f_7a0a_415c_9716_54ffb6c51e67.slice/crio-6c655051d107739fb0203ae6f21d58a7ad473118528bd64f1ff7d1834120f219 WatchSource:0}: Error finding container 6c655051d107739fb0203ae6f21d58a7ad473118528bd64f1ff7d1834120f219: Status 404 returned error can't find the container with id 6c655051d107739fb0203ae6f21d58a7ad473118528bd64f1ff7d1834120f219 Jul 08 19:28:15.330159 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.330073 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:28:15.354305 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.354277 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-7ntp9" Jul 08 19:28:15.362968 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.362936 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-gfkmh" event={"ID":"6c0d602f-7a0a-415c-9716-54ffb6c51e67","Type":"ContainerStarted","Data":"6c655051d107739fb0203ae6f21d58a7ad473118528bd64f1ff7d1834120f219"} Jul 08 19:28:15.489764 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.489672 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-7ntp9"] Jul 08 19:28:15.492098 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:28:15.492066 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05373d28_0e17_422e_96ec_1add0377dd9d.slice/crio-a803ac771986bf37c3981f17d31f08b0c6a6f06b5f7876a1e6256b721109647d WatchSource:0}: Error finding container a803ac771986bf37c3981f17d31f08b0c6a6f06b5f7876a1e6256b721109647d: Status 404 returned error can't find the container with id a803ac771986bf37c3981f17d31f08b0c6a6f06b5f7876a1e6256b721109647d Jul 08 19:28:15.693904 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.693850 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cee815ab-2b57-4584-8b6c-a8748c849be9-metrics-certs\") pod \"router-default-5846dc5c45-fgr6x\" (UID: \"cee815ab-2b57-4584-8b6c-a8748c849be9\") " pod="openshift-ingress/router-default-5846dc5c45-fgr6x" Jul 08 19:28:15.694081 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.693930 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6e934fe3-7407-4bd1-88cc-21be3143047c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-v4w6v\" (UID: \"6e934fe3-7407-4bd1-88cc-21be3143047c\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-v4w6v" Jul 08 19:28:15.694081 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:28:15.694008 2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 08 19:28:15.694081 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.694017 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cee815ab-2b57-4584-8b6c-a8748c849be9-service-ca-bundle\") pod \"router-default-5846dc5c45-fgr6x\" (UID: \"cee815ab-2b57-4584-8b6c-a8748c849be9\") " pod="openshift-ingress/router-default-5846dc5c45-fgr6x" Jul 08 19:28:15.694081 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:15.694048 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ec79a684-b3d4-4942-aed4-41d82124967d-registry-tls\") pod \"image-registry-6d9dd85c4b-s89jc\" (UID: \"ec79a684-b3d4-4942-aed4-41d82124967d\") " pod="openshift-image-registry/image-registry-6d9dd85c4b-s89jc" Jul 08 19:28:15.694081 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:28:15.694060 2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 08 19:28:15.694081 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:28:15.694073 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cee815ab-2b57-4584-8b6c-a8748c849be9-metrics-certs podName:cee815ab-2b57-4584-8b6c-a8748c849be9 nodeName:}" failed. No retries permitted until 2026-07-08 19:28:16.694053666 +0000 UTC m=+262.580243704 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cee815ab-2b57-4584-8b6c-a8748c849be9-metrics-certs") pod "router-default-5846dc5c45-fgr6x" (UID: "cee815ab-2b57-4584-8b6c-a8748c849be9") : secret "router-metrics-certs-default" not found Jul 08 19:28:15.694339 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:28:15.694109 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e934fe3-7407-4bd1-88cc-21be3143047c-cluster-monitoring-operator-tls podName:6e934fe3-7407-4bd1-88cc-21be3143047c nodeName:}" failed. No retries permitted until 2026-07-08 19:28:16.694095845 +0000 UTC m=+262.580285873 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6e934fe3-7407-4bd1-88cc-21be3143047c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-v4w6v" (UID: "6e934fe3-7407-4bd1-88cc-21be3143047c") : secret "cluster-monitoring-operator-tls" not found Jul 08 19:28:15.694339 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:28:15.694119 2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 08 19:28:15.694339 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:28:15.694130 2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d9dd85c4b-s89jc: secret "image-registry-tls" not found Jul 08 19:28:15.694339 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:28:15.694165 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ec79a684-b3d4-4942-aed4-41d82124967d-registry-tls podName:ec79a684-b3d4-4942-aed4-41d82124967d nodeName:}" failed. No retries permitted until 2026-07-08 19:28:16.694153589 +0000 UTC m=+262.580343645 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ec79a684-b3d4-4942-aed4-41d82124967d-registry-tls") pod "image-registry-6d9dd85c4b-s89jc" (UID: "ec79a684-b3d4-4942-aed4-41d82124967d") : secret "image-registry-tls" not found Jul 08 19:28:15.694339 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:28:15.694190 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cee815ab-2b57-4584-8b6c-a8748c849be9-service-ca-bundle podName:cee815ab-2b57-4584-8b6c-a8748c849be9 nodeName:}" failed. No retries permitted until 2026-07-08 19:28:16.69417028 +0000 UTC m=+262.580360304 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cee815ab-2b57-4584-8b6c-a8748c849be9-service-ca-bundle") pod "router-default-5846dc5c45-fgr6x" (UID: "cee815ab-2b57-4584-8b6c-a8748c849be9") : configmap references non-existent config key: service-ca.crt Jul 08 19:28:16.367554 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:16.367514 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-7ntp9" event={"ID":"05373d28-0e17-422e-96ec-1add0377dd9d","Type":"ContainerStarted","Data":"a803ac771986bf37c3981f17d31f08b0c6a6f06b5f7876a1e6256b721109647d"} Jul 08 19:28:16.703893 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:16.703566 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cee815ab-2b57-4584-8b6c-a8748c849be9-service-ca-bundle\") pod \"router-default-5846dc5c45-fgr6x\" (UID: \"cee815ab-2b57-4584-8b6c-a8748c849be9\") " pod="openshift-ingress/router-default-5846dc5c45-fgr6x" Jul 08 19:28:16.703893 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:16.703618 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ec79a684-b3d4-4942-aed4-41d82124967d-registry-tls\") pod \"image-registry-6d9dd85c4b-s89jc\" (UID: \"ec79a684-b3d4-4942-aed4-41d82124967d\") " pod="openshift-image-registry/image-registry-6d9dd85c4b-s89jc" Jul 08 19:28:16.703893 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:16.703676 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cee815ab-2b57-4584-8b6c-a8748c849be9-metrics-certs\") pod \"router-default-5846dc5c45-fgr6x\" (UID: \"cee815ab-2b57-4584-8b6c-a8748c849be9\") " pod="openshift-ingress/router-default-5846dc5c45-fgr6x" Jul 08 19:28:16.703893 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:28:16.703689 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cee815ab-2b57-4584-8b6c-a8748c849be9-service-ca-bundle podName:cee815ab-2b57-4584-8b6c-a8748c849be9 nodeName:}" failed. No retries permitted until 2026-07-08 19:28:18.70366833 +0000 UTC m=+264.589858361 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cee815ab-2b57-4584-8b6c-a8748c849be9-service-ca-bundle") pod "router-default-5846dc5c45-fgr6x" (UID: "cee815ab-2b57-4584-8b6c-a8748c849be9") : configmap references non-existent config key: service-ca.crt Jul 08 19:28:16.703893 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:16.703754 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6e934fe3-7407-4bd1-88cc-21be3143047c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-v4w6v\" (UID: \"6e934fe3-7407-4bd1-88cc-21be3143047c\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-v4w6v" Jul 08 19:28:16.703893 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:28:16.703763 2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 08 19:28:16.703893 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:28:16.703825 2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 08 19:28:16.703893 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:28:16.703837 2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d9dd85c4b-s89jc: secret "image-registry-tls" not found Jul 08 19:28:16.703893 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:28:16.703849 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cee815ab-2b57-4584-8b6c-a8748c849be9-metrics-certs podName:cee815ab-2b57-4584-8b6c-a8748c849be9 nodeName:}" failed. No retries permitted until 2026-07-08 19:28:18.703836968 +0000 UTC m=+264.590026992 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cee815ab-2b57-4584-8b6c-a8748c849be9-metrics-certs") pod "router-default-5846dc5c45-fgr6x" (UID: "cee815ab-2b57-4584-8b6c-a8748c849be9") : secret "router-metrics-certs-default" not found Jul 08 19:28:16.703893 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:28:16.703912 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ec79a684-b3d4-4942-aed4-41d82124967d-registry-tls podName:ec79a684-b3d4-4942-aed4-41d82124967d nodeName:}" failed. No retries permitted until 2026-07-08 19:28:18.703885211 +0000 UTC m=+264.590075240 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ec79a684-b3d4-4942-aed4-41d82124967d-registry-tls") pod "image-registry-6d9dd85c4b-s89jc" (UID: "ec79a684-b3d4-4942-aed4-41d82124967d") : secret "image-registry-tls" not found Jul 08 19:28:16.704450 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:28:16.703910 2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 08 19:28:16.704450 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:28:16.703953 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e934fe3-7407-4bd1-88cc-21be3143047c-cluster-monitoring-operator-tls podName:6e934fe3-7407-4bd1-88cc-21be3143047c nodeName:}" failed. No retries permitted until 2026-07-08 19:28:18.703941561 +0000 UTC m=+264.590131589 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6e934fe3-7407-4bd1-88cc-21be3143047c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-v4w6v" (UID: "6e934fe3-7407-4bd1-88cc-21be3143047c") : secret "cluster-monitoring-operator-tls" not found Jul 08 19:28:17.376530 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:17.376468 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:28:17.433128 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:17.433095 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:28:18.375596 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:18.375555 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-gfkmh" event={"ID":"6c0d602f-7a0a-415c-9716-54ffb6c51e67","Type":"ContainerStarted","Data":"a9e0335b74d47058567cea0ca77a67ec109d67d1c8c077b80b42a9b9e2ac9fb2"} Jul 08 19:28:18.397405 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:18.397341 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5c847dcfc4-gfkmh" podStartSLOduration=2.207309664 podStartE2EDuration="4.397325884s" totalImagesPullingTime="2.19001622s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:28:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:28:18.396685348 +0000 UTC m=+264.282875396" watchObservedRunningTime="2026-07-08 19:28:18.397325884 +0000 UTC m=+264.283515985" Jul 08 19:28:18.597620 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:18.597585 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-db7nv"] Jul 08 19:28:18.600048 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:18.600030 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-db7nv" Jul 08 19:28:18.602285 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:18.602256 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 08 19:28:18.602385 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:18.602298 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 08 19:28:18.602645 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:18.602627 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5vrqz\"" Jul 08 19:28:18.609310 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:18.609289 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-db7nv"] Jul 08 19:28:18.624301 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:18.624274 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:28:18.722378 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:18.722338 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cee815ab-2b57-4584-8b6c-a8748c849be9-metrics-certs\") pod \"router-default-5846dc5c45-fgr6x\" (UID: \"cee815ab-2b57-4584-8b6c-a8748c849be9\") " pod="openshift-ingress/router-default-5846dc5c45-fgr6x" Jul 08 19:28:18.722378 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:18.722383 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/baee8938-f6f6-4ed9-99f7-832c16ebbd4a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-db7nv\" (UID: \"baee8938-f6f6-4ed9-99f7-832c16ebbd4a\") " pod="openshift-insights/insights-runtime-extractor-db7nv" Jul 08 19:28:18.722608 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:18.722410 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttvkq\" (UniqueName: \"kubernetes.io/projected/baee8938-f6f6-4ed9-99f7-832c16ebbd4a-kube-api-access-ttvkq\") pod \"insights-runtime-extractor-db7nv\" (UID: \"baee8938-f6f6-4ed9-99f7-832c16ebbd4a\") " pod="openshift-insights/insights-runtime-extractor-db7nv" Jul 08 19:28:18.722608 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:28:18.722492 2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 08 19:28:18.722608 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:28:18.722552 2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 08 19:28:18.722608 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:18.722491 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6e934fe3-7407-4bd1-88cc-21be3143047c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-v4w6v\" (UID: \"6e934fe3-7407-4bd1-88cc-21be3143047c\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-v4w6v" Jul 08 19:28:18.722608 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:28:18.722556 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cee815ab-2b57-4584-8b6c-a8748c849be9-metrics-certs podName:cee815ab-2b57-4584-8b6c-a8748c849be9 nodeName:}" failed. No retries permitted until 2026-07-08 19:28:22.722540186 +0000 UTC m=+268.608730211 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cee815ab-2b57-4584-8b6c-a8748c849be9-metrics-certs") pod "router-default-5846dc5c45-fgr6x" (UID: "cee815ab-2b57-4584-8b6c-a8748c849be9") : secret "router-metrics-certs-default" not found Jul 08 19:28:18.722765 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:28:18.722615 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e934fe3-7407-4bd1-88cc-21be3143047c-cluster-monitoring-operator-tls podName:6e934fe3-7407-4bd1-88cc-21be3143047c nodeName:}" failed. No retries permitted until 2026-07-08 19:28:22.722599534 +0000 UTC m=+268.608789563 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6e934fe3-7407-4bd1-88cc-21be3143047c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-v4w6v" (UID: "6e934fe3-7407-4bd1-88cc-21be3143047c") : secret "cluster-monitoring-operator-tls" not found Jul 08 19:28:18.722765 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:18.722648 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/baee8938-f6f6-4ed9-99f7-832c16ebbd4a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-db7nv\" (UID: \"baee8938-f6f6-4ed9-99f7-832c16ebbd4a\") " pod="openshift-insights/insights-runtime-extractor-db7nv" Jul 08 19:28:18.722765 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:18.722693 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/baee8938-f6f6-4ed9-99f7-832c16ebbd4a-crio-socket\") pod \"insights-runtime-extractor-db7nv\" (UID: \"baee8938-f6f6-4ed9-99f7-832c16ebbd4a\") " pod="openshift-insights/insights-runtime-extractor-db7nv" Jul 08 19:28:18.722765 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:18.722742 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cee815ab-2b57-4584-8b6c-a8748c849be9-service-ca-bundle\") pod \"router-default-5846dc5c45-fgr6x\" (UID: \"cee815ab-2b57-4584-8b6c-a8748c849be9\") " pod="openshift-ingress/router-default-5846dc5c45-fgr6x" Jul 08 19:28:18.722765 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:18.722764 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ec79a684-b3d4-4942-aed4-41d82124967d-registry-tls\") pod \"image-registry-6d9dd85c4b-s89jc\" (UID: \"ec79a684-b3d4-4942-aed4-41d82124967d\") " pod="openshift-image-registry/image-registry-6d9dd85c4b-s89jc" Jul 08 19:28:18.722961 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:18.722788 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/baee8938-f6f6-4ed9-99f7-832c16ebbd4a-data-volume\") pod \"insights-runtime-extractor-db7nv\" (UID: \"baee8938-f6f6-4ed9-99f7-832c16ebbd4a\") " pod="openshift-insights/insights-runtime-extractor-db7nv" Jul 08 19:28:18.722961 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:28:18.722837 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cee815ab-2b57-4584-8b6c-a8748c849be9-service-ca-bundle podName:cee815ab-2b57-4584-8b6c-a8748c849be9 nodeName:}" failed. No retries permitted until 2026-07-08 19:28:22.72282496 +0000 UTC m=+268.609014984 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cee815ab-2b57-4584-8b6c-a8748c849be9-service-ca-bundle") pod "router-default-5846dc5c45-fgr6x" (UID: "cee815ab-2b57-4584-8b6c-a8748c849be9") : configmap references non-existent config key: service-ca.crt Jul 08 19:28:18.722961 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:28:18.722886 2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 08 19:28:18.722961 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:28:18.722897 2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d9dd85c4b-s89jc: secret "image-registry-tls" not found Jul 08 19:28:18.722961 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:28:18.722936 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ec79a684-b3d4-4942-aed4-41d82124967d-registry-tls podName:ec79a684-b3d4-4942-aed4-41d82124967d nodeName:}" failed. No retries permitted until 2026-07-08 19:28:22.722927926 +0000 UTC m=+268.609117952 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ec79a684-b3d4-4942-aed4-41d82124967d-registry-tls") pod "image-registry-6d9dd85c4b-s89jc" (UID: "ec79a684-b3d4-4942-aed4-41d82124967d") : secret "image-registry-tls" not found Jul 08 19:28:18.823968 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:18.823934 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/baee8938-f6f6-4ed9-99f7-832c16ebbd4a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-db7nv\" (UID: \"baee8938-f6f6-4ed9-99f7-832c16ebbd4a\") " pod="openshift-insights/insights-runtime-extractor-db7nv" Jul 08 19:28:18.824149 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:18.823985 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/baee8938-f6f6-4ed9-99f7-832c16ebbd4a-crio-socket\") pod \"insights-runtime-extractor-db7nv\" (UID: \"baee8938-f6f6-4ed9-99f7-832c16ebbd4a\") " pod="openshift-insights/insights-runtime-extractor-db7nv" Jul 08 19:28:18.824149 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:18.824033 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/baee8938-f6f6-4ed9-99f7-832c16ebbd4a-data-volume\") pod \"insights-runtime-extractor-db7nv\" (UID: \"baee8938-f6f6-4ed9-99f7-832c16ebbd4a\") " pod="openshift-insights/insights-runtime-extractor-db7nv" Jul 08 19:28:18.824149 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:18.824068 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/baee8938-f6f6-4ed9-99f7-832c16ebbd4a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-db7nv\" (UID: \"baee8938-f6f6-4ed9-99f7-832c16ebbd4a\") " pod="openshift-insights/insights-runtime-extractor-db7nv" Jul 08 19:28:18.824149 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:18.824085 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttvkq\" (UniqueName: \"kubernetes.io/projected/baee8938-f6f6-4ed9-99f7-832c16ebbd4a-kube-api-access-ttvkq\") pod \"insights-runtime-extractor-db7nv\" (UID: \"baee8938-f6f6-4ed9-99f7-832c16ebbd4a\") " pod="openshift-insights/insights-runtime-extractor-db7nv" Jul 08 19:28:18.824149 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:28:18.824092 2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 08 19:28:18.824331 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:28:18.824152 2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 08 19:28:18.824331 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:28:18.824171 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/baee8938-f6f6-4ed9-99f7-832c16ebbd4a-insights-runtime-extractor-tls podName:baee8938-f6f6-4ed9-99f7-832c16ebbd4a nodeName:}" failed. No retries permitted until 2026-07-08 19:28:19.324150492 +0000 UTC m=+265.210340540 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/baee8938-f6f6-4ed9-99f7-832c16ebbd4a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-db7nv" (UID: "baee8938-f6f6-4ed9-99f7-832c16ebbd4a") : secret "insights-runtime-extractor-tls" not found Jul 08 19:28:18.824331 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:28:18.824206 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/baee8938-f6f6-4ed9-99f7-832c16ebbd4a-kube-rbac-proxy-cm podName:baee8938-f6f6-4ed9-99f7-832c16ebbd4a nodeName:}" failed. No retries permitted until 2026-07-08 19:28:19.324194044 +0000 UTC m=+265.210384084 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/baee8938-f6f6-4ed9-99f7-832c16ebbd4a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-db7nv" (UID: "baee8938-f6f6-4ed9-99f7-832c16ebbd4a") : configmap "kube-rbac-proxy" not found Jul 08 19:28:18.824331 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:18.824150 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/baee8938-f6f6-4ed9-99f7-832c16ebbd4a-crio-socket\") pod \"insights-runtime-extractor-db7nv\" (UID: \"baee8938-f6f6-4ed9-99f7-832c16ebbd4a\") " pod="openshift-insights/insights-runtime-extractor-db7nv" Jul 08 19:28:18.824459 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:18.824383 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/baee8938-f6f6-4ed9-99f7-832c16ebbd4a-data-volume\") pod \"insights-runtime-extractor-db7nv\" (UID: \"baee8938-f6f6-4ed9-99f7-832c16ebbd4a\") " pod="openshift-insights/insights-runtime-extractor-db7nv" Jul 08 19:28:18.831603 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:18.831581 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttvkq\" (UniqueName: \"kubernetes.io/projected/baee8938-f6f6-4ed9-99f7-832c16ebbd4a-kube-api-access-ttvkq\") pod \"insights-runtime-extractor-db7nv\" (UID: \"baee8938-f6f6-4ed9-99f7-832c16ebbd4a\") " pod="openshift-insights/insights-runtime-extractor-db7nv" Jul 08 19:28:19.329008 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:19.328973 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/baee8938-f6f6-4ed9-99f7-832c16ebbd4a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-db7nv\" (UID: \"baee8938-f6f6-4ed9-99f7-832c16ebbd4a\") " pod="openshift-insights/insights-runtime-extractor-db7nv" Jul 08 19:28:19.329179 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:19.329037 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/baee8938-f6f6-4ed9-99f7-832c16ebbd4a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-db7nv\" (UID: \"baee8938-f6f6-4ed9-99f7-832c16ebbd4a\") " pod="openshift-insights/insights-runtime-extractor-db7nv" Jul 08 19:28:19.329179 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:28:19.329101 2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 08 19:28:19.329179 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:28:19.329162 2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 08 19:28:19.329179 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:28:19.329169 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/baee8938-f6f6-4ed9-99f7-832c16ebbd4a-kube-rbac-proxy-cm podName:baee8938-f6f6-4ed9-99f7-832c16ebbd4a nodeName:}" failed. No retries permitted until 2026-07-08 19:28:20.329151325 +0000 UTC m=+266.215341366 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/baee8938-f6f6-4ed9-99f7-832c16ebbd4a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-db7nv" (UID: "baee8938-f6f6-4ed9-99f7-832c16ebbd4a") : configmap "kube-rbac-proxy" not found Jul 08 19:28:19.329304 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:28:19.329196 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/baee8938-f6f6-4ed9-99f7-832c16ebbd4a-insights-runtime-extractor-tls podName:baee8938-f6f6-4ed9-99f7-832c16ebbd4a nodeName:}" failed. No retries permitted until 2026-07-08 19:28:20.329186879 +0000 UTC m=+266.215376904 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/baee8938-f6f6-4ed9-99f7-832c16ebbd4a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-db7nv" (UID: "baee8938-f6f6-4ed9-99f7-832c16ebbd4a") : secret "insights-runtime-extractor-tls" not found Jul 08 19:28:19.954685 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:19.954646 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:28:20.065876 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:20.065832 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:28:20.182183 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:20.182150 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:28:20.337671 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:20.337637 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/baee8938-f6f6-4ed9-99f7-832c16ebbd4a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-db7nv\" (UID: \"baee8938-f6f6-4ed9-99f7-832c16ebbd4a\") " pod="openshift-insights/insights-runtime-extractor-db7nv" Jul 08 19:28:20.337836 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:20.337761 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/baee8938-f6f6-4ed9-99f7-832c16ebbd4a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-db7nv\" (UID: \"baee8938-f6f6-4ed9-99f7-832c16ebbd4a\") " pod="openshift-insights/insights-runtime-extractor-db7nv" Jul 08 19:28:20.337836 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:28:20.337788 2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 08 19:28:20.337936 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:28:20.337846 2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 08 19:28:20.337936 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:28:20.337850 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/baee8938-f6f6-4ed9-99f7-832c16ebbd4a-insights-runtime-extractor-tls podName:baee8938-f6f6-4ed9-99f7-832c16ebbd4a nodeName:}" failed. No retries permitted until 2026-07-08 19:28:22.337834845 +0000 UTC m=+268.224024870 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/baee8938-f6f6-4ed9-99f7-832c16ebbd4a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-db7nv" (UID: "baee8938-f6f6-4ed9-99f7-832c16ebbd4a") : secret "insights-runtime-extractor-tls" not found Jul 08 19:28:20.337936 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:28:20.337913 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/baee8938-f6f6-4ed9-99f7-832c16ebbd4a-kube-rbac-proxy-cm podName:baee8938-f6f6-4ed9-99f7-832c16ebbd4a nodeName:}" failed. No retries permitted until 2026-07-08 19:28:22.337901581 +0000 UTC m=+268.224091606 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/baee8938-f6f6-4ed9-99f7-832c16ebbd4a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-db7nv" (UID: "baee8938-f6f6-4ed9-99f7-832c16ebbd4a") : configmap "kube-rbac-proxy" not found Jul 08 19:28:20.377207 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:20.377180 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dghh8_474e5893-2b36-4b57-b154-15b0f8b4f300/dns-node-resolver/0.log" Jul 08 19:28:20.382957 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:20.382930 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-7ntp9" event={"ID":"05373d28-0e17-422e-96ec-1add0377dd9d","Type":"ContainerStarted","Data":"d01e0c9dc5c50c4ce9c48083a3093b06da7826a74e5a276c7c17715e1bd448a5"} Jul 08 19:28:20.440109 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:20.440003 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-7ntp9" podStartSLOduration=2.060737625 podStartE2EDuration="6.439986957s" totalImagesPullingTime="4.379249332s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:28:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:28:20.438819363 +0000 UTC m=+266.325009417" watchObservedRunningTime="2026-07-08 19:28:20.439986957 +0000 UTC m=+266.326177003" Jul 08 19:28:20.882738 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:20.882658 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:28:20.980792 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:20.980758 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:28:21.376467 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:21.376444 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6w52p_8b78ed73-731b-49fa-9220-4b1e45787806/node-ca/0.log" Jul 08 19:28:21.602126 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:21.602091 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:28:21.710114 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:21.710082 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:28:22.176233 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:22.176207 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5nfnp_d36b2cab-e4d2-4a23-85f9-0bd24ccbb497/serve-healthcheck-canary/0.log" Jul 08 19:28:22.358429 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:22.358373 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/baee8938-f6f6-4ed9-99f7-832c16ebbd4a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-db7nv\" (UID: \"baee8938-f6f6-4ed9-99f7-832c16ebbd4a\") " pod="openshift-insights/insights-runtime-extractor-db7nv" Jul 08 19:28:22.358632 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:22.358543 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/baee8938-f6f6-4ed9-99f7-832c16ebbd4a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-db7nv\" (UID: \"baee8938-f6f6-4ed9-99f7-832c16ebbd4a\") " pod="openshift-insights/insights-runtime-extractor-db7nv" Jul 08 19:28:22.358632 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:28:22.358548 2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 08 19:28:22.358811 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:28:22.358581 2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 08 19:28:22.358811 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:28:22.358663 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/baee8938-f6f6-4ed9-99f7-832c16ebbd4a-insights-runtime-extractor-tls podName:baee8938-f6f6-4ed9-99f7-832c16ebbd4a nodeName:}" failed. No retries permitted until 2026-07-08 19:28:26.358642708 +0000 UTC m=+272.244832739 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/baee8938-f6f6-4ed9-99f7-832c16ebbd4a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-db7nv" (UID: "baee8938-f6f6-4ed9-99f7-832c16ebbd4a") : secret "insights-runtime-extractor-tls" not found Jul 08 19:28:22.358811 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:28:22.358700 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/baee8938-f6f6-4ed9-99f7-832c16ebbd4a-kube-rbac-proxy-cm podName:baee8938-f6f6-4ed9-99f7-832c16ebbd4a nodeName:}" failed. No retries permitted until 2026-07-08 19:28:26.358686412 +0000 UTC m=+272.244876436 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/baee8938-f6f6-4ed9-99f7-832c16ebbd4a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-db7nv" (UID: "baee8938-f6f6-4ed9-99f7-832c16ebbd4a") : configmap "kube-rbac-proxy" not found Jul 08 19:28:22.762576 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:22.762534 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cee815ab-2b57-4584-8b6c-a8748c849be9-metrics-certs\") pod \"router-default-5846dc5c45-fgr6x\" (UID: \"cee815ab-2b57-4584-8b6c-a8748c849be9\") " pod="openshift-ingress/router-default-5846dc5c45-fgr6x" Jul 08 19:28:22.762786 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:22.762606 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6e934fe3-7407-4bd1-88cc-21be3143047c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-v4w6v\" (UID: \"6e934fe3-7407-4bd1-88cc-21be3143047c\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-v4w6v" Jul 08 19:28:22.762786 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:28:22.762684 2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 08 19:28:22.762786 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:22.762698 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cee815ab-2b57-4584-8b6c-a8748c849be9-service-ca-bundle\") pod \"router-default-5846dc5c45-fgr6x\" (UID: \"cee815ab-2b57-4584-8b6c-a8748c849be9\") " pod="openshift-ingress/router-default-5846dc5c45-fgr6x" Jul 08 19:28:22.762786 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:22.762726 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ec79a684-b3d4-4942-aed4-41d82124967d-registry-tls\") pod \"image-registry-6d9dd85c4b-s89jc\" (UID: \"ec79a684-b3d4-4942-aed4-41d82124967d\") " pod="openshift-image-registry/image-registry-6d9dd85c4b-s89jc" Jul 08 19:28:22.762786 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:28:22.762741 2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 08 19:28:22.762786 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:28:22.762748 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cee815ab-2b57-4584-8b6c-a8748c849be9-metrics-certs podName:cee815ab-2b57-4584-8b6c-a8748c849be9 nodeName:}" failed. No retries permitted until 2026-07-08 19:28:30.762732648 +0000 UTC m=+276.648922673 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cee815ab-2b57-4584-8b6c-a8748c849be9-metrics-certs") pod "router-default-5846dc5c45-fgr6x" (UID: "cee815ab-2b57-4584-8b6c-a8748c849be9") : secret "router-metrics-certs-default" not found Jul 08 19:28:22.763066 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:28:22.762804 2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 08 19:28:22.763066 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:28:22.762811 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e934fe3-7407-4bd1-88cc-21be3143047c-cluster-monitoring-operator-tls podName:6e934fe3-7407-4bd1-88cc-21be3143047c nodeName:}" failed. No retries permitted until 2026-07-08 19:28:30.762796196 +0000 UTC m=+276.648986232 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6e934fe3-7407-4bd1-88cc-21be3143047c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-v4w6v" (UID: "6e934fe3-7407-4bd1-88cc-21be3143047c") : secret "cluster-monitoring-operator-tls" not found Jul 08 19:28:22.763066 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:28:22.762817 2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d9dd85c4b-s89jc: secret "image-registry-tls" not found Jul 08 19:28:22.763066 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:28:22.762882 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ec79a684-b3d4-4942-aed4-41d82124967d-registry-tls podName:ec79a684-b3d4-4942-aed4-41d82124967d nodeName:}" failed. No retries permitted until 2026-07-08 19:28:30.762845553 +0000 UTC m=+276.649035584 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ec79a684-b3d4-4942-aed4-41d82124967d-registry-tls") pod "image-registry-6d9dd85c4b-s89jc" (UID: "ec79a684-b3d4-4942-aed4-41d82124967d") : secret "image-registry-tls" not found Jul 08 19:28:22.763066 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:28:22.762911 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cee815ab-2b57-4584-8b6c-a8748c849be9-service-ca-bundle podName:cee815ab-2b57-4584-8b6c-a8748c849be9 nodeName:}" failed. No retries permitted until 2026-07-08 19:28:30.762899513 +0000 UTC m=+276.649089538 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cee815ab-2b57-4584-8b6c-a8748c849be9-service-ca-bundle") pod "router-default-5846dc5c45-fgr6x" (UID: "cee815ab-2b57-4584-8b6c-a8748c849be9") : configmap references non-existent config key: service-ca.crt Jul 08 19:28:23.263700 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:23.263664 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:28:23.390577 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:23.390535 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:28:24.589914 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:24.589857 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:28:26.393043 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:26.392997 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/baee8938-f6f6-4ed9-99f7-832c16ebbd4a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-db7nv\" (UID: \"baee8938-f6f6-4ed9-99f7-832c16ebbd4a\") " pod="openshift-insights/insights-runtime-extractor-db7nv" Jul 08 19:28:26.393526 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:28:26.393152 2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 08 19:28:26.393526 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:26.393159 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/baee8938-f6f6-4ed9-99f7-832c16ebbd4a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-db7nv\" (UID: \"baee8938-f6f6-4ed9-99f7-832c16ebbd4a\") " pod="openshift-insights/insights-runtime-extractor-db7nv" Jul 08 19:28:26.393526 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:28:26.393225 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/baee8938-f6f6-4ed9-99f7-832c16ebbd4a-insights-runtime-extractor-tls podName:baee8938-f6f6-4ed9-99f7-832c16ebbd4a nodeName:}" failed. No retries permitted until 2026-07-08 19:28:34.393202196 +0000 UTC m=+280.279392235 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/baee8938-f6f6-4ed9-99f7-832c16ebbd4a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-db7nv" (UID: "baee8938-f6f6-4ed9-99f7-832c16ebbd4a") : secret "insights-runtime-extractor-tls" not found Jul 08 19:28:26.393526 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:28:26.393266 2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 08 19:28:26.393526 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:28:26.393317 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/baee8938-f6f6-4ed9-99f7-832c16ebbd4a-kube-rbac-proxy-cm podName:baee8938-f6f6-4ed9-99f7-832c16ebbd4a nodeName:}" failed. No retries permitted until 2026-07-08 19:28:34.393303354 +0000 UTC m=+280.279493382 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/baee8938-f6f6-4ed9-99f7-832c16ebbd4a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-db7nv" (UID: "baee8938-f6f6-4ed9-99f7-832c16ebbd4a") : configmap "kube-rbac-proxy" not found Jul 08 19:28:27.461571 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:27.461534 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:28:27.718045 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:27.717949 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:28:27.811789 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:27.811756 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:28:27.908112 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:27.908082 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:28:28.010062 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:28.009983 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:28:28.161874 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:28.161832 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:28:29.142488 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:29.142454 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:28:29.536190 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:29.536138 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:28:29.738182 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:29.738136 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:28:29.942005 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:29.941976 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:28:30.334844 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:30.334760 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:28:30.832977 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:30.832942 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cee815ab-2b57-4584-8b6c-a8748c849be9-service-ca-bundle\") pod \"router-default-5846dc5c45-fgr6x\" (UID: \"cee815ab-2b57-4584-8b6c-a8748c849be9\") " pod="openshift-ingress/router-default-5846dc5c45-fgr6x" Jul 08 19:28:30.832977 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:30.832979 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ec79a684-b3d4-4942-aed4-41d82124967d-registry-tls\") pod \"image-registry-6d9dd85c4b-s89jc\" (UID: \"ec79a684-b3d4-4942-aed4-41d82124967d\") " pod="openshift-image-registry/image-registry-6d9dd85c4b-s89jc" Jul 08 19:28:30.833215 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:30.833028 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cee815ab-2b57-4584-8b6c-a8748c849be9-metrics-certs\") pod \"router-default-5846dc5c45-fgr6x\" (UID: \"cee815ab-2b57-4584-8b6c-a8748c849be9\") " pod="openshift-ingress/router-default-5846dc5c45-fgr6x" Jul 08 19:28:30.833215 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:30.833076 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6e934fe3-7407-4bd1-88cc-21be3143047c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-v4w6v\" (UID: \"6e934fe3-7407-4bd1-88cc-21be3143047c\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-v4w6v" Jul 08 19:28:30.833215 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:28:30.833109 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cee815ab-2b57-4584-8b6c-a8748c849be9-service-ca-bundle podName:cee815ab-2b57-4584-8b6c-a8748c849be9 nodeName:}" failed. No retries permitted until 2026-07-08 19:28:46.833089095 +0000 UTC m=+292.719279140 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cee815ab-2b57-4584-8b6c-a8748c849be9-service-ca-bundle") pod "router-default-5846dc5c45-fgr6x" (UID: "cee815ab-2b57-4584-8b6c-a8748c849be9") : configmap references non-existent config key: service-ca.crt Jul 08 19:28:30.833215 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:28:30.833158 2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 08 19:28:30.833215 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:28:30.833163 2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 08 19:28:30.833215 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:28:30.833197 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e934fe3-7407-4bd1-88cc-21be3143047c-cluster-monitoring-operator-tls podName:6e934fe3-7407-4bd1-88cc-21be3143047c nodeName:}" failed. No retries permitted until 2026-07-08 19:28:46.833186193 +0000 UTC m=+292.719376218 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6e934fe3-7407-4bd1-88cc-21be3143047c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-v4w6v" (UID: "6e934fe3-7407-4bd1-88cc-21be3143047c") : secret "cluster-monitoring-operator-tls" not found Jul 08 19:28:30.833215 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:28:30.833163 2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 08 19:28:30.833215 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:28:30.833213 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cee815ab-2b57-4584-8b6c-a8748c849be9-metrics-certs podName:cee815ab-2b57-4584-8b6c-a8748c849be9 nodeName:}" failed. No retries permitted until 2026-07-08 19:28:46.83320746 +0000 UTC m=+292.719397485 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cee815ab-2b57-4584-8b6c-a8748c849be9-metrics-certs") pod "router-default-5846dc5c45-fgr6x" (UID: "cee815ab-2b57-4584-8b6c-a8748c849be9") : secret "router-metrics-certs-default" not found Jul 08 19:28:30.833479 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:28:30.833225 2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d9dd85c4b-s89jc: secret "image-registry-tls" not found Jul 08 19:28:30.833479 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:28:30.833283 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ec79a684-b3d4-4942-aed4-41d82124967d-registry-tls podName:ec79a684-b3d4-4942-aed4-41d82124967d nodeName:}" failed. No retries permitted until 2026-07-08 19:28:46.833267873 +0000 UTC m=+292.719457898 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ec79a684-b3d4-4942-aed4-41d82124967d-registry-tls") pod "image-registry-6d9dd85c4b-s89jc" (UID: "ec79a684-b3d4-4942-aed4-41d82124967d") : secret "image-registry-tls" not found Jul 08 19:28:31.149078 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:31.148990 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:28:31.256062 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:31.256033 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:28:31.356008 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:31.355982 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:28:31.420648 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:31.420620 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:28:31.746765 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:31.746681 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:28:31.842877 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:31.842838 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:28:31.941739 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:31.941704 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:28:32.349546 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:32.349516 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:28:32.486899 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:32.486852 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:28:32.584775 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:32.584746 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:28:32.946041 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:32.946003 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:28:33.534450 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:33.534422 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:28:33.735798 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:33.735763 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:28:33.970073 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:33.970039 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:28:34.042013 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:34.041985 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:28:34.167236 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:34.167204 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:28:34.463754 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:34.463718 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/baee8938-f6f6-4ed9-99f7-832c16ebbd4a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-db7nv\" (UID: \"baee8938-f6f6-4ed9-99f7-832c16ebbd4a\") " pod="openshift-insights/insights-runtime-extractor-db7nv" Jul 08 19:28:34.463960 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:34.463769 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/baee8938-f6f6-4ed9-99f7-832c16ebbd4a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-db7nv\" (UID: \"baee8938-f6f6-4ed9-99f7-832c16ebbd4a\") " pod="openshift-insights/insights-runtime-extractor-db7nv" Jul 08 19:28:34.463960 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:28:34.463889 2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 08 19:28:34.463960 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:28:34.463950 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/baee8938-f6f6-4ed9-99f7-832c16ebbd4a-insights-runtime-extractor-tls podName:baee8938-f6f6-4ed9-99f7-832c16ebbd4a nodeName:}" failed. No retries permitted until 2026-07-08 19:28:50.463934048 +0000 UTC m=+296.350124089 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/baee8938-f6f6-4ed9-99f7-832c16ebbd4a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-db7nv" (UID: "baee8938-f6f6-4ed9-99f7-832c16ebbd4a") : secret "insights-runtime-extractor-tls" not found Jul 08 19:28:34.464205 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:34.464187 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/baee8938-f6f6-4ed9-99f7-832c16ebbd4a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-db7nv\" (UID: \"baee8938-f6f6-4ed9-99f7-832c16ebbd4a\") " pod="openshift-insights/insights-runtime-extractor-db7nv" Jul 08 19:28:35.116022 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:28:35.115975 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-npv6z" podUID="e2bd191d-a59d-4dbe-811f-44e2c4ca16e1" Jul 08 19:28:35.347121 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:35.347086 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:28:35.426117 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:35.426094 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-npv6z" Jul 08 19:28:35.548162 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:35.548131 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:28:35.743571 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:35.743475 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:28:35.855686 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:35.855655 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:28:35.966883 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:35.966833 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:28:36.152060 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:36.152036 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:28:37.355260 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:37.355222 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:28:37.544227 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:37.544195 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:28:37.738594 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:37.738565 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:28:37.936563 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:37.936509 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:28:38.134349 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:38.134273 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:28:38.894413 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:38.894373 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2bd191d-a59d-4dbe-811f-44e2c4ca16e1-metrics-tls\") pod \"dns-default-npv6z\" (UID: \"e2bd191d-a59d-4dbe-811f-44e2c4ca16e1\") " pod="openshift-dns/dns-default-npv6z" Jul 08 19:28:38.896693 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:38.896662 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2bd191d-a59d-4dbe-811f-44e2c4ca16e1-metrics-tls\") pod \"dns-default-npv6z\" (UID: \"e2bd191d-a59d-4dbe-811f-44e2c4ca16e1\") " pod="openshift-dns/dns-default-npv6z" Jul 08 19:28:39.030988 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:39.030955 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dclc4\"" Jul 08 19:28:39.037775 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:39.037747 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-npv6z" Jul 08 19:28:39.163285 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:39.163255 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-npv6z"] Jul 08 19:28:39.166586 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:28:39.166552 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2bd191d_a59d_4dbe_811f_44e2c4ca16e1.slice/crio-f5c3a25b1b6a9934e394aff33df48ab6d2a167d23a46c42b2fd19b76593ed332 WatchSource:0}: Error finding container f5c3a25b1b6a9934e394aff33df48ab6d2a167d23a46c42b2fd19b76593ed332: Status 404 returned error can't find the container with id f5c3a25b1b6a9934e394aff33df48ab6d2a167d23a46c42b2fd19b76593ed332 Jul 08 19:28:39.438145 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:39.438057 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-npv6z" event={"ID":"e2bd191d-a59d-4dbe-811f-44e2c4ca16e1","Type":"ContainerStarted","Data":"f5c3a25b1b6a9934e394aff33df48ab6d2a167d23a46c42b2fd19b76593ed332"} Jul 08 19:28:41.449772 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:41.449734 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-npv6z" event={"ID":"e2bd191d-a59d-4dbe-811f-44e2c4ca16e1","Type":"ContainerStarted","Data":"11ecf2699e38921eafab3f7685fd46c0e33edc1d27ade35b8b151eafcca5ac30"} Jul 08 19:28:41.450224 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:41.449809 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-npv6z" event={"ID":"e2bd191d-a59d-4dbe-811f-44e2c4ca16e1","Type":"ContainerStarted","Data":"7b2e761bc814899a5612657680715677a98caed6f780b871a41950f4aff323c9"} Jul 08 19:28:41.450224 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:41.449898 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-npv6z" Jul 08 19:28:41.473155 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:41.473104 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-npv6z" podStartSLOduration=251.927525658 podStartE2EDuration="4m13.47308953s" totalImagesPullingTime="1.545563872s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:24:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:28:41.47221625 +0000 UTC m=+287.358406309" watchObservedRunningTime="2026-07-08 19:28:41.47308953 +0000 UTC m=+287.359279577" Jul 08 19:28:43.400156 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:43.400121 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6d9dd85c4b-s89jc"] Jul 08 19:28:43.400545 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:28:43.400347 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-6d9dd85c4b-s89jc" podUID="ec79a684-b3d4-4942-aed4-41d82124967d" Jul 08 19:28:43.445461 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:43.445430 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6955c4f578-svncz"] Jul 08 19:28:43.448550 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:43.448530 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6955c4f578-svncz" Jul 08 19:28:43.455705 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:43.455685 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d9dd85c4b-s89jc" Jul 08 19:28:43.461563 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:43.461547 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d9dd85c4b-s89jc" Jul 08 19:28:43.464691 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:43.464669 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6955c4f578-svncz"] Jul 08 19:28:43.533220 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:43.533190 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8n9kp\" (UniqueName: \"kubernetes.io/projected/ec79a684-b3d4-4942-aed4-41d82124967d-kube-api-access-8n9kp\") pod \"ec79a684-b3d4-4942-aed4-41d82124967d\" (UID: \"ec79a684-b3d4-4942-aed4-41d82124967d\") " Jul 08 19:28:43.533402 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:43.533230 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ec79a684-b3d4-4942-aed4-41d82124967d-bound-sa-token\") pod \"ec79a684-b3d4-4942-aed4-41d82124967d\" (UID: \"ec79a684-b3d4-4942-aed4-41d82124967d\") " Jul 08 19:28:43.533402 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:43.533266 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ec79a684-b3d4-4942-aed4-41d82124967d-registry-certificates\") pod \"ec79a684-b3d4-4942-aed4-41d82124967d\" (UID: \"ec79a684-b3d4-4942-aed4-41d82124967d\") " Jul 08 19:28:43.533402 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:43.533300 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ec79a684-b3d4-4942-aed4-41d82124967d-trusted-ca\") pod \"ec79a684-b3d4-4942-aed4-41d82124967d\" (UID: \"ec79a684-b3d4-4942-aed4-41d82124967d\") " Jul 08 19:28:43.533544 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:43.533400 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ec79a684-b3d4-4942-aed4-41d82124967d-ca-trust-extracted\") pod \"ec79a684-b3d4-4942-aed4-41d82124967d\" (UID: \"ec79a684-b3d4-4942-aed4-41d82124967d\") " Jul 08 19:28:43.533544 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:43.533467 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ec79a684-b3d4-4942-aed4-41d82124967d-installation-pull-secrets\") pod \"ec79a684-b3d4-4942-aed4-41d82124967d\" (UID: \"ec79a684-b3d4-4942-aed4-41d82124967d\") " Jul 08 19:28:43.533544 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:43.533518 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ec79a684-b3d4-4942-aed4-41d82124967d-image-registry-private-configuration\") pod \"ec79a684-b3d4-4942-aed4-41d82124967d\" (UID: \"ec79a684-b3d4-4942-aed4-41d82124967d\") " Jul 08 19:28:43.533688 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:43.533665 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwl7r\" (UniqueName: \"kubernetes.io/projected/f6bf0fd3-2e15-4d5e-91cf-418a6255cf75-kube-api-access-cwl7r\") pod \"image-registry-6955c4f578-svncz\" (UID: \"f6bf0fd3-2e15-4d5e-91cf-418a6255cf75\") " pod="openshift-image-registry/image-registry-6955c4f578-svncz" Jul 08 19:28:43.533741 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:43.533667 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec79a684-b3d4-4942-aed4-41d82124967d-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "ec79a684-b3d4-4942-aed4-41d82124967d" (UID: "ec79a684-b3d4-4942-aed4-41d82124967d"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:28:43.533741 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:43.533682 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec79a684-b3d4-4942-aed4-41d82124967d-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "ec79a684-b3d4-4942-aed4-41d82124967d" (UID: "ec79a684-b3d4-4942-aed4-41d82124967d"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:28:43.533855 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:43.533747 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec79a684-b3d4-4942-aed4-41d82124967d-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "ec79a684-b3d4-4942-aed4-41d82124967d" (UID: "ec79a684-b3d4-4942-aed4-41d82124967d"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:28:43.533855 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:43.533760 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f6bf0fd3-2e15-4d5e-91cf-418a6255cf75-bound-sa-token\") pod \"image-registry-6955c4f578-svncz\" (UID: \"f6bf0fd3-2e15-4d5e-91cf-418a6255cf75\") " pod="openshift-image-registry/image-registry-6955c4f578-svncz" Jul 08 19:28:43.533855 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:43.533807 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f6bf0fd3-2e15-4d5e-91cf-418a6255cf75-registry-certificates\") pod \"image-registry-6955c4f578-svncz\" (UID: \"f6bf0fd3-2e15-4d5e-91cf-418a6255cf75\") " pod="openshift-image-registry/image-registry-6955c4f578-svncz" Jul 08 19:28:43.533855 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:43.533835 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f6bf0fd3-2e15-4d5e-91cf-418a6255cf75-ca-trust-extracted\") pod \"image-registry-6955c4f578-svncz\" (UID: \"f6bf0fd3-2e15-4d5e-91cf-418a6255cf75\") " pod="openshift-image-registry/image-registry-6955c4f578-svncz" Jul 08 19:28:43.534073 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:43.533930 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f6bf0fd3-2e15-4d5e-91cf-418a6255cf75-registry-tls\") pod \"image-registry-6955c4f578-svncz\" (UID: \"f6bf0fd3-2e15-4d5e-91cf-418a6255cf75\") " pod="openshift-image-registry/image-registry-6955c4f578-svncz" Jul 08 19:28:43.534073 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:43.533986 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f6bf0fd3-2e15-4d5e-91cf-418a6255cf75-image-registry-private-configuration\") pod \"image-registry-6955c4f578-svncz\" (UID: \"f6bf0fd3-2e15-4d5e-91cf-418a6255cf75\") " pod="openshift-image-registry/image-registry-6955c4f578-svncz" Jul 08 19:28:43.534073 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:43.534028 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f6bf0fd3-2e15-4d5e-91cf-418a6255cf75-trusted-ca\") pod \"image-registry-6955c4f578-svncz\" (UID: \"f6bf0fd3-2e15-4d5e-91cf-418a6255cf75\") " pod="openshift-image-registry/image-registry-6955c4f578-svncz" Jul 08 19:28:43.534073 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:43.534070 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f6bf0fd3-2e15-4d5e-91cf-418a6255cf75-installation-pull-secrets\") pod \"image-registry-6955c4f578-svncz\" (UID: \"f6bf0fd3-2e15-4d5e-91cf-418a6255cf75\") " pod="openshift-image-registry/image-registry-6955c4f578-svncz" Jul 08 19:28:43.534285 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:43.534207 2566 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ec79a684-b3d4-4942-aed4-41d82124967d-registry-certificates\") on node \"ip-10-0-142-195.ec2.internal\" DevicePath \"\"" Jul 08 19:28:43.534285 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:43.534224 2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ec79a684-b3d4-4942-aed4-41d82124967d-trusted-ca\") on node \"ip-10-0-142-195.ec2.internal\" DevicePath \"\"" Jul 08 19:28:43.534285 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:43.534234 2566 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ec79a684-b3d4-4942-aed4-41d82124967d-ca-trust-extracted\") on node \"ip-10-0-142-195.ec2.internal\" DevicePath \"\"" Jul 08 19:28:43.535630 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:43.535607 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec79a684-b3d4-4942-aed4-41d82124967d-kube-api-access-8n9kp" (OuterVolumeSpecName: "kube-api-access-8n9kp") pod "ec79a684-b3d4-4942-aed4-41d82124967d" (UID: "ec79a684-b3d4-4942-aed4-41d82124967d"). InnerVolumeSpecName "kube-api-access-8n9kp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:28:43.535724 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:43.535604 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec79a684-b3d4-4942-aed4-41d82124967d-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "ec79a684-b3d4-4942-aed4-41d82124967d" (UID: "ec79a684-b3d4-4942-aed4-41d82124967d"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:28:43.535784 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:43.535771 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec79a684-b3d4-4942-aed4-41d82124967d-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "ec79a684-b3d4-4942-aed4-41d82124967d" (UID: "ec79a684-b3d4-4942-aed4-41d82124967d"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:28:43.535833 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:43.535814 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec79a684-b3d4-4942-aed4-41d82124967d-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "ec79a684-b3d4-4942-aed4-41d82124967d" (UID: "ec79a684-b3d4-4942-aed4-41d82124967d"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:28:43.549524 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:43.549482 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:28:43.635451 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:43.635412 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f6bf0fd3-2e15-4d5e-91cf-418a6255cf75-installation-pull-secrets\") pod \"image-registry-6955c4f578-svncz\" (UID: \"f6bf0fd3-2e15-4d5e-91cf-418a6255cf75\") " pod="openshift-image-registry/image-registry-6955c4f578-svncz" Jul 08 19:28:43.635640 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:43.635501 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwl7r\" (UniqueName: \"kubernetes.io/projected/f6bf0fd3-2e15-4d5e-91cf-418a6255cf75-kube-api-access-cwl7r\") pod \"image-registry-6955c4f578-svncz\" (UID: \"f6bf0fd3-2e15-4d5e-91cf-418a6255cf75\") " pod="openshift-image-registry/image-registry-6955c4f578-svncz" Jul 08 19:28:43.635640 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:43.635537 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f6bf0fd3-2e15-4d5e-91cf-418a6255cf75-bound-sa-token\") pod \"image-registry-6955c4f578-svncz\" (UID: \"f6bf0fd3-2e15-4d5e-91cf-418a6255cf75\") " pod="openshift-image-registry/image-registry-6955c4f578-svncz" Jul 08 19:28:43.635640 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:43.635569 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f6bf0fd3-2e15-4d5e-91cf-418a6255cf75-registry-certificates\") pod \"image-registry-6955c4f578-svncz\" (UID: \"f6bf0fd3-2e15-4d5e-91cf-418a6255cf75\") " pod="openshift-image-registry/image-registry-6955c4f578-svncz" Jul 08 19:28:43.635640 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:43.635589 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f6bf0fd3-2e15-4d5e-91cf-418a6255cf75-ca-trust-extracted\") pod \"image-registry-6955c4f578-svncz\" (UID: \"f6bf0fd3-2e15-4d5e-91cf-418a6255cf75\") " pod="openshift-image-registry/image-registry-6955c4f578-svncz" Jul 08 19:28:43.635831 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:43.635714 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f6bf0fd3-2e15-4d5e-91cf-418a6255cf75-registry-tls\") pod \"image-registry-6955c4f578-svncz\" (UID: \"f6bf0fd3-2e15-4d5e-91cf-418a6255cf75\") " pod="openshift-image-registry/image-registry-6955c4f578-svncz" Jul 08 19:28:43.635831 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:43.635761 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f6bf0fd3-2e15-4d5e-91cf-418a6255cf75-image-registry-private-configuration\") pod \"image-registry-6955c4f578-svncz\" (UID: \"f6bf0fd3-2e15-4d5e-91cf-418a6255cf75\") " pod="openshift-image-registry/image-registry-6955c4f578-svncz" Jul 08 19:28:43.635831 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:43.635793 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f6bf0fd3-2e15-4d5e-91cf-418a6255cf75-trusted-ca\") pod \"image-registry-6955c4f578-svncz\" (UID: \"f6bf0fd3-2e15-4d5e-91cf-418a6255cf75\") " pod="openshift-image-registry/image-registry-6955c4f578-svncz" Jul 08 19:28:43.636002 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:43.635881 2566 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ec79a684-b3d4-4942-aed4-41d82124967d-installation-pull-secrets\") on node \"ip-10-0-142-195.ec2.internal\" DevicePath \"\"" Jul 08 19:28:43.636002 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:43.635901 2566 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ec79a684-b3d4-4942-aed4-41d82124967d-image-registry-private-configuration\") on node \"ip-10-0-142-195.ec2.internal\" DevicePath \"\"" Jul 08 19:28:43.636002 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:43.635925 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8n9kp\" (UniqueName: \"kubernetes.io/projected/ec79a684-b3d4-4942-aed4-41d82124967d-kube-api-access-8n9kp\") on node \"ip-10-0-142-195.ec2.internal\" DevicePath \"\"" Jul 08 19:28:43.636002 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:43.635944 2566 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ec79a684-b3d4-4942-aed4-41d82124967d-bound-sa-token\") on node \"ip-10-0-142-195.ec2.internal\" DevicePath \"\"" Jul 08 19:28:43.636153 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:43.636002 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f6bf0fd3-2e15-4d5e-91cf-418a6255cf75-ca-trust-extracted\") pod \"image-registry-6955c4f578-svncz\" (UID: \"f6bf0fd3-2e15-4d5e-91cf-418a6255cf75\") " pod="openshift-image-registry/image-registry-6955c4f578-svncz" Jul 08 19:28:43.636506 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:43.636481 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f6bf0fd3-2e15-4d5e-91cf-418a6255cf75-registry-certificates\") pod \"image-registry-6955c4f578-svncz\" (UID: \"f6bf0fd3-2e15-4d5e-91cf-418a6255cf75\") " pod="openshift-image-registry/image-registry-6955c4f578-svncz" Jul 08 19:28:43.637051 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:43.637028 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f6bf0fd3-2e15-4d5e-91cf-418a6255cf75-trusted-ca\") pod \"image-registry-6955c4f578-svncz\" (UID: \"f6bf0fd3-2e15-4d5e-91cf-418a6255cf75\") " pod="openshift-image-registry/image-registry-6955c4f578-svncz" Jul 08 19:28:43.638066 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:43.638040 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f6bf0fd3-2e15-4d5e-91cf-418a6255cf75-installation-pull-secrets\") pod \"image-registry-6955c4f578-svncz\" (UID: \"f6bf0fd3-2e15-4d5e-91cf-418a6255cf75\") " pod="openshift-image-registry/image-registry-6955c4f578-svncz" Jul 08 19:28:43.638350 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:43.638329 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f6bf0fd3-2e15-4d5e-91cf-418a6255cf75-registry-tls\") pod \"image-registry-6955c4f578-svncz\" (UID: \"f6bf0fd3-2e15-4d5e-91cf-418a6255cf75\") " pod="openshift-image-registry/image-registry-6955c4f578-svncz" Jul 08 19:28:43.638416 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:43.638329 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f6bf0fd3-2e15-4d5e-91cf-418a6255cf75-image-registry-private-configuration\") pod \"image-registry-6955c4f578-svncz\" (UID: \"f6bf0fd3-2e15-4d5e-91cf-418a6255cf75\") " pod="openshift-image-registry/image-registry-6955c4f578-svncz" Jul 08 19:28:43.648282 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:43.648262 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f6bf0fd3-2e15-4d5e-91cf-418a6255cf75-bound-sa-token\") pod \"image-registry-6955c4f578-svncz\" (UID: \"f6bf0fd3-2e15-4d5e-91cf-418a6255cf75\") " pod="openshift-image-registry/image-registry-6955c4f578-svncz" Jul 08 19:28:43.648398 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:43.648291 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwl7r\" (UniqueName: \"kubernetes.io/projected/f6bf0fd3-2e15-4d5e-91cf-418a6255cf75-kube-api-access-cwl7r\") pod \"image-registry-6955c4f578-svncz\" (UID: \"f6bf0fd3-2e15-4d5e-91cf-418a6255cf75\") " pod="openshift-image-registry/image-registry-6955c4f578-svncz" Jul 08 19:28:43.651055 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:43.650991 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:28:43.757242 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:43.757212 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6955c4f578-svncz" Jul 08 19:28:43.888940 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:43.888907 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6955c4f578-svncz"] Jul 08 19:28:43.891812 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:28:43.891786 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6bf0fd3_2e15_4d5e_91cf_418a6255cf75.slice/crio-0aa996e3059822f679568b0d8f3770d1d9251c0225684fc81f265882251eaeb5 WatchSource:0}: Error finding container 0aa996e3059822f679568b0d8f3770d1d9251c0225684fc81f265882251eaeb5: Status 404 returned error can't find the container with id 0aa996e3059822f679568b0d8f3770d1d9251c0225684fc81f265882251eaeb5 Jul 08 19:28:44.460883 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:44.460832 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d9dd85c4b-s89jc" Jul 08 19:28:44.461278 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:44.460889 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6955c4f578-svncz" event={"ID":"f6bf0fd3-2e15-4d5e-91cf-418a6255cf75","Type":"ContainerStarted","Data":"7bbeb7b66465fc20a894f6c49cc62d6e9d42fd502d0323363738df10c21ca4d2"} Jul 08 19:28:44.461278 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:44.460935 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6955c4f578-svncz" event={"ID":"f6bf0fd3-2e15-4d5e-91cf-418a6255cf75","Type":"ContainerStarted","Data":"0aa996e3059822f679568b0d8f3770d1d9251c0225684fc81f265882251eaeb5"} Jul 08 19:28:44.461278 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:44.460998 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6955c4f578-svncz" Jul 08 19:28:44.495202 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:44.495156 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6955c4f578-svncz" podStartSLOduration=1.495141147 podStartE2EDuration="1.495141147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:28:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:28:44.495011505 +0000 UTC m=+290.381201555" watchObservedRunningTime="2026-07-08 19:28:44.495141147 +0000 UTC m=+290.381331194" Jul 08 19:28:44.532714 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:44.532687 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6d9dd85c4b-s89jc"] Jul 08 19:28:44.534810 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:44.534789 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6d9dd85c4b-s89jc"] Jul 08 19:28:44.621226 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:44.621189 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec79a684-b3d4-4942-aed4-41d82124967d" path="/var/lib/kubelet/pods/ec79a684-b3d4-4942-aed4-41d82124967d/volumes" Jul 08 19:28:44.645147 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:44.645122 2566 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ec79a684-b3d4-4942-aed4-41d82124967d-registry-tls\") on node \"ip-10-0-142-195.ec2.internal\" DevicePath \"\"" Jul 08 19:28:46.864431 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:46.864393 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cee815ab-2b57-4584-8b6c-a8748c849be9-service-ca-bundle\") pod \"router-default-5846dc5c45-fgr6x\" (UID: \"cee815ab-2b57-4584-8b6c-a8748c849be9\") " pod="openshift-ingress/router-default-5846dc5c45-fgr6x" Jul 08 19:28:46.864929 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:46.864446 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cee815ab-2b57-4584-8b6c-a8748c849be9-metrics-certs\") pod \"router-default-5846dc5c45-fgr6x\" (UID: \"cee815ab-2b57-4584-8b6c-a8748c849be9\") " pod="openshift-ingress/router-default-5846dc5c45-fgr6x" Jul 08 19:28:46.864929 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:46.864478 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6e934fe3-7407-4bd1-88cc-21be3143047c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-v4w6v\" (UID: \"6e934fe3-7407-4bd1-88cc-21be3143047c\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-v4w6v" Jul 08 19:28:46.865149 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:46.865126 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cee815ab-2b57-4584-8b6c-a8748c849be9-service-ca-bundle\") pod \"router-default-5846dc5c45-fgr6x\" (UID: \"cee815ab-2b57-4584-8b6c-a8748c849be9\") " pod="openshift-ingress/router-default-5846dc5c45-fgr6x" Jul 08 19:28:46.866789 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:46.866755 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cee815ab-2b57-4584-8b6c-a8748c849be9-metrics-certs\") pod \"router-default-5846dc5c45-fgr6x\" (UID: \"cee815ab-2b57-4584-8b6c-a8748c849be9\") " pod="openshift-ingress/router-default-5846dc5c45-fgr6x" Jul 08 19:28:46.866917 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:46.866814 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6e934fe3-7407-4bd1-88cc-21be3143047c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-v4w6v\" (UID: \"6e934fe3-7407-4bd1-88cc-21be3143047c\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-v4w6v" Jul 08 19:28:47.127762 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:47.127663 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-v4w6v" Jul 08 19:28:47.132902 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:47.132854 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5846dc5c45-fgr6x" Jul 08 19:28:47.263147 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:47.262987 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-v4w6v"] Jul 08 19:28:47.266897 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:28:47.266847 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e934fe3_7407_4bd1_88cc_21be3143047c.slice/crio-a043183a839cc9f5db35bd4f059f7a1caa215c2e7efd5ea2a2440a23477172fc WatchSource:0}: Error finding container a043183a839cc9f5db35bd4f059f7a1caa215c2e7efd5ea2a2440a23477172fc: Status 404 returned error can't find the container with id a043183a839cc9f5db35bd4f059f7a1caa215c2e7efd5ea2a2440a23477172fc Jul 08 19:28:47.281924 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:47.281898 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5846dc5c45-fgr6x"] Jul 08 19:28:47.285156 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:28:47.285127 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcee815ab_2b57_4584_8b6c_a8748c849be9.slice/crio-7a21f574e929574a624b8062e04d3aaf00014bfeb0d8330bcb848f2d2fd4a60b WatchSource:0}: Error finding container 7a21f574e929574a624b8062e04d3aaf00014bfeb0d8330bcb848f2d2fd4a60b: Status 404 returned error can't find the container with id 7a21f574e929574a624b8062e04d3aaf00014bfeb0d8330bcb848f2d2fd4a60b Jul 08 19:28:47.473202 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:47.473160 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5846dc5c45-fgr6x" event={"ID":"cee815ab-2b57-4584-8b6c-a8748c849be9","Type":"ContainerStarted","Data":"877669cea17e869789a7f236e307237814db6ea7fa206e66dbbc69a507bce382"} Jul 08 19:28:47.473391 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:47.473220 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5846dc5c45-fgr6x" event={"ID":"cee815ab-2b57-4584-8b6c-a8748c849be9","Type":"ContainerStarted","Data":"7a21f574e929574a624b8062e04d3aaf00014bfeb0d8330bcb848f2d2fd4a60b"} Jul 08 19:28:47.474312 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:47.474291 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-v4w6v" event={"ID":"6e934fe3-7407-4bd1-88cc-21be3143047c","Type":"ContainerStarted","Data":"a043183a839cc9f5db35bd4f059f7a1caa215c2e7efd5ea2a2440a23477172fc"} Jul 08 19:28:47.496398 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:47.496327 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-5846dc5c45-fgr6x" podStartSLOduration=33.496304908 podStartE2EDuration="33.496304908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:28:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:28:47.494896374 +0000 UTC m=+293.381086422" watchObservedRunningTime="2026-07-08 19:28:47.496304908 +0000 UTC m=+293.382494956" Jul 08 19:28:48.133071 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:48.133032 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5846dc5c45-fgr6x" Jul 08 19:28:48.135807 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:48.135787 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5846dc5c45-fgr6x" Jul 08 19:28:48.477618 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:48.477585 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5846dc5c45-fgr6x" Jul 08 19:28:48.478824 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:48.478806 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5846dc5c45-fgr6x" Jul 08 19:28:48.618552 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:28:48.618504 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-rfbjf" podUID="55818509-538d-4733-9e10-eaaf841ef071" Jul 08 19:28:49.482686 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:49.482647 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-v4w6v" event={"ID":"6e934fe3-7407-4bd1-88cc-21be3143047c","Type":"ContainerStarted","Data":"d2706dd7fd3219549717a479837ad7471855d083e198dfaf42235565a1c2a2ed"} Jul 08 19:28:49.503557 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:49.503504 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-v4w6v" podStartSLOduration=33.504621267 podStartE2EDuration="35.50348563s" totalImagesPullingTime="1.998864363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:28:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:28:49.501408351 +0000 UTC m=+295.387598400" watchObservedRunningTime="2026-07-08 19:28:49.50348563 +0000 UTC m=+295.389675678" Jul 08 19:28:49.507800 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:49.507761 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:28:49.608170 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:49.608086 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:28:49.707576 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:49.707544 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:28:50.495141 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:50.495109 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/baee8938-f6f6-4ed9-99f7-832c16ebbd4a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-db7nv\" (UID: \"baee8938-f6f6-4ed9-99f7-832c16ebbd4a\") " pod="openshift-insights/insights-runtime-extractor-db7nv" Jul 08 19:28:50.497361 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:50.497337 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/baee8938-f6f6-4ed9-99f7-832c16ebbd4a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-db7nv\" (UID: \"baee8938-f6f6-4ed9-99f7-832c16ebbd4a\") " pod="openshift-insights/insights-runtime-extractor-db7nv" Jul 08 19:28:50.708905 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:50.708833 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-db7nv" Jul 08 19:28:50.840089 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:50.840057 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-db7nv"] Jul 08 19:28:50.843225 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:28:50.843193 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaee8938_f6f6_4ed9_99f7_832c16ebbd4a.slice/crio-ce006acdde37884d04fccb83e36bd551f399e25efdfc94e1e0c8f04a9a6f0304 WatchSource:0}: Error finding container ce006acdde37884d04fccb83e36bd551f399e25efdfc94e1e0c8f04a9a6f0304: Status 404 returned error can't find the container with id ce006acdde37884d04fccb83e36bd551f399e25efdfc94e1e0c8f04a9a6f0304 Jul 08 19:28:51.144364 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:51.144271 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:28:51.455771 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:51.455737 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-npv6z" Jul 08 19:28:51.491030 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:51.490974 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-db7nv" event={"ID":"baee8938-f6f6-4ed9-99f7-832c16ebbd4a","Type":"ContainerStarted","Data":"ce006acdde37884d04fccb83e36bd551f399e25efdfc94e1e0c8f04a9a6f0304"} Jul 08 19:28:53.796407 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:53.796374 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:28:53.862390 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:53.862357 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:28:54.038603 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:54.038572 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:28:54.722489 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:54.722457 2566 kubelet.go:1648] "Image garbage collection succeeded" Jul 08 19:28:55.505438 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:55.505402 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-db7nv" event={"ID":"baee8938-f6f6-4ed9-99f7-832c16ebbd4a","Type":"ContainerStarted","Data":"119597efb4fbdd4fa868a8becbde641dd6468aae7a5017a61e1d3a4f90f0d72b"} Jul 08 19:28:55.505827 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:55.505459 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-db7nv" event={"ID":"baee8938-f6f6-4ed9-99f7-832c16ebbd4a","Type":"ContainerStarted","Data":"d64aaa7cd185e6bd1817944dd851b6620d2821c3f378e43abfe800d883f8a26d"} Jul 08 19:28:56.389038 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:56.389002 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:28:56.489710 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:56.489654 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:28:56.729388 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:56.729358 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:28:58.518425 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:58.518392 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-db7nv" event={"ID":"baee8938-f6f6-4ed9-99f7-832c16ebbd4a","Type":"ContainerStarted","Data":"dbca811be3660b5b9ce33f8cbcd0a5a124e3b2786b00ea70fbf66f6b0c7588af"} Jul 08 19:28:58.538946 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:58.538884 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-db7nv" podStartSLOduration=33.728042655 podStartE2EDuration="40.538849578s" totalImagesPullingTime="6.810806923s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 19:28:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:28:58.538632463 +0000 UTC m=+304.424822542" watchObservedRunningTime="2026-07-08 19:28:58.538849578 +0000 UTC m=+304.425039627" Jul 08 19:28:58.729999 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:58.729962 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:28:58.795071 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:58.794977 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:28:58.965288 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:58.965256 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:28:59.199504 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:59.199468 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:28:59.302712 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:59.302683 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:28:59.302980 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:59.302930 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-nz2hj"] Jul 08 19:28:59.306200 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:59.306180 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-nz2hj" Jul 08 19:28:59.310474 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:59.310448 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 08 19:28:59.310474 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:59.310469 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 08 19:28:59.310641 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:59.310498 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 08 19:28:59.310641 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:59.310505 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-k4t8v\"" Jul 08 19:28:59.310641 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:59.310502 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 08 19:28:59.320190 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:59.320169 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-lps28"] Jul 08 19:28:59.326592 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:59.326571 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-nz2hj"] Jul 08 19:28:59.326705 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:59.326668 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lps28" Jul 08 19:28:59.329128 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:59.329110 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 08 19:28:59.329128 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:59.329124 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-7zzsr\"" Jul 08 19:28:59.329364 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:59.329351 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 08 19:28:59.329427 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:59.329411 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 08 19:28:59.370775 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:59.370742 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/322a5d1a-2482-4930-a546-31774f7f0c61-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-nz2hj\" (UID: \"322a5d1a-2482-4930-a546-31774f7f0c61\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-nz2hj" Jul 08 19:28:59.370961 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:59.370784 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/322a5d1a-2482-4930-a546-31774f7f0c61-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-nz2hj\" (UID: \"322a5d1a-2482-4930-a546-31774f7f0c61\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-nz2hj" Jul 08 19:28:59.370961 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:59.370810 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf2gl\" (UniqueName: \"kubernetes.io/projected/322a5d1a-2482-4930-a546-31774f7f0c61-kube-api-access-nf2gl\") pod \"kube-state-metrics-7c9dbb8b7d-nz2hj\" (UID: \"322a5d1a-2482-4930-a546-31774f7f0c61\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-nz2hj" Jul 08 19:28:59.370961 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:59.370836 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/322a5d1a-2482-4930-a546-31774f7f0c61-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-nz2hj\" (UID: \"322a5d1a-2482-4930-a546-31774f7f0c61\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-nz2hj" Jul 08 19:28:59.370961 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:59.370925 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/322a5d1a-2482-4930-a546-31774f7f0c61-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-nz2hj\" (UID: \"322a5d1a-2482-4930-a546-31774f7f0c61\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-nz2hj" Jul 08 19:28:59.370961 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:59.370952 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/322a5d1a-2482-4930-a546-31774f7f0c61-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-nz2hj\" (UID: \"322a5d1a-2482-4930-a546-31774f7f0c61\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-nz2hj" Jul 08 19:28:59.380986 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:59.380956 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:28:59.472220 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:59.472132 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/322a5d1a-2482-4930-a546-31774f7f0c61-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-nz2hj\" (UID: \"322a5d1a-2482-4930-a546-31774f7f0c61\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-nz2hj" Jul 08 19:28:59.472220 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:59.472175 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/406246a8-ab16-4792-8582-83f15d00cafc-sys\") pod \"node-exporter-lps28\" (UID: \"406246a8-ab16-4792-8582-83f15d00cafc\") " pod="openshift-monitoring/node-exporter-lps28" Jul 08 19:28:59.472220 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:59.472207 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/406246a8-ab16-4792-8582-83f15d00cafc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lps28\" (UID: \"406246a8-ab16-4792-8582-83f15d00cafc\") " pod="openshift-monitoring/node-exporter-lps28" Jul 08 19:28:59.472486 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:59.472241 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/406246a8-ab16-4792-8582-83f15d00cafc-metrics-client-ca\") pod \"node-exporter-lps28\" (UID: \"406246a8-ab16-4792-8582-83f15d00cafc\") " pod="openshift-monitoring/node-exporter-lps28" Jul 08 19:28:59.472486 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:59.472271 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/322a5d1a-2482-4930-a546-31774f7f0c61-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-nz2hj\" (UID: \"322a5d1a-2482-4930-a546-31774f7f0c61\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-nz2hj" Jul 08 19:28:59.472486 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:59.472336 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf2gl\" (UniqueName: \"kubernetes.io/projected/322a5d1a-2482-4930-a546-31774f7f0c61-kube-api-access-nf2gl\") pod \"kube-state-metrics-7c9dbb8b7d-nz2hj\" (UID: \"322a5d1a-2482-4930-a546-31774f7f0c61\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-nz2hj" Jul 08 19:28:59.472486 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:59.472378 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/406246a8-ab16-4792-8582-83f15d00cafc-node-exporter-accelerators-collector-config\") pod \"node-exporter-lps28\" (UID: \"406246a8-ab16-4792-8582-83f15d00cafc\") " pod="openshift-monitoring/node-exporter-lps28" Jul 08 19:28:59.472486 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:59.472474 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/322a5d1a-2482-4930-a546-31774f7f0c61-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-nz2hj\" (UID: \"322a5d1a-2482-4930-a546-31774f7f0c61\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-nz2hj" Jul 08 19:28:59.472716 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:59.472508 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/406246a8-ab16-4792-8582-83f15d00cafc-node-exporter-tls\") pod \"node-exporter-lps28\" (UID: \"406246a8-ab16-4792-8582-83f15d00cafc\") " pod="openshift-monitoring/node-exporter-lps28" Jul 08 19:28:59.472716 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:59.472535 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/322a5d1a-2482-4930-a546-31774f7f0c61-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-nz2hj\" (UID: \"322a5d1a-2482-4930-a546-31774f7f0c61\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-nz2hj" Jul 08 19:28:59.472716 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:59.472551 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/406246a8-ab16-4792-8582-83f15d00cafc-node-exporter-textfile\") pod \"node-exporter-lps28\" (UID: \"406246a8-ab16-4792-8582-83f15d00cafc\") " pod="openshift-monitoring/node-exporter-lps28" Jul 08 19:28:59.472716 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:28:59.472619 2566 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 08 19:28:59.472716 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:59.472642 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl82d\" (UniqueName: \"kubernetes.io/projected/406246a8-ab16-4792-8582-83f15d00cafc-kube-api-access-xl82d\") pod \"node-exporter-lps28\" (UID: \"406246a8-ab16-4792-8582-83f15d00cafc\") " pod="openshift-monitoring/node-exporter-lps28" Jul 08 19:28:59.472716 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:28:59.472674 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/322a5d1a-2482-4930-a546-31774f7f0c61-kube-state-metrics-tls podName:322a5d1a-2482-4930-a546-31774f7f0c61 nodeName:}" failed. No retries permitted until 2026-07-08 19:28:59.972656064 +0000 UTC m=+305.858846092 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/322a5d1a-2482-4930-a546-31774f7f0c61-kube-state-metrics-tls") pod "kube-state-metrics-7c9dbb8b7d-nz2hj" (UID: "322a5d1a-2482-4930-a546-31774f7f0c61") : secret "kube-state-metrics-tls" not found Jul 08 19:28:59.472716 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:59.472687 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/406246a8-ab16-4792-8582-83f15d00cafc-root\") pod \"node-exporter-lps28\" (UID: \"406246a8-ab16-4792-8582-83f15d00cafc\") " pod="openshift-monitoring/node-exporter-lps28" Jul 08 19:28:59.473010 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:59.472734 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/322a5d1a-2482-4930-a546-31774f7f0c61-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-nz2hj\" (UID: \"322a5d1a-2482-4930-a546-31774f7f0c61\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-nz2hj" Jul 08 19:28:59.473010 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:59.472775 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/322a5d1a-2482-4930-a546-31774f7f0c61-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-nz2hj\" (UID: \"322a5d1a-2482-4930-a546-31774f7f0c61\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-nz2hj" Jul 08 19:28:59.473010 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:59.472811 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/406246a8-ab16-4792-8582-83f15d00cafc-node-exporter-wtmp\") pod \"node-exporter-lps28\" (UID: \"406246a8-ab16-4792-8582-83f15d00cafc\") " pod="openshift-monitoring/node-exporter-lps28" Jul 08 19:28:59.473120 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:59.473024 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/322a5d1a-2482-4930-a546-31774f7f0c61-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-nz2hj\" (UID: \"322a5d1a-2482-4930-a546-31774f7f0c61\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-nz2hj" Jul 08 19:28:59.473349 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:59.473325 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/322a5d1a-2482-4930-a546-31774f7f0c61-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-nz2hj\" (UID: \"322a5d1a-2482-4930-a546-31774f7f0c61\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-nz2hj" Jul 08 19:28:59.475209 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:59.475187 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/322a5d1a-2482-4930-a546-31774f7f0c61-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-nz2hj\" (UID: \"322a5d1a-2482-4930-a546-31774f7f0c61\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-nz2hj" Jul 08 19:28:59.483772 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:59.483753 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf2gl\" (UniqueName: \"kubernetes.io/projected/322a5d1a-2482-4930-a546-31774f7f0c61-kube-api-access-nf2gl\") pod \"kube-state-metrics-7c9dbb8b7d-nz2hj\" (UID: \"322a5d1a-2482-4930-a546-31774f7f0c61\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-nz2hj" Jul 08 19:28:59.503383 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:59.503353 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:28:59.573574 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:59.573540 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl82d\" (UniqueName: \"kubernetes.io/projected/406246a8-ab16-4792-8582-83f15d00cafc-kube-api-access-xl82d\") pod \"node-exporter-lps28\" (UID: \"406246a8-ab16-4792-8582-83f15d00cafc\") " pod="openshift-monitoring/node-exporter-lps28" Jul 08 19:28:59.573574 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:59.573576 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/406246a8-ab16-4792-8582-83f15d00cafc-root\") pod \"node-exporter-lps28\" (UID: \"406246a8-ab16-4792-8582-83f15d00cafc\") " pod="openshift-monitoring/node-exporter-lps28" Jul 08 19:28:59.574136 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:59.573598 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/406246a8-ab16-4792-8582-83f15d00cafc-node-exporter-wtmp\") pod \"node-exporter-lps28\" (UID: \"406246a8-ab16-4792-8582-83f15d00cafc\") " pod="openshift-monitoring/node-exporter-lps28" Jul 08 19:28:59.574136 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:59.573628 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/406246a8-ab16-4792-8582-83f15d00cafc-sys\") pod \"node-exporter-lps28\" (UID: \"406246a8-ab16-4792-8582-83f15d00cafc\") " pod="openshift-monitoring/node-exporter-lps28" Jul 08 19:28:59.574136 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:59.573654 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/406246a8-ab16-4792-8582-83f15d00cafc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lps28\" (UID: \"406246a8-ab16-4792-8582-83f15d00cafc\") " pod="openshift-monitoring/node-exporter-lps28" Jul 08 19:28:59.574136 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:59.573682 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/406246a8-ab16-4792-8582-83f15d00cafc-metrics-client-ca\") pod \"node-exporter-lps28\" (UID: \"406246a8-ab16-4792-8582-83f15d00cafc\") " pod="openshift-monitoring/node-exporter-lps28" Jul 08 19:28:59.574136 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:59.573704 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/406246a8-ab16-4792-8582-83f15d00cafc-sys\") pod \"node-exporter-lps28\" (UID: \"406246a8-ab16-4792-8582-83f15d00cafc\") " pod="openshift-monitoring/node-exporter-lps28" Jul 08 19:28:59.574136 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:59.573680 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/406246a8-ab16-4792-8582-83f15d00cafc-root\") pod \"node-exporter-lps28\" (UID: \"406246a8-ab16-4792-8582-83f15d00cafc\") " pod="openshift-monitoring/node-exporter-lps28" Jul 08 19:28:59.574136 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:59.573755 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/406246a8-ab16-4792-8582-83f15d00cafc-node-exporter-wtmp\") pod \"node-exporter-lps28\" (UID: \"406246a8-ab16-4792-8582-83f15d00cafc\") " pod="openshift-monitoring/node-exporter-lps28" Jul 08 19:28:59.574136 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:59.573742 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/406246a8-ab16-4792-8582-83f15d00cafc-node-exporter-accelerators-collector-config\") pod \"node-exporter-lps28\" (UID: \"406246a8-ab16-4792-8582-83f15d00cafc\") " pod="openshift-monitoring/node-exporter-lps28" Jul 08 19:28:59.574136 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:59.573827 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/406246a8-ab16-4792-8582-83f15d00cafc-node-exporter-tls\") pod \"node-exporter-lps28\" (UID: \"406246a8-ab16-4792-8582-83f15d00cafc\") " pod="openshift-monitoring/node-exporter-lps28" Jul 08 19:28:59.574136 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:59.573909 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/406246a8-ab16-4792-8582-83f15d00cafc-node-exporter-textfile\") pod \"node-exporter-lps28\" (UID: \"406246a8-ab16-4792-8582-83f15d00cafc\") " pod="openshift-monitoring/node-exporter-lps28" Jul 08 19:28:59.574636 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:59.574219 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/406246a8-ab16-4792-8582-83f15d00cafc-node-exporter-textfile\") pod \"node-exporter-lps28\" (UID: \"406246a8-ab16-4792-8582-83f15d00cafc\") " pod="openshift-monitoring/node-exporter-lps28" Jul 08 19:28:59.574636 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:59.574361 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/406246a8-ab16-4792-8582-83f15d00cafc-metrics-client-ca\") pod \"node-exporter-lps28\" (UID: \"406246a8-ab16-4792-8582-83f15d00cafc\") " pod="openshift-monitoring/node-exporter-lps28" Jul 08 19:28:59.574636 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:59.574518 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/406246a8-ab16-4792-8582-83f15d00cafc-node-exporter-accelerators-collector-config\") pod \"node-exporter-lps28\" (UID: \"406246a8-ab16-4792-8582-83f15d00cafc\") " pod="openshift-monitoring/node-exporter-lps28" Jul 08 19:28:59.576162 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:59.576135 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/406246a8-ab16-4792-8582-83f15d00cafc-node-exporter-tls\") pod \"node-exporter-lps28\" (UID: \"406246a8-ab16-4792-8582-83f15d00cafc\") " pod="openshift-monitoring/node-exporter-lps28" Jul 08 19:28:59.576281 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:59.576219 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/406246a8-ab16-4792-8582-83f15d00cafc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lps28\" (UID: \"406246a8-ab16-4792-8582-83f15d00cafc\") " pod="openshift-monitoring/node-exporter-lps28" Jul 08 19:28:59.583677 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:59.583656 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl82d\" (UniqueName: \"kubernetes.io/projected/406246a8-ab16-4792-8582-83f15d00cafc-kube-api-access-xl82d\") pod \"node-exporter-lps28\" (UID: \"406246a8-ab16-4792-8582-83f15d00cafc\") " pod="openshift-monitoring/node-exporter-lps28" Jul 08 19:28:59.604257 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:59.604224 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:28:59.635680 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:59.635631 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lps28" Jul 08 19:28:59.647314 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:28:59.647287 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod406246a8_ab16_4792_8582_83f15d00cafc.slice/crio-7f3bce6504fd97190a97d010d487a07d5b85ea63d5b81341328ae57fefcf0642 WatchSource:0}: Error finding container 7f3bce6504fd97190a97d010d487a07d5b85ea63d5b81341328ae57fefcf0642: Status 404 returned error can't find the container with id 7f3bce6504fd97190a97d010d487a07d5b85ea63d5b81341328ae57fefcf0642 Jul 08 19:28:59.649052 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:59.649037 2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 19:28:59.702956 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:59.702920 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:28:59.855818 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:59.855745 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:28:59.977269 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:59.977231 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/322a5d1a-2482-4930-a546-31774f7f0c61-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-nz2hj\" (UID: \"322a5d1a-2482-4930-a546-31774f7f0c61\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-nz2hj" Jul 08 19:28:59.979612 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:28:59.979591 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/322a5d1a-2482-4930-a546-31774f7f0c61-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-nz2hj\" (UID: \"322a5d1a-2482-4930-a546-31774f7f0c61\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-nz2hj" Jul 08 19:29:00.215018 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:00.214986 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-nz2hj" Jul 08 19:29:00.245563 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:00.245519 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:29:00.352086 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:00.352034 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:29:00.361555 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:00.361522 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 08 19:29:00.366771 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:00.366745 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:29:00.369816 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:00.369791 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 08 19:29:00.369953 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:00.369791 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 08 19:29:00.370117 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:00.370103 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 08 19:29:00.370182 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:00.370118 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 08 19:29:00.370229 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:00.370124 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 08 19:29:00.371015 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:00.370900 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-99g6l\"" Jul 08 19:29:00.371015 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:00.370910 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 08 19:29:00.371913 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:00.371896 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 08 19:29:00.373169 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:00.373153 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 08 19:29:00.374171 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:00.374152 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 08 19:29:00.380930 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:00.380902 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-nz2hj"] Jul 08 19:29:00.382378 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:00.382358 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 08 19:29:00.385500 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:29:00.385472 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod322a5d1a_2482_4930_a546_31774f7f0c61.slice/crio-5b8df08ae78343c25bac4ada5d6af47e06d1d21165c56aa77f7f3dadfe505511 WatchSource:0}: Error finding container 5b8df08ae78343c25bac4ada5d6af47e06d1d21165c56aa77f7f3dadfe505511: Status 404 returned error can't find the container with id 5b8df08ae78343c25bac4ada5d6af47e06d1d21165c56aa77f7f3dadfe505511 Jul 08 19:29:00.483324 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:00.483230 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/12079511-682f-45ae-b4c3-af244b463829-tls-assets\") pod \"alertmanager-main-0\" (UID: \"12079511-682f-45ae-b4c3-af244b463829\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:29:00.483324 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:00.483284 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/12079511-682f-45ae-b4c3-af244b463829-web-config\") pod \"alertmanager-main-0\" (UID: \"12079511-682f-45ae-b4c3-af244b463829\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:29:00.483324 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:00.483309 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/12079511-682f-45ae-b4c3-af244b463829-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"12079511-682f-45ae-b4c3-af244b463829\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:29:00.483581 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:00.483379 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12079511-682f-45ae-b4c3-af244b463829-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"12079511-682f-45ae-b4c3-af244b463829\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:29:00.483581 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:00.483418 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/12079511-682f-45ae-b4c3-af244b463829-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"12079511-682f-45ae-b4c3-af244b463829\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:29:00.483581 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:00.483445 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/12079511-682f-45ae-b4c3-af244b463829-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"12079511-682f-45ae-b4c3-af244b463829\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:29:00.483581 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:00.483482 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x2qp\" (UniqueName: \"kubernetes.io/projected/12079511-682f-45ae-b4c3-af244b463829-kube-api-access-2x2qp\") pod \"alertmanager-main-0\" (UID: \"12079511-682f-45ae-b4c3-af244b463829\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:29:00.483581 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:00.483526 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/12079511-682f-45ae-b4c3-af244b463829-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"12079511-682f-45ae-b4c3-af244b463829\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:29:00.483823 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:00.483590 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/12079511-682f-45ae-b4c3-af244b463829-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"12079511-682f-45ae-b4c3-af244b463829\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:29:00.483823 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:00.483621 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/12079511-682f-45ae-b4c3-af244b463829-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"12079511-682f-45ae-b4c3-af244b463829\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:29:00.483823 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:00.483664 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/12079511-682f-45ae-b4c3-af244b463829-config-out\") pod \"alertmanager-main-0\" (UID: \"12079511-682f-45ae-b4c3-af244b463829\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:29:00.483823 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:00.483694 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/12079511-682f-45ae-b4c3-af244b463829-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"12079511-682f-45ae-b4c3-af244b463829\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:29:00.483823 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:00.483726 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/12079511-682f-45ae-b4c3-af244b463829-config-volume\") pod \"alertmanager-main-0\" (UID: \"12079511-682f-45ae-b4c3-af244b463829\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:29:00.527179 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:00.527137 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lps28" event={"ID":"406246a8-ab16-4792-8582-83f15d00cafc","Type":"ContainerStarted","Data":"7f3bce6504fd97190a97d010d487a07d5b85ea63d5b81341328ae57fefcf0642"} Jul 08 19:29:00.528472 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:00.528437 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-nz2hj" event={"ID":"322a5d1a-2482-4930-a546-31774f7f0c61","Type":"ContainerStarted","Data":"5b8df08ae78343c25bac4ada5d6af47e06d1d21165c56aa77f7f3dadfe505511"} Jul 08 19:29:00.585101 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:00.585065 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/12079511-682f-45ae-b4c3-af244b463829-tls-assets\") pod \"alertmanager-main-0\" (UID: \"12079511-682f-45ae-b4c3-af244b463829\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:29:00.585101 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:00.585107 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/12079511-682f-45ae-b4c3-af244b463829-web-config\") pod \"alertmanager-main-0\" (UID: \"12079511-682f-45ae-b4c3-af244b463829\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:29:00.585655 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:00.585144 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/12079511-682f-45ae-b4c3-af244b463829-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"12079511-682f-45ae-b4c3-af244b463829\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:29:00.585655 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:00.585191 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12079511-682f-45ae-b4c3-af244b463829-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"12079511-682f-45ae-b4c3-af244b463829\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:29:00.585655 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:00.585225 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/12079511-682f-45ae-b4c3-af244b463829-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"12079511-682f-45ae-b4c3-af244b463829\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:29:00.585655 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:00.585256 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/12079511-682f-45ae-b4c3-af244b463829-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"12079511-682f-45ae-b4c3-af244b463829\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:29:00.585655 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:00.585279 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x2qp\" (UniqueName: \"kubernetes.io/projected/12079511-682f-45ae-b4c3-af244b463829-kube-api-access-2x2qp\") pod \"alertmanager-main-0\" (UID: \"12079511-682f-45ae-b4c3-af244b463829\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:29:00.585655 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:00.585309 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/12079511-682f-45ae-b4c3-af244b463829-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"12079511-682f-45ae-b4c3-af244b463829\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:29:00.585655 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:00.585357 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/12079511-682f-45ae-b4c3-af244b463829-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"12079511-682f-45ae-b4c3-af244b463829\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:29:00.585655 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:00.585383 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/12079511-682f-45ae-b4c3-af244b463829-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"12079511-682f-45ae-b4c3-af244b463829\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:29:00.585655 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:00.585426 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/12079511-682f-45ae-b4c3-af244b463829-config-out\") pod \"alertmanager-main-0\" (UID: \"12079511-682f-45ae-b4c3-af244b463829\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:29:00.585655 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:00.585455 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/12079511-682f-45ae-b4c3-af244b463829-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"12079511-682f-45ae-b4c3-af244b463829\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:29:00.585655 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:00.585484 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/12079511-682f-45ae-b4c3-af244b463829-config-volume\") pod \"alertmanager-main-0\" (UID: \"12079511-682f-45ae-b4c3-af244b463829\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:29:00.586458 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:00.586313 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/12079511-682f-45ae-b4c3-af244b463829-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"12079511-682f-45ae-b4c3-af244b463829\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:29:00.586734 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:00.586708 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/12079511-682f-45ae-b4c3-af244b463829-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"12079511-682f-45ae-b4c3-af244b463829\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:29:00.587618 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:00.587586 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12079511-682f-45ae-b4c3-af244b463829-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"12079511-682f-45ae-b4c3-af244b463829\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:29:00.588855 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:00.588826 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/12079511-682f-45ae-b4c3-af244b463829-tls-assets\") pod \"alertmanager-main-0\" (UID: \"12079511-682f-45ae-b4c3-af244b463829\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:29:00.588855 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:00.588846 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/12079511-682f-45ae-b4c3-af244b463829-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"12079511-682f-45ae-b4c3-af244b463829\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:29:00.589292 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:00.589265 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/12079511-682f-45ae-b4c3-af244b463829-config-out\") pod \"alertmanager-main-0\" (UID: \"12079511-682f-45ae-b4c3-af244b463829\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:29:00.589550 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:00.589527 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/12079511-682f-45ae-b4c3-af244b463829-config-volume\") pod \"alertmanager-main-0\" (UID: \"12079511-682f-45ae-b4c3-af244b463829\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:29:00.589638 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:00.589573 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/12079511-682f-45ae-b4c3-af244b463829-web-config\") pod \"alertmanager-main-0\" (UID: \"12079511-682f-45ae-b4c3-af244b463829\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:29:00.590484 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:00.590252 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/12079511-682f-45ae-b4c3-af244b463829-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"12079511-682f-45ae-b4c3-af244b463829\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:29:00.590484 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:00.590337 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/12079511-682f-45ae-b4c3-af244b463829-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"12079511-682f-45ae-b4c3-af244b463829\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:29:00.590660 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:00.590638 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/12079511-682f-45ae-b4c3-af244b463829-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"12079511-682f-45ae-b4c3-af244b463829\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:29:00.591883 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:00.591836 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/12079511-682f-45ae-b4c3-af244b463829-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"12079511-682f-45ae-b4c3-af244b463829\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:29:00.592691 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:00.592668 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:29:00.593941 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:00.593920 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x2qp\" (UniqueName: \"kubernetes.io/projected/12079511-682f-45ae-b4c3-af244b463829-kube-api-access-2x2qp\") pod \"alertmanager-main-0\" (UID: \"12079511-682f-45ae-b4c3-af244b463829\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:29:00.679633 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:00.679602 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:29:00.825793 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:00.825674 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 08 19:29:00.830354 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:29:00.830312 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12079511_682f_45ae_b4c3_af244b463829.slice/crio-a733e70cf9d1056648f7207ee64b7bda7bfed64332f539ab4b88b4828cf5e16e WatchSource:0}: Error finding container a733e70cf9d1056648f7207ee64b7bda7bfed64332f539ab4b88b4828cf5e16e: Status 404 returned error can't find the container with id a733e70cf9d1056648f7207ee64b7bda7bfed64332f539ab4b88b4828cf5e16e Jul 08 19:29:01.329276 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:01.329243 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5c4467cbb7-wlnbt"] Jul 08 19:29:01.333143 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:01.333115 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5c4467cbb7-wlnbt" Jul 08 19:29:01.335650 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:01.335629 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-trgpq\"" Jul 08 19:29:01.336544 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:01.336312 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 08 19:29:01.336544 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:01.336411 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 08 19:29:01.336722 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:01.336601 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 08 19:29:01.336897 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:01.336878 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 08 19:29:01.337008 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:01.336906 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-1arpeuocigrfj\"" Jul 08 19:29:01.337008 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:01.336921 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 08 19:29:01.346601 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:01.346556 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5c4467cbb7-wlnbt"] Jul 08 19:29:01.394237 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:01.394204 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a7df7b87-f170-426a-89b0-9cf26efdeb29-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5c4467cbb7-wlnbt\" (UID: \"a7df7b87-f170-426a-89b0-9cf26efdeb29\") " pod="openshift-monitoring/thanos-querier-5c4467cbb7-wlnbt" Jul 08 19:29:01.394396 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:01.394262 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a7df7b87-f170-426a-89b0-9cf26efdeb29-secret-thanos-querier-tls\") pod \"thanos-querier-5c4467cbb7-wlnbt\" (UID: \"a7df7b87-f170-426a-89b0-9cf26efdeb29\") " pod="openshift-monitoring/thanos-querier-5c4467cbb7-wlnbt" Jul 08 19:29:01.394396 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:01.394310 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a7df7b87-f170-426a-89b0-9cf26efdeb29-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5c4467cbb7-wlnbt\" (UID: \"a7df7b87-f170-426a-89b0-9cf26efdeb29\") " pod="openshift-monitoring/thanos-querier-5c4467cbb7-wlnbt" Jul 08 19:29:01.394396 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:01.394335 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a7df7b87-f170-426a-89b0-9cf26efdeb29-metrics-client-ca\") pod \"thanos-querier-5c4467cbb7-wlnbt\" (UID: \"a7df7b87-f170-426a-89b0-9cf26efdeb29\") " pod="openshift-monitoring/thanos-querier-5c4467cbb7-wlnbt" Jul 08 19:29:01.394396 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:01.394358 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cvbb\" (UniqueName: \"kubernetes.io/projected/a7df7b87-f170-426a-89b0-9cf26efdeb29-kube-api-access-8cvbb\") pod \"thanos-querier-5c4467cbb7-wlnbt\" (UID: \"a7df7b87-f170-426a-89b0-9cf26efdeb29\") " pod="openshift-monitoring/thanos-querier-5c4467cbb7-wlnbt" Jul 08 19:29:01.394586 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:01.394428 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a7df7b87-f170-426a-89b0-9cf26efdeb29-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5c4467cbb7-wlnbt\" (UID: \"a7df7b87-f170-426a-89b0-9cf26efdeb29\") " pod="openshift-monitoring/thanos-querier-5c4467cbb7-wlnbt" Jul 08 19:29:01.394586 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:01.394502 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a7df7b87-f170-426a-89b0-9cf26efdeb29-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5c4467cbb7-wlnbt\" (UID: \"a7df7b87-f170-426a-89b0-9cf26efdeb29\") " pod="openshift-monitoring/thanos-querier-5c4467cbb7-wlnbt" Jul 08 19:29:01.394586 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:01.394542 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a7df7b87-f170-426a-89b0-9cf26efdeb29-secret-grpc-tls\") pod \"thanos-querier-5c4467cbb7-wlnbt\" (UID: \"a7df7b87-f170-426a-89b0-9cf26efdeb29\") " pod="openshift-monitoring/thanos-querier-5c4467cbb7-wlnbt" Jul 08 19:29:01.435209 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:01.435168 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:29:01.495729 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:01.495690 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a7df7b87-f170-426a-89b0-9cf26efdeb29-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5c4467cbb7-wlnbt\" (UID: \"a7df7b87-f170-426a-89b0-9cf26efdeb29\") " pod="openshift-monitoring/thanos-querier-5c4467cbb7-wlnbt" Jul 08 19:29:01.495925 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:01.495740 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a7df7b87-f170-426a-89b0-9cf26efdeb29-metrics-client-ca\") pod \"thanos-querier-5c4467cbb7-wlnbt\" (UID: \"a7df7b87-f170-426a-89b0-9cf26efdeb29\") " pod="openshift-monitoring/thanos-querier-5c4467cbb7-wlnbt" Jul 08 19:29:01.495925 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:01.495759 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cvbb\" (UniqueName: \"kubernetes.io/projected/a7df7b87-f170-426a-89b0-9cf26efdeb29-kube-api-access-8cvbb\") pod \"thanos-querier-5c4467cbb7-wlnbt\" (UID: \"a7df7b87-f170-426a-89b0-9cf26efdeb29\") " pod="openshift-monitoring/thanos-querier-5c4467cbb7-wlnbt" Jul 08 19:29:01.495925 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:01.495790 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a7df7b87-f170-426a-89b0-9cf26efdeb29-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5c4467cbb7-wlnbt\" (UID: \"a7df7b87-f170-426a-89b0-9cf26efdeb29\") " pod="openshift-monitoring/thanos-querier-5c4467cbb7-wlnbt" Jul 08 19:29:01.495925 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:01.495826 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a7df7b87-f170-426a-89b0-9cf26efdeb29-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5c4467cbb7-wlnbt\" (UID: \"a7df7b87-f170-426a-89b0-9cf26efdeb29\") " pod="openshift-monitoring/thanos-querier-5c4467cbb7-wlnbt" Jul 08 19:29:01.495925 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:01.495852 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a7df7b87-f170-426a-89b0-9cf26efdeb29-secret-grpc-tls\") pod \"thanos-querier-5c4467cbb7-wlnbt\" (UID: \"a7df7b87-f170-426a-89b0-9cf26efdeb29\") " pod="openshift-monitoring/thanos-querier-5c4467cbb7-wlnbt" Jul 08 19:29:01.495925 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:01.495924 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a7df7b87-f170-426a-89b0-9cf26efdeb29-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5c4467cbb7-wlnbt\" (UID: \"a7df7b87-f170-426a-89b0-9cf26efdeb29\") " pod="openshift-monitoring/thanos-querier-5c4467cbb7-wlnbt" Jul 08 19:29:01.496230 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:01.495961 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a7df7b87-f170-426a-89b0-9cf26efdeb29-secret-thanos-querier-tls\") pod \"thanos-querier-5c4467cbb7-wlnbt\" (UID: \"a7df7b87-f170-426a-89b0-9cf26efdeb29\") " pod="openshift-monitoring/thanos-querier-5c4467cbb7-wlnbt" Jul 08 19:29:01.496611 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:01.496555 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a7df7b87-f170-426a-89b0-9cf26efdeb29-metrics-client-ca\") pod \"thanos-querier-5c4467cbb7-wlnbt\" (UID: \"a7df7b87-f170-426a-89b0-9cf26efdeb29\") " pod="openshift-monitoring/thanos-querier-5c4467cbb7-wlnbt" Jul 08 19:29:01.498381 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:01.498357 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a7df7b87-f170-426a-89b0-9cf26efdeb29-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5c4467cbb7-wlnbt\" (UID: \"a7df7b87-f170-426a-89b0-9cf26efdeb29\") " pod="openshift-monitoring/thanos-querier-5c4467cbb7-wlnbt" Jul 08 19:29:01.498639 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:01.498614 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a7df7b87-f170-426a-89b0-9cf26efdeb29-secret-grpc-tls\") pod \"thanos-querier-5c4467cbb7-wlnbt\" (UID: \"a7df7b87-f170-426a-89b0-9cf26efdeb29\") " pod="openshift-monitoring/thanos-querier-5c4467cbb7-wlnbt" Jul 08 19:29:01.498740 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:01.498715 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a7df7b87-f170-426a-89b0-9cf26efdeb29-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5c4467cbb7-wlnbt\" (UID: \"a7df7b87-f170-426a-89b0-9cf26efdeb29\") " pod="openshift-monitoring/thanos-querier-5c4467cbb7-wlnbt" Jul 08 19:29:01.498840 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:01.498822 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a7df7b87-f170-426a-89b0-9cf26efdeb29-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5c4467cbb7-wlnbt\" (UID: \"a7df7b87-f170-426a-89b0-9cf26efdeb29\") " pod="openshift-monitoring/thanos-querier-5c4467cbb7-wlnbt" Jul 08 19:29:01.498916 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:01.498836 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a7df7b87-f170-426a-89b0-9cf26efdeb29-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5c4467cbb7-wlnbt\" (UID: \"a7df7b87-f170-426a-89b0-9cf26efdeb29\") " pod="openshift-monitoring/thanos-querier-5c4467cbb7-wlnbt" Jul 08 19:29:01.498958 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:01.498926 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a7df7b87-f170-426a-89b0-9cf26efdeb29-secret-thanos-querier-tls\") pod \"thanos-querier-5c4467cbb7-wlnbt\" (UID: \"a7df7b87-f170-426a-89b0-9cf26efdeb29\") " pod="openshift-monitoring/thanos-querier-5c4467cbb7-wlnbt" Jul 08 19:29:01.521870 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:01.521825 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cvbb\" (UniqueName: \"kubernetes.io/projected/a7df7b87-f170-426a-89b0-9cf26efdeb29-kube-api-access-8cvbb\") pod \"thanos-querier-5c4467cbb7-wlnbt\" (UID: \"a7df7b87-f170-426a-89b0-9cf26efdeb29\") " pod="openshift-monitoring/thanos-querier-5c4467cbb7-wlnbt" Jul 08 19:29:01.533179 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:01.533150 2566 generic.go:356] "Generic (PLEG): container finished" podID="406246a8-ab16-4792-8582-83f15d00cafc" containerID="94ebf3baa235cdfa53e029c806186028023d9842fc6a0451e9f41d1f7417c482" exitCode=0 Jul 08 19:29:01.533295 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:01.533225 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lps28" event={"ID":"406246a8-ab16-4792-8582-83f15d00cafc","Type":"ContainerDied","Data":"94ebf3baa235cdfa53e029c806186028023d9842fc6a0451e9f41d1f7417c482"} Jul 08 19:29:01.534480 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:01.534456 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"12079511-682f-45ae-b4c3-af244b463829","Type":"ContainerStarted","Data":"a733e70cf9d1056648f7207ee64b7bda7bfed64332f539ab4b88b4828cf5e16e"} Jul 08 19:29:01.556444 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:01.556411 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:29:01.643732 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:01.643707 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5c4467cbb7-wlnbt" Jul 08 19:29:01.724607 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:01.724570 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:29:01.810448 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:01.810407 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5c4467cbb7-wlnbt"] Jul 08 19:29:02.015334 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:29:02.015300 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7df7b87_f170_426a_89b0_9cf26efdeb29.slice/crio-1baf9e45ecf600abb55e6c34e662c42726a3c6f5f2cfca78fd7a27bdc5f8aeb0 WatchSource:0}: Error finding container 1baf9e45ecf600abb55e6c34e662c42726a3c6f5f2cfca78fd7a27bdc5f8aeb0: Status 404 returned error can't find the container with id 1baf9e45ecf600abb55e6c34e662c42726a3c6f5f2cfca78fd7a27bdc5f8aeb0 Jul 08 19:29:02.102505 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:02.102476 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:29:02.202071 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:02.201981 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:29:02.305164 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:02.305083 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:29:02.410361 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:02.410322 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:29:02.513270 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:02.513238 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:29:02.541497 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:02.541455 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-nz2hj" event={"ID":"322a5d1a-2482-4930-a546-31774f7f0c61","Type":"ContainerStarted","Data":"65e9afccc70c0f3c1e41431dc4cb5519e29eb549b5858f1a45b04141b51480d5"} Jul 08 19:29:02.541679 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:02.541528 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-nz2hj" event={"ID":"322a5d1a-2482-4930-a546-31774f7f0c61","Type":"ContainerStarted","Data":"c37c91498b978086d221208dea0bbcd28aa14c10a03bb0b0c5a9db3149b9cfbb"} Jul 08 19:29:02.541679 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:02.541559 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-nz2hj" event={"ID":"322a5d1a-2482-4930-a546-31774f7f0c61","Type":"ContainerStarted","Data":"d67fe8b9d37958285854b70c48903115944ac38ac0e5be960b2ad1a942c7eb4d"} Jul 08 19:29:02.542930 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:02.542895 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5c4467cbb7-wlnbt" event={"ID":"a7df7b87-f170-426a-89b0-9cf26efdeb29","Type":"ContainerStarted","Data":"1baf9e45ecf600abb55e6c34e662c42726a3c6f5f2cfca78fd7a27bdc5f8aeb0"} Jul 08 19:29:02.545528 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:02.545481 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lps28" event={"ID":"406246a8-ab16-4792-8582-83f15d00cafc","Type":"ContainerStarted","Data":"ed4cc094d232430739c9743b11dd405c1028b2bb51fdfdee8e20c358c498a491"} Jul 08 19:29:02.545528 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:02.545529 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lps28" event={"ID":"406246a8-ab16-4792-8582-83f15d00cafc","Type":"ContainerStarted","Data":"65990745fa83e24f5bf28999a4061c002acfbb83ccb623444ebc4d5a4aaae31a"} Jul 08 19:29:02.562555 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:02.562454 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-nz2hj" podStartSLOduration=1.89111413 podStartE2EDuration="3.562440816s" totalImagesPullingTime="1.671326686s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:28:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:29:02.561303465 +0000 UTC m=+308.447493510" watchObservedRunningTime="2026-07-08 19:29:02.562440816 +0000 UTC m=+308.448630863" Jul 08 19:29:02.584713 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:02.584657 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-lps28" podStartSLOduration=2.431423192 podStartE2EDuration="3.58463777s" totalImagesPullingTime="1.087006989s" totalInitContainerRuntime="66.207589ms" isStatefulPod=true podCreationTimestamp="2026-07-08 19:28:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:29:02.582929051 +0000 UTC m=+308.469119116" watchObservedRunningTime="2026-07-08 19:29:02.58463777 +0000 UTC m=+308.470827817" Jul 08 19:29:02.603878 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:02.603834 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:29:02.617781 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:02.617754 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfbjf" Jul 08 19:29:02.917448 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:02.917417 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:29:03.112778 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:03.112750 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:29:03.259152 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:03.259076 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:29:03.550301 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:03.550221 2566 generic.go:356] "Generic (PLEG): container finished" podID="12079511-682f-45ae-b4c3-af244b463829" containerID="3e4b269fcb1f5b11375fb6ec4658322e026215e1ee807fb22dcd4884929e0ade" exitCode=0 Jul 08 19:29:03.550445 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:03.550302 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"12079511-682f-45ae-b4c3-af244b463829","Type":"ContainerDied","Data":"3e4b269fcb1f5b11375fb6ec4658322e026215e1ee807fb22dcd4884929e0ade"} Jul 08 19:29:03.595564 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:03.595530 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-98b9b7785-bj4qt"] Jul 08 19:29:03.600981 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:03.599933 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-98b9b7785-bj4qt" Jul 08 19:29:03.603087 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:03.603054 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-b8rmr\"" Jul 08 19:29:03.605641 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:03.605615 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-bukfhk6s4e7j8\"" Jul 08 19:29:03.605822 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:03.605630 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 08 19:29:03.605924 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:03.605629 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 08 19:29:03.605924 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:03.605630 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 08 19:29:03.607021 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:03.606999 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 08 19:29:03.614016 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:03.613995 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-98b9b7785-bj4qt"] Jul 08 19:29:03.716932 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:03.716894 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e47f05c9-7873-4be6-a51f-ed0ff59dd3fd-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-98b9b7785-bj4qt\" (UID: \"e47f05c9-7873-4be6-a51f-ed0ff59dd3fd\") " pod="openshift-monitoring/metrics-server-98b9b7785-bj4qt" Jul 08 19:29:03.717113 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:03.716957 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e47f05c9-7873-4be6-a51f-ed0ff59dd3fd-secret-metrics-server-client-certs\") pod \"metrics-server-98b9b7785-bj4qt\" (UID: \"e47f05c9-7873-4be6-a51f-ed0ff59dd3fd\") " pod="openshift-monitoring/metrics-server-98b9b7785-bj4qt" Jul 08 19:29:03.717113 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:03.717018 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e47f05c9-7873-4be6-a51f-ed0ff59dd3fd-secret-metrics-server-tls\") pod \"metrics-server-98b9b7785-bj4qt\" (UID: \"e47f05c9-7873-4be6-a51f-ed0ff59dd3fd\") " pod="openshift-monitoring/metrics-server-98b9b7785-bj4qt" Jul 08 19:29:03.717113 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:03.717069 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e47f05c9-7873-4be6-a51f-ed0ff59dd3fd-audit-log\") pod \"metrics-server-98b9b7785-bj4qt\" (UID: \"e47f05c9-7873-4be6-a51f-ed0ff59dd3fd\") " pod="openshift-monitoring/metrics-server-98b9b7785-bj4qt" Jul 08 19:29:03.717241 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:03.717115 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6s2k\" (UniqueName: \"kubernetes.io/projected/e47f05c9-7873-4be6-a51f-ed0ff59dd3fd-kube-api-access-h6s2k\") pod \"metrics-server-98b9b7785-bj4qt\" (UID: \"e47f05c9-7873-4be6-a51f-ed0ff59dd3fd\") " pod="openshift-monitoring/metrics-server-98b9b7785-bj4qt" Jul 08 19:29:03.717241 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:03.717136 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e47f05c9-7873-4be6-a51f-ed0ff59dd3fd-metrics-server-audit-profiles\") pod \"metrics-server-98b9b7785-bj4qt\" (UID: \"e47f05c9-7873-4be6-a51f-ed0ff59dd3fd\") " pod="openshift-monitoring/metrics-server-98b9b7785-bj4qt" Jul 08 19:29:03.717241 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:03.717168 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e47f05c9-7873-4be6-a51f-ed0ff59dd3fd-client-ca-bundle\") pod \"metrics-server-98b9b7785-bj4qt\" (UID: \"e47f05c9-7873-4be6-a51f-ed0ff59dd3fd\") " pod="openshift-monitoring/metrics-server-98b9b7785-bj4qt" Jul 08 19:29:03.818620 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:03.818547 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e47f05c9-7873-4be6-a51f-ed0ff59dd3fd-secret-metrics-server-tls\") pod \"metrics-server-98b9b7785-bj4qt\" (UID: \"e47f05c9-7873-4be6-a51f-ed0ff59dd3fd\") " pod="openshift-monitoring/metrics-server-98b9b7785-bj4qt" Jul 08 19:29:03.818620 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:03.818590 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e47f05c9-7873-4be6-a51f-ed0ff59dd3fd-audit-log\") pod \"metrics-server-98b9b7785-bj4qt\" (UID: \"e47f05c9-7873-4be6-a51f-ed0ff59dd3fd\") " pod="openshift-monitoring/metrics-server-98b9b7785-bj4qt" Jul 08 19:29:03.818814 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:03.818633 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6s2k\" (UniqueName: \"kubernetes.io/projected/e47f05c9-7873-4be6-a51f-ed0ff59dd3fd-kube-api-access-h6s2k\") pod \"metrics-server-98b9b7785-bj4qt\" (UID: \"e47f05c9-7873-4be6-a51f-ed0ff59dd3fd\") " pod="openshift-monitoring/metrics-server-98b9b7785-bj4qt" Jul 08 19:29:03.818814 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:03.818653 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e47f05c9-7873-4be6-a51f-ed0ff59dd3fd-metrics-server-audit-profiles\") pod \"metrics-server-98b9b7785-bj4qt\" (UID: \"e47f05c9-7873-4be6-a51f-ed0ff59dd3fd\") " pod="openshift-monitoring/metrics-server-98b9b7785-bj4qt" Jul 08 19:29:03.818814 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:03.818687 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e47f05c9-7873-4be6-a51f-ed0ff59dd3fd-client-ca-bundle\") pod \"metrics-server-98b9b7785-bj4qt\" (UID: \"e47f05c9-7873-4be6-a51f-ed0ff59dd3fd\") " pod="openshift-monitoring/metrics-server-98b9b7785-bj4qt" Jul 08 19:29:03.818814 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:03.818719 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e47f05c9-7873-4be6-a51f-ed0ff59dd3fd-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-98b9b7785-bj4qt\" (UID: \"e47f05c9-7873-4be6-a51f-ed0ff59dd3fd\") " pod="openshift-monitoring/metrics-server-98b9b7785-bj4qt" Jul 08 19:29:03.818814 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:03.818754 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e47f05c9-7873-4be6-a51f-ed0ff59dd3fd-secret-metrics-server-client-certs\") pod \"metrics-server-98b9b7785-bj4qt\" (UID: \"e47f05c9-7873-4be6-a51f-ed0ff59dd3fd\") " pod="openshift-monitoring/metrics-server-98b9b7785-bj4qt" Jul 08 19:29:03.819232 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:03.819208 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e47f05c9-7873-4be6-a51f-ed0ff59dd3fd-audit-log\") pod \"metrics-server-98b9b7785-bj4qt\" (UID: \"e47f05c9-7873-4be6-a51f-ed0ff59dd3fd\") " pod="openshift-monitoring/metrics-server-98b9b7785-bj4qt" Jul 08 19:29:03.819447 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:03.819429 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e47f05c9-7873-4be6-a51f-ed0ff59dd3fd-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-98b9b7785-bj4qt\" (UID: \"e47f05c9-7873-4be6-a51f-ed0ff59dd3fd\") " pod="openshift-monitoring/metrics-server-98b9b7785-bj4qt" Jul 08 19:29:03.819802 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:03.819776 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e47f05c9-7873-4be6-a51f-ed0ff59dd3fd-metrics-server-audit-profiles\") pod \"metrics-server-98b9b7785-bj4qt\" (UID: \"e47f05c9-7873-4be6-a51f-ed0ff59dd3fd\") " pod="openshift-monitoring/metrics-server-98b9b7785-bj4qt" Jul 08 19:29:03.821085 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:03.821057 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e47f05c9-7873-4be6-a51f-ed0ff59dd3fd-secret-metrics-server-tls\") pod \"metrics-server-98b9b7785-bj4qt\" (UID: \"e47f05c9-7873-4be6-a51f-ed0ff59dd3fd\") " pod="openshift-monitoring/metrics-server-98b9b7785-bj4qt" Jul 08 19:29:03.821173 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:03.821141 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e47f05c9-7873-4be6-a51f-ed0ff59dd3fd-secret-metrics-server-client-certs\") pod \"metrics-server-98b9b7785-bj4qt\" (UID: \"e47f05c9-7873-4be6-a51f-ed0ff59dd3fd\") " pod="openshift-monitoring/metrics-server-98b9b7785-bj4qt" Jul 08 19:29:03.821215 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:03.821180 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e47f05c9-7873-4be6-a51f-ed0ff59dd3fd-client-ca-bundle\") pod \"metrics-server-98b9b7785-bj4qt\" (UID: \"e47f05c9-7873-4be6-a51f-ed0ff59dd3fd\") " pod="openshift-monitoring/metrics-server-98b9b7785-bj4qt" Jul 08 19:29:03.827526 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:03.827496 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6s2k\" (UniqueName: \"kubernetes.io/projected/e47f05c9-7873-4be6-a51f-ed0ff59dd3fd-kube-api-access-h6s2k\") pod \"metrics-server-98b9b7785-bj4qt\" (UID: \"e47f05c9-7873-4be6-a51f-ed0ff59dd3fd\") " pod="openshift-monitoring/metrics-server-98b9b7785-bj4qt" Jul 08 19:29:03.912586 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:03.912550 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-98b9b7785-bj4qt" Jul 08 19:29:03.982482 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:03.982455 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:29:04.055173 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:04.055140 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-98b9b7785-bj4qt"] Jul 08 19:29:04.058404 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:29:04.058370 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode47f05c9_7873_4be6_a51f_ed0ff59dd3fd.slice/crio-c353d0824b74f0f7818d76c6dddca216233b7012a45b905b6bf68dbb870dec07 WatchSource:0}: Error finding container c353d0824b74f0f7818d76c6dddca216233b7012a45b905b6bf68dbb870dec07: Status 404 returned error can't find the container with id c353d0824b74f0f7818d76c6dddca216233b7012a45b905b6bf68dbb870dec07 Jul 08 19:29:04.058902 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:04.058838 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:29:04.201250 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:04.201203 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:29:04.278257 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:04.278227 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:29:04.555990 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:04.555901 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-98b9b7785-bj4qt" event={"ID":"e47f05c9-7873-4be6-a51f-ed0ff59dd3fd","Type":"ContainerStarted","Data":"c353d0824b74f0f7818d76c6dddca216233b7012a45b905b6bf68dbb870dec07"} Jul 08 19:29:05.471218 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:05.470982 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6955c4f578-svncz" Jul 08 19:29:05.519504 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:05.519458 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:29:05.562703 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:05.562674 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5c4467cbb7-wlnbt" event={"ID":"a7df7b87-f170-426a-89b0-9cf26efdeb29","Type":"ContainerStarted","Data":"a8dd9f5be44aa7e10ff0ae4fd63cf4406d1f98f29c14b4f692f35cf4ff7206ae"} Jul 08 19:29:05.562814 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:05.562753 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5c4467cbb7-wlnbt" event={"ID":"a7df7b87-f170-426a-89b0-9cf26efdeb29","Type":"ContainerStarted","Data":"433a8032df6d6ac5162b8fbf74af198cddc37e9e3799e72dfeb202204718114e"} Jul 08 19:29:05.566501 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:05.566434 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"12079511-682f-45ae-b4c3-af244b463829","Type":"ContainerStarted","Data":"66ee9bea168cd0f38a906021ad3992a682ed326d74702846821bc8951b29c8e4"} Jul 08 19:29:05.566616 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:05.566516 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"12079511-682f-45ae-b4c3-af244b463829","Type":"ContainerStarted","Data":"65bab8dc868e908071f91d0fadd9cc63017ab5a12ecb190e4cc4771a48086929"} Jul 08 19:29:05.656105 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:05.656068 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:29:05.762132 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:05.762104 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:29:05.878804 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:05.874461 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:29:05.976654 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:05.976615 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:29:06.573143 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:06.573070 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5c4467cbb7-wlnbt" event={"ID":"a7df7b87-f170-426a-89b0-9cf26efdeb29","Type":"ContainerStarted","Data":"985ba6512ee5fbeb4640c5545096abbf022c35650af4732617b5eb5e92c0439a"} Jul 08 19:29:06.576811 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:06.576784 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"12079511-682f-45ae-b4c3-af244b463829","Type":"ContainerStarted","Data":"86928b2d2629a6c3ca019cafa275f72d1801d5a7137c7c29d8f59452a45e92bc"} Jul 08 19:29:06.576961 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:06.576875 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"12079511-682f-45ae-b4c3-af244b463829","Type":"ContainerStarted","Data":"6d9943ddb635dba5892ff153093281e71c274533ba8962696f33ce68e72493c1"} Jul 08 19:29:06.576961 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:06.576924 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"12079511-682f-45ae-b4c3-af244b463829","Type":"ContainerStarted","Data":"159b0ac6ae48cea9a0e3ae32b971a8da13fe2f284e8607dfe6927f24c18bb5c1"} Jul 08 19:29:06.578428 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:06.578403 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-98b9b7785-bj4qt" event={"ID":"e47f05c9-7873-4be6-a51f-ed0ff59dd3fd","Type":"ContainerStarted","Data":"690fd2250059552ae6fd2d6229953b3549d8074ca9fca42755e3bee4f6c3813a"} Jul 08 19:29:06.606938 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:06.606874 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-98b9b7785-bj4qt" podStartSLOduration=1.4695070590000001 podStartE2EDuration="3.606839899s" totalImagesPullingTime="2.13733284s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:29:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:29:06.606785574 +0000 UTC m=+312.492975622" watchObservedRunningTime="2026-07-08 19:29:06.606839899 +0000 UTC m=+312.493029949" Jul 08 19:29:07.585072 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:07.585029 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5c4467cbb7-wlnbt" event={"ID":"a7df7b87-f170-426a-89b0-9cf26efdeb29","Type":"ContainerStarted","Data":"4b9afc7854ff59d050c29bdd4b46eba04461509c52652a95f3e38c4517bbac84"} Jul 08 19:29:07.585571 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:07.585111 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5c4467cbb7-wlnbt" event={"ID":"a7df7b87-f170-426a-89b0-9cf26efdeb29","Type":"ContainerStarted","Data":"d9763ad7e86279c224eb45331cd873c3ea92c9d847a14eb0ef11b4603f123513"} Jul 08 19:29:07.585571 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:07.585159 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5c4467cbb7-wlnbt" Jul 08 19:29:07.585571 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:07.585203 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5c4467cbb7-wlnbt" event={"ID":"a7df7b87-f170-426a-89b0-9cf26efdeb29","Type":"ContainerStarted","Data":"39722771324ee1d25f0ea374c6b72e02cfa7c506e7c6ea68c2be7b177f86cae7"} Jul 08 19:29:07.588163 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:07.588134 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"12079511-682f-45ae-b4c3-af244b463829","Type":"ContainerStarted","Data":"3728d9f434d001e11e4e62972618fd1f553411569b170767907ea5367de3b347"} Jul 08 19:29:07.618452 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:07.618399 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5c4467cbb7-wlnbt" podStartSLOduration=2.149406598 podStartE2EDuration="6.618377718s" totalImagesPullingTime="4.46897112s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:29:01 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:29:07.617190601 +0000 UTC m=+313.503380649" watchObservedRunningTime="2026-07-08 19:29:07.618377718 +0000 UTC m=+313.504567765" Jul 08 19:29:07.662509 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:07.662461 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.985638441 podStartE2EDuration="7.662444352s" totalImagesPullingTime="4.65604399s" totalInitContainerRuntime="20.761921ms" isStatefulPod=false podCreationTimestamp="2026-07-08 19:29:00 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:29:07.662120295 +0000 UTC m=+313.548310341" watchObservedRunningTime="2026-07-08 19:29:07.662444352 +0000 UTC m=+313.548634400" Jul 08 19:29:08.596441 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:08.596407 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:29:08.709651 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:08.709618 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:29:13.348852 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:13.348817 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:29:13.424483 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:13.424450 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:29:13.597692 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:13.597656 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5c4467cbb7-wlnbt" Jul 08 19:29:16.743788 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:16.743757 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:29:16.811196 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:16.811168 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:29:17.473268 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:17.473230 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:29:17.574171 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:17.574119 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:29:18.137588 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:18.137553 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:29:18.190989 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:18.190956 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:29:20.544325 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:20.544288 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:29:23.912911 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:23.912857 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-98b9b7785-bj4qt" Jul 08 19:29:23.913352 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:23.912978 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-98b9b7785-bj4qt" Jul 08 19:29:30.546513 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:30.546479 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:29:30.621528 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:30.621490 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:29:32.468150 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:32.468120 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:29:32.569090 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:32.569057 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:29:32.629466 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:32.629422 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:29:33.342034 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:33.341998 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:29:33.403663 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:33.403633 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:29:34.141154 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:34.141122 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:29:34.201987 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:34.201951 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:29:43.719238 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:43.719203 2566 generic.go:356] "Generic (PLEG): container finished" podID="6c0d602f-7a0a-415c-9716-54ffb6c51e67" containerID="a9e0335b74d47058567cea0ca77a67ec109d67d1c8c077b80b42a9b9e2ac9fb2" exitCode=0 Jul 08 19:29:43.719654 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:43.719245 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-gfkmh" event={"ID":"6c0d602f-7a0a-415c-9716-54ffb6c51e67","Type":"ContainerDied","Data":"a9e0335b74d47058567cea0ca77a67ec109d67d1c8c077b80b42a9b9e2ac9fb2"} Jul 08 19:29:43.719654 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:43.719539 2566 scope.go:117] "RemoveContainer" containerID="a9e0335b74d47058567cea0ca77a67ec109d67d1c8c077b80b42a9b9e2ac9fb2" Jul 08 19:29:43.919000 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:43.918966 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-98b9b7785-bj4qt" Jul 08 19:29:43.924617 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:43.924590 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-98b9b7785-bj4qt" Jul 08 19:29:44.537193 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:44.537160 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5846dc5c45-fgr6x_cee815ab-2b57-4584-8b6c-a8748c849be9/router/0.log" Jul 08 19:29:44.543261 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:44.543238 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5nfnp_d36b2cab-e4d2-4a23-85f9-0bd24ccbb497/serve-healthcheck-canary/0.log" Jul 08 19:29:44.726397 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:44.724769 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-gfkmh" event={"ID":"6c0d602f-7a0a-415c-9716-54ffb6c51e67","Type":"ContainerStarted","Data":"aa227d67c0385d1c03b60f433b23e23bf399d9078bc8f47d9c0106a0509ea4fb"} Jul 08 19:29:47.475703 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:47.475617 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:29:47.576027 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:47.575995 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:29:50.746773 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:50.746740 2566 generic.go:356] "Generic (PLEG): container finished" podID="05373d28-0e17-422e-96ec-1add0377dd9d" containerID="d01e0c9dc5c50c4ce9c48083a3093b06da7826a74e5a276c7c17715e1bd448a5" exitCode=0 Jul 08 19:29:50.747181 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:50.746813 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-7ntp9" event={"ID":"05373d28-0e17-422e-96ec-1add0377dd9d","Type":"ContainerDied","Data":"d01e0c9dc5c50c4ce9c48083a3093b06da7826a74e5a276c7c17715e1bd448a5"} Jul 08 19:29:50.747226 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:50.747179 2566 scope.go:117] "RemoveContainer" containerID="d01e0c9dc5c50c4ce9c48083a3093b06da7826a74e5a276c7c17715e1bd448a5" Jul 08 19:29:50.851493 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:50.851400 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:29:51.579249 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:51.579202 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:29:51.679894 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:51.679842 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:29:51.753164 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:51.753123 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-7ntp9" event={"ID":"05373d28-0e17-422e-96ec-1add0377dd9d","Type":"ContainerStarted","Data":"c36494b888ea79f093b3ff31e0f6e2360e589e5d948ed89906c3cd29122a1d48"} Jul 08 19:29:51.774981 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:29:51.774942 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:30:04.886561 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:04.886515 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:30:04.951349 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:04.951315 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:30:08.485977 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:08.485936 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55818509-538d-4733-9e10-eaaf841ef071-metrics-certs\") pod \"network-metrics-daemon-rfbjf\" (UID: \"55818509-538d-4733-9e10-eaaf841ef071\") " pod="openshift-multus/network-metrics-daemon-rfbjf" Jul 08 19:30:08.488140 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:08.488118 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55818509-538d-4733-9e10-eaaf841ef071-metrics-certs\") pod \"network-metrics-daemon-rfbjf\" (UID: \"55818509-538d-4733-9e10-eaaf841ef071\") " pod="openshift-multus/network-metrics-daemon-rfbjf" Jul 08 19:30:08.621268 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:08.621240 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hgx5c\"" Jul 08 19:30:08.629244 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:08.629218 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfbjf" Jul 08 19:30:08.757087 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:08.757012 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rfbjf"] Jul 08 19:30:08.759947 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:30:08.759912 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55818509_538d_4733_9e10_eaaf841ef071.slice/crio-15e48e58eee9cd110a6b86d49db2c1a703f07a00686c7a008b70cc71eb4819cd WatchSource:0}: Error finding container 15e48e58eee9cd110a6b86d49db2c1a703f07a00686c7a008b70cc71eb4819cd: Status 404 returned error can't find the container with id 15e48e58eee9cd110a6b86d49db2c1a703f07a00686c7a008b70cc71eb4819cd Jul 08 19:30:08.826744 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:08.826701 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rfbjf" event={"ID":"55818509-538d-4733-9e10-eaaf841ef071","Type":"ContainerStarted","Data":"15e48e58eee9cd110a6b86d49db2c1a703f07a00686c7a008b70cc71eb4819cd"} Jul 08 19:30:09.832507 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:09.832469 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rfbjf" event={"ID":"55818509-538d-4733-9e10-eaaf841ef071","Type":"ContainerStarted","Data":"1a3e6cb43c49408e14c5ba600cbbf6f97fddb2bcddb95e50ca89b7833ff1597f"} Jul 08 19:30:10.840460 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:10.840422 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rfbjf" event={"ID":"55818509-538d-4733-9e10-eaaf841ef071","Type":"ContainerStarted","Data":"595c4d637f2d768ec8504303d9c51259ecc041979e0a34528ff1c45ac222df51"} Jul 08 19:30:10.859105 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:10.859045 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-rfbjf" podStartSLOduration=375.907269551 podStartE2EDuration="6m16.859026783s" totalImagesPullingTime="951.757232ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:23:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:30:10.857974687 +0000 UTC m=+376.744164735" watchObservedRunningTime="2026-07-08 19:30:10.859026783 +0000 UTC m=+376.745216829" Jul 08 19:30:10.965943 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:10.965888 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:30:11.029650 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:11.029618 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:30:11.089590 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:11.089534 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:30:15.404825 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:15.404791 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:30:15.464076 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:15.464042 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:30:16.447785 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:16.447741 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:30:16.553970 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:16.553934 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:30:17.021093 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:17.021054 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:30:17.102790 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:17.102750 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:30:18.457522 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:18.457490 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:30:19.727016 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:19.726980 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 08 19:30:19.727437 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:19.727276 2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="12079511-682f-45ae-b4c3-af244b463829" containerName="alertmanager" containerID="cri-o://65bab8dc868e908071f91d0fadd9cc63017ab5a12ecb190e4cc4771a48086929" gracePeriod=120 Jul 08 19:30:19.727437 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:19.727354 2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="12079511-682f-45ae-b4c3-af244b463829" containerName="kube-rbac-proxy-metric" containerID="cri-o://86928b2d2629a6c3ca019cafa275f72d1801d5a7137c7c29d8f59452a45e92bc" gracePeriod=120 Jul 08 19:30:19.727437 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:19.727372 2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="12079511-682f-45ae-b4c3-af244b463829" containerName="kube-rbac-proxy-web" containerID="cri-o://159b0ac6ae48cea9a0e3ae32b971a8da13fe2f284e8607dfe6927f24c18bb5c1" gracePeriod=120 Jul 08 19:30:19.727437 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:19.727385 2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="12079511-682f-45ae-b4c3-af244b463829" containerName="kube-rbac-proxy" containerID="cri-o://6d9943ddb635dba5892ff153093281e71c274533ba8962696f33ce68e72493c1" gracePeriod=120 Jul 08 19:30:19.727630 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:19.727416 2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="12079511-682f-45ae-b4c3-af244b463829" containerName="prom-label-proxy" containerID="cri-o://3728d9f434d001e11e4e62972618fd1f553411569b170767907ea5367de3b347" gracePeriod=120 Jul 08 19:30:19.727630 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:19.727453 2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="12079511-682f-45ae-b4c3-af244b463829" containerName="config-reloader" containerID="cri-o://66ee9bea168cd0f38a906021ad3992a682ed326d74702846821bc8951b29c8e4" gracePeriod=120 Jul 08 19:30:19.877938 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:19.877906 2566 generic.go:356] "Generic (PLEG): container finished" podID="12079511-682f-45ae-b4c3-af244b463829" containerID="3728d9f434d001e11e4e62972618fd1f553411569b170767907ea5367de3b347" exitCode=0 Jul 08 19:30:19.877938 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:19.877931 2566 generic.go:356] "Generic (PLEG): container finished" podID="12079511-682f-45ae-b4c3-af244b463829" containerID="6d9943ddb635dba5892ff153093281e71c274533ba8962696f33ce68e72493c1" exitCode=0 Jul 08 19:30:19.877938 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:19.877937 2566 generic.go:356] "Generic (PLEG): container finished" podID="12079511-682f-45ae-b4c3-af244b463829" containerID="66ee9bea168cd0f38a906021ad3992a682ed326d74702846821bc8951b29c8e4" exitCode=0 Jul 08 19:30:19.878106 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:19.877945 2566 generic.go:356] "Generic (PLEG): container finished" podID="12079511-682f-45ae-b4c3-af244b463829" containerID="65bab8dc868e908071f91d0fadd9cc63017ab5a12ecb190e4cc4771a48086929" exitCode=0 Jul 08 19:30:19.878106 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:19.877972 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"12079511-682f-45ae-b4c3-af244b463829","Type":"ContainerDied","Data":"3728d9f434d001e11e4e62972618fd1f553411569b170767907ea5367de3b347"} Jul 08 19:30:19.878106 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:19.878059 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"12079511-682f-45ae-b4c3-af244b463829","Type":"ContainerDied","Data":"6d9943ddb635dba5892ff153093281e71c274533ba8962696f33ce68e72493c1"} Jul 08 19:30:19.878106 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:19.878085 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"12079511-682f-45ae-b4c3-af244b463829","Type":"ContainerDied","Data":"66ee9bea168cd0f38a906021ad3992a682ed326d74702846821bc8951b29c8e4"} Jul 08 19:30:19.878241 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:19.878111 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"12079511-682f-45ae-b4c3-af244b463829","Type":"ContainerDied","Data":"65bab8dc868e908071f91d0fadd9cc63017ab5a12ecb190e4cc4771a48086929"} Jul 08 19:30:20.886233 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:20.886205 2566 generic.go:356] "Generic (PLEG): container finished" podID="12079511-682f-45ae-b4c3-af244b463829" containerID="86928b2d2629a6c3ca019cafa275f72d1801d5a7137c7c29d8f59452a45e92bc" exitCode=0 Jul 08 19:30:20.886233 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:20.886230 2566 generic.go:356] "Generic (PLEG): container finished" podID="12079511-682f-45ae-b4c3-af244b463829" containerID="159b0ac6ae48cea9a0e3ae32b971a8da13fe2f284e8607dfe6927f24c18bb5c1" exitCode=0 Jul 08 19:30:20.886616 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:20.886255 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"12079511-682f-45ae-b4c3-af244b463829","Type":"ContainerDied","Data":"86928b2d2629a6c3ca019cafa275f72d1801d5a7137c7c29d8f59452a45e92bc"} Jul 08 19:30:20.886616 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:20.886332 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"12079511-682f-45ae-b4c3-af244b463829","Type":"ContainerDied","Data":"159b0ac6ae48cea9a0e3ae32b971a8da13fe2f284e8607dfe6927f24c18bb5c1"} Jul 08 19:30:20.988631 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:20.988608 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:30:21.092238 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.092152 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/12079511-682f-45ae-b4c3-af244b463829-tls-assets\") pod \"12079511-682f-45ae-b4c3-af244b463829\" (UID: \"12079511-682f-45ae-b4c3-af244b463829\") " Jul 08 19:30:21.092238 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.092190 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/12079511-682f-45ae-b4c3-af244b463829-secret-alertmanager-kube-rbac-proxy-metric\") pod \"12079511-682f-45ae-b4c3-af244b463829\" (UID: \"12079511-682f-45ae-b4c3-af244b463829\") " Jul 08 19:30:21.092238 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.092228 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/12079511-682f-45ae-b4c3-af244b463829-secret-alertmanager-kube-rbac-proxy\") pod \"12079511-682f-45ae-b4c3-af244b463829\" (UID: \"12079511-682f-45ae-b4c3-af244b463829\") " Jul 08 19:30:21.092604 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.092260 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/12079511-682f-45ae-b4c3-af244b463829-secret-alertmanager-main-tls\") pod \"12079511-682f-45ae-b4c3-af244b463829\" (UID: \"12079511-682f-45ae-b4c3-af244b463829\") " Jul 08 19:30:21.092604 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.092282 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12079511-682f-45ae-b4c3-af244b463829-alertmanager-trusted-ca-bundle\") pod \"12079511-682f-45ae-b4c3-af244b463829\" (UID: \"12079511-682f-45ae-b4c3-af244b463829\") " Jul 08 19:30:21.092604 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.092304 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/12079511-682f-45ae-b4c3-af244b463829-secret-alertmanager-kube-rbac-proxy-web\") pod \"12079511-682f-45ae-b4c3-af244b463829\" (UID: \"12079511-682f-45ae-b4c3-af244b463829\") " Jul 08 19:30:21.092604 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.092320 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/12079511-682f-45ae-b4c3-af244b463829-config-volume\") pod \"12079511-682f-45ae-b4c3-af244b463829\" (UID: \"12079511-682f-45ae-b4c3-af244b463829\") " Jul 08 19:30:21.092604 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.092507 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/12079511-682f-45ae-b4c3-af244b463829-alertmanager-main-db\") pod \"12079511-682f-45ae-b4c3-af244b463829\" (UID: \"12079511-682f-45ae-b4c3-af244b463829\") " Jul 08 19:30:21.092604 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.092554 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2x2qp\" (UniqueName: \"kubernetes.io/projected/12079511-682f-45ae-b4c3-af244b463829-kube-api-access-2x2qp\") pod \"12079511-682f-45ae-b4c3-af244b463829\" (UID: \"12079511-682f-45ae-b4c3-af244b463829\") " Jul 08 19:30:21.092604 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.092590 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/12079511-682f-45ae-b4c3-af244b463829-metrics-client-ca\") pod \"12079511-682f-45ae-b4c3-af244b463829\" (UID: \"12079511-682f-45ae-b4c3-af244b463829\") " Jul 08 19:30:21.092961 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.092617 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/12079511-682f-45ae-b4c3-af244b463829-cluster-tls-config\") pod \"12079511-682f-45ae-b4c3-af244b463829\" (UID: \"12079511-682f-45ae-b4c3-af244b463829\") " Jul 08 19:30:21.092961 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.092652 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/12079511-682f-45ae-b4c3-af244b463829-config-out\") pod \"12079511-682f-45ae-b4c3-af244b463829\" (UID: \"12079511-682f-45ae-b4c3-af244b463829\") " Jul 08 19:30:21.092961 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.092687 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/12079511-682f-45ae-b4c3-af244b463829-web-config\") pod \"12079511-682f-45ae-b4c3-af244b463829\" (UID: \"12079511-682f-45ae-b4c3-af244b463829\") " Jul 08 19:30:21.093120 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.093094 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12079511-682f-45ae-b4c3-af244b463829-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "12079511-682f-45ae-b4c3-af244b463829" (UID: "12079511-682f-45ae-b4c3-af244b463829"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:30:21.093180 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.093146 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12079511-682f-45ae-b4c3-af244b463829-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "12079511-682f-45ae-b4c3-af244b463829" (UID: "12079511-682f-45ae-b4c3-af244b463829"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:30:21.093526 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.093499 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12079511-682f-45ae-b4c3-af244b463829-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "12079511-682f-45ae-b4c3-af244b463829" (UID: "12079511-682f-45ae-b4c3-af244b463829"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:30:21.095097 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.095062 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12079511-682f-45ae-b4c3-af244b463829-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "12079511-682f-45ae-b4c3-af244b463829" (UID: "12079511-682f-45ae-b4c3-af244b463829"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:30:21.095414 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.095386 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12079511-682f-45ae-b4c3-af244b463829-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "12079511-682f-45ae-b4c3-af244b463829" (UID: "12079511-682f-45ae-b4c3-af244b463829"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:30:21.095488 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.095464 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12079511-682f-45ae-b4c3-af244b463829-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "12079511-682f-45ae-b4c3-af244b463829" (UID: "12079511-682f-45ae-b4c3-af244b463829"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:30:21.095565 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.095486 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12079511-682f-45ae-b4c3-af244b463829-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "12079511-682f-45ae-b4c3-af244b463829" (UID: "12079511-682f-45ae-b4c3-af244b463829"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:30:21.096025 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.095992 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12079511-682f-45ae-b4c3-af244b463829-config-volume" (OuterVolumeSpecName: "config-volume") pod "12079511-682f-45ae-b4c3-af244b463829" (UID: "12079511-682f-45ae-b4c3-af244b463829"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:30:21.096025 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.096013 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12079511-682f-45ae-b4c3-af244b463829-config-out" (OuterVolumeSpecName: "config-out") pod "12079511-682f-45ae-b4c3-af244b463829" (UID: "12079511-682f-45ae-b4c3-af244b463829"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:30:21.096193 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.096040 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12079511-682f-45ae-b4c3-af244b463829-kube-api-access-2x2qp" (OuterVolumeSpecName: "kube-api-access-2x2qp") pod "12079511-682f-45ae-b4c3-af244b463829" (UID: "12079511-682f-45ae-b4c3-af244b463829"). InnerVolumeSpecName "kube-api-access-2x2qp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:30:21.096691 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.096670 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12079511-682f-45ae-b4c3-af244b463829-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "12079511-682f-45ae-b4c3-af244b463829" (UID: "12079511-682f-45ae-b4c3-af244b463829"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:30:21.100434 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.100413 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12079511-682f-45ae-b4c3-af244b463829-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "12079511-682f-45ae-b4c3-af244b463829" (UID: "12079511-682f-45ae-b4c3-af244b463829"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:30:21.106779 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.106759 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12079511-682f-45ae-b4c3-af244b463829-web-config" (OuterVolumeSpecName: "web-config") pod "12079511-682f-45ae-b4c3-af244b463829" (UID: "12079511-682f-45ae-b4c3-af244b463829"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:30:21.194100 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.194068 2566 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/12079511-682f-45ae-b4c3-af244b463829-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-142-195.ec2.internal\" DevicePath \"\"" Jul 08 19:30:21.194100 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.194099 2566 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/12079511-682f-45ae-b4c3-af244b463829-secret-alertmanager-main-tls\") on node \"ip-10-0-142-195.ec2.internal\" DevicePath \"\"" Jul 08 19:30:21.194255 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.194110 2566 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12079511-682f-45ae-b4c3-af244b463829-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-142-195.ec2.internal\" DevicePath \"\"" Jul 08 19:30:21.194255 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.194119 2566 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/12079511-682f-45ae-b4c3-af244b463829-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-142-195.ec2.internal\" DevicePath \"\"" Jul 08 19:30:21.194255 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.194129 2566 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/12079511-682f-45ae-b4c3-af244b463829-config-volume\") on node \"ip-10-0-142-195.ec2.internal\" DevicePath \"\"" Jul 08 19:30:21.194255 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.194137 2566 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/12079511-682f-45ae-b4c3-af244b463829-alertmanager-main-db\") on node \"ip-10-0-142-195.ec2.internal\" DevicePath \"\"" Jul 08 19:30:21.194255 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.194146 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2x2qp\" (UniqueName: \"kubernetes.io/projected/12079511-682f-45ae-b4c3-af244b463829-kube-api-access-2x2qp\") on node \"ip-10-0-142-195.ec2.internal\" DevicePath \"\"" Jul 08 19:30:21.194255 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.194154 2566 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/12079511-682f-45ae-b4c3-af244b463829-metrics-client-ca\") on node \"ip-10-0-142-195.ec2.internal\" DevicePath \"\"" Jul 08 19:30:21.194255 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.194162 2566 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/12079511-682f-45ae-b4c3-af244b463829-cluster-tls-config\") on node \"ip-10-0-142-195.ec2.internal\" DevicePath \"\"" Jul 08 19:30:21.194255 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.194170 2566 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/12079511-682f-45ae-b4c3-af244b463829-config-out\") on node \"ip-10-0-142-195.ec2.internal\" DevicePath \"\"" Jul 08 19:30:21.194255 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.194177 2566 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/12079511-682f-45ae-b4c3-af244b463829-web-config\") on node \"ip-10-0-142-195.ec2.internal\" DevicePath \"\"" Jul 08 19:30:21.194255 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.194184 2566 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/12079511-682f-45ae-b4c3-af244b463829-tls-assets\") on node \"ip-10-0-142-195.ec2.internal\" DevicePath \"\"" Jul 08 19:30:21.194255 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.194193 2566 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/12079511-682f-45ae-b4c3-af244b463829-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-142-195.ec2.internal\" DevicePath \"\"" Jul 08 19:30:21.894036 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.894003 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"12079511-682f-45ae-b4c3-af244b463829","Type":"ContainerDied","Data":"a733e70cf9d1056648f7207ee64b7bda7bfed64332f539ab4b88b4828cf5e16e"} Jul 08 19:30:21.894036 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.894033 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:30:21.894529 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.894078 2566 scope.go:117] "RemoveContainer" containerID="3728d9f434d001e11e4e62972618fd1f553411569b170767907ea5367de3b347" Jul 08 19:30:21.902019 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.901997 2566 scope.go:117] "RemoveContainer" containerID="86928b2d2629a6c3ca019cafa275f72d1801d5a7137c7c29d8f59452a45e92bc" Jul 08 19:30:21.909223 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.909195 2566 scope.go:117] "RemoveContainer" containerID="6d9943ddb635dba5892ff153093281e71c274533ba8962696f33ce68e72493c1" Jul 08 19:30:21.916114 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.916096 2566 scope.go:117] "RemoveContainer" containerID="159b0ac6ae48cea9a0e3ae32b971a8da13fe2f284e8607dfe6927f24c18bb5c1" Jul 08 19:30:21.918493 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.918473 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 08 19:30:21.925496 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.925341 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 08 19:30:21.925683 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.925592 2566 scope.go:117] "RemoveContainer" containerID="66ee9bea168cd0f38a906021ad3992a682ed326d74702846821bc8951b29c8e4" Jul 08 19:30:21.933158 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.933142 2566 scope.go:117] "RemoveContainer" containerID="65bab8dc868e908071f91d0fadd9cc63017ab5a12ecb190e4cc4771a48086929" Jul 08 19:30:21.940296 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.940219 2566 scope.go:117] "RemoveContainer" containerID="3e4b269fcb1f5b11375fb6ec4658322e026215e1ee807fb22dcd4884929e0ade" Jul 08 19:30:21.951104 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.951082 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 08 19:30:21.951357 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.951345 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12079511-682f-45ae-b4c3-af244b463829" containerName="kube-rbac-proxy-metric" Jul 08 19:30:21.951397 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.951359 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="12079511-682f-45ae-b4c3-af244b463829" containerName="kube-rbac-proxy-metric" Jul 08 19:30:21.951397 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.951369 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12079511-682f-45ae-b4c3-af244b463829" containerName="init-config-reloader" Jul 08 19:30:21.951397 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.951376 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="12079511-682f-45ae-b4c3-af244b463829" containerName="init-config-reloader" Jul 08 19:30:21.951397 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.951383 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12079511-682f-45ae-b4c3-af244b463829" containerName="prom-label-proxy" Jul 08 19:30:21.951397 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.951388 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="12079511-682f-45ae-b4c3-af244b463829" containerName="prom-label-proxy" Jul 08 19:30:21.951397 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.951395 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12079511-682f-45ae-b4c3-af244b463829" containerName="kube-rbac-proxy" Jul 08 19:30:21.951569 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.951400 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="12079511-682f-45ae-b4c3-af244b463829" containerName="kube-rbac-proxy" Jul 08 19:30:21.951569 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.951410 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12079511-682f-45ae-b4c3-af244b463829" containerName="kube-rbac-proxy-web" Jul 08 19:30:21.951569 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.951415 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="12079511-682f-45ae-b4c3-af244b463829" containerName="kube-rbac-proxy-web" Jul 08 19:30:21.951569 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.951428 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12079511-682f-45ae-b4c3-af244b463829" containerName="alertmanager" Jul 08 19:30:21.951569 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.951433 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="12079511-682f-45ae-b4c3-af244b463829" containerName="alertmanager" Jul 08 19:30:21.951569 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.951441 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12079511-682f-45ae-b4c3-af244b463829" containerName="config-reloader" Jul 08 19:30:21.951569 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.951446 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="12079511-682f-45ae-b4c3-af244b463829" containerName="config-reloader" Jul 08 19:30:21.951569 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.951486 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="12079511-682f-45ae-b4c3-af244b463829" containerName="prom-label-proxy" Jul 08 19:30:21.951569 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.951494 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="12079511-682f-45ae-b4c3-af244b463829" containerName="kube-rbac-proxy" Jul 08 19:30:21.951569 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.951500 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="12079511-682f-45ae-b4c3-af244b463829" containerName="config-reloader" Jul 08 19:30:21.951569 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.951506 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="12079511-682f-45ae-b4c3-af244b463829" containerName="alertmanager" Jul 08 19:30:21.951569 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.951511 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="12079511-682f-45ae-b4c3-af244b463829" containerName="kube-rbac-proxy-metric" Jul 08 19:30:21.951569 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.951516 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="12079511-682f-45ae-b4c3-af244b463829" containerName="kube-rbac-proxy-web" Jul 08 19:30:21.955155 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.955139 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:30:21.957443 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.957422 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 08 19:30:21.957760 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.957730 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 08 19:30:21.957853 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.957768 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 08 19:30:21.957853 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.957774 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-99g6l\"" Jul 08 19:30:21.957853 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.957847 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 08 19:30:21.958063 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.958048 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 08 19:30:21.958141 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.958072 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 08 19:30:21.958141 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.958084 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 08 19:30:21.958317 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.958303 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 08 19:30:21.963558 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.963537 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 08 19:30:21.970937 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.970916 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 08 19:30:21.999614 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.999580 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a0321d1f-21e9-4d20-af13-ed8a1fa71638-web-config\") pod \"alertmanager-main-0\" (UID: \"a0321d1f-21e9-4d20-af13-ed8a1fa71638\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:30:21.999614 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.999611 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a0321d1f-21e9-4d20-af13-ed8a1fa71638-config-out\") pod \"alertmanager-main-0\" (UID: \"a0321d1f-21e9-4d20-af13-ed8a1fa71638\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:30:21.999832 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.999630 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a0321d1f-21e9-4d20-af13-ed8a1fa71638-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a0321d1f-21e9-4d20-af13-ed8a1fa71638\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:30:21.999832 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.999652 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a0321d1f-21e9-4d20-af13-ed8a1fa71638-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a0321d1f-21e9-4d20-af13-ed8a1fa71638\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:30:21.999832 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.999673 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2gks\" (UniqueName: \"kubernetes.io/projected/a0321d1f-21e9-4d20-af13-ed8a1fa71638-kube-api-access-r2gks\") pod \"alertmanager-main-0\" (UID: \"a0321d1f-21e9-4d20-af13-ed8a1fa71638\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:30:21.999832 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.999705 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a0321d1f-21e9-4d20-af13-ed8a1fa71638-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a0321d1f-21e9-4d20-af13-ed8a1fa71638\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:30:21.999832 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.999722 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a0321d1f-21e9-4d20-af13-ed8a1fa71638-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a0321d1f-21e9-4d20-af13-ed8a1fa71638\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:30:21.999832 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.999774 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a0321d1f-21e9-4d20-af13-ed8a1fa71638-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a0321d1f-21e9-4d20-af13-ed8a1fa71638\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:30:21.999832 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.999818 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a0321d1f-21e9-4d20-af13-ed8a1fa71638-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a0321d1f-21e9-4d20-af13-ed8a1fa71638\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:30:22.000130 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.999847 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0321d1f-21e9-4d20-af13-ed8a1fa71638-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a0321d1f-21e9-4d20-af13-ed8a1fa71638\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:30:22.000130 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.999911 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0321d1f-21e9-4d20-af13-ed8a1fa71638-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a0321d1f-21e9-4d20-af13-ed8a1fa71638\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:30:22.000130 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.999965 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a0321d1f-21e9-4d20-af13-ed8a1fa71638-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a0321d1f-21e9-4d20-af13-ed8a1fa71638\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:30:22.000130 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:21.999984 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a0321d1f-21e9-4d20-af13-ed8a1fa71638-config-volume\") pod \"alertmanager-main-0\" (UID: \"a0321d1f-21e9-4d20-af13-ed8a1fa71638\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:30:22.100692 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:22.100596 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a0321d1f-21e9-4d20-af13-ed8a1fa71638-web-config\") pod \"alertmanager-main-0\" (UID: \"a0321d1f-21e9-4d20-af13-ed8a1fa71638\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:30:22.100692 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:22.100637 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a0321d1f-21e9-4d20-af13-ed8a1fa71638-config-out\") pod \"alertmanager-main-0\" (UID: \"a0321d1f-21e9-4d20-af13-ed8a1fa71638\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:30:22.100692 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:22.100656 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a0321d1f-21e9-4d20-af13-ed8a1fa71638-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a0321d1f-21e9-4d20-af13-ed8a1fa71638\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:30:22.100692 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:22.100690 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a0321d1f-21e9-4d20-af13-ed8a1fa71638-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a0321d1f-21e9-4d20-af13-ed8a1fa71638\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:30:22.101056 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:22.100714 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2gks\" (UniqueName: \"kubernetes.io/projected/a0321d1f-21e9-4d20-af13-ed8a1fa71638-kube-api-access-r2gks\") pod \"alertmanager-main-0\" (UID: \"a0321d1f-21e9-4d20-af13-ed8a1fa71638\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:30:22.101056 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:22.100750 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a0321d1f-21e9-4d20-af13-ed8a1fa71638-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a0321d1f-21e9-4d20-af13-ed8a1fa71638\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:30:22.101056 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:22.100782 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a0321d1f-21e9-4d20-af13-ed8a1fa71638-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a0321d1f-21e9-4d20-af13-ed8a1fa71638\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:30:22.101056 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:22.100807 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a0321d1f-21e9-4d20-af13-ed8a1fa71638-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a0321d1f-21e9-4d20-af13-ed8a1fa71638\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:30:22.101056 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:22.100833 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a0321d1f-21e9-4d20-af13-ed8a1fa71638-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a0321d1f-21e9-4d20-af13-ed8a1fa71638\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:30:22.101056 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:22.100881 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0321d1f-21e9-4d20-af13-ed8a1fa71638-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a0321d1f-21e9-4d20-af13-ed8a1fa71638\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:30:22.101667 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:22.101641 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0321d1f-21e9-4d20-af13-ed8a1fa71638-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a0321d1f-21e9-4d20-af13-ed8a1fa71638\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:30:22.101786 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:22.101715 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a0321d1f-21e9-4d20-af13-ed8a1fa71638-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a0321d1f-21e9-4d20-af13-ed8a1fa71638\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:30:22.101786 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:22.101740 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0321d1f-21e9-4d20-af13-ed8a1fa71638-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a0321d1f-21e9-4d20-af13-ed8a1fa71638\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:30:22.101786 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:22.101748 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a0321d1f-21e9-4d20-af13-ed8a1fa71638-config-volume\") pod \"alertmanager-main-0\" (UID: \"a0321d1f-21e9-4d20-af13-ed8a1fa71638\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:30:22.101985 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:22.101920 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a0321d1f-21e9-4d20-af13-ed8a1fa71638-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a0321d1f-21e9-4d20-af13-ed8a1fa71638\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:30:22.102755 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:22.102730 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0321d1f-21e9-4d20-af13-ed8a1fa71638-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a0321d1f-21e9-4d20-af13-ed8a1fa71638\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:30:22.103583 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:22.103541 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a0321d1f-21e9-4d20-af13-ed8a1fa71638-config-out\") pod \"alertmanager-main-0\" (UID: \"a0321d1f-21e9-4d20-af13-ed8a1fa71638\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:30:22.104151 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:22.104107 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a0321d1f-21e9-4d20-af13-ed8a1fa71638-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a0321d1f-21e9-4d20-af13-ed8a1fa71638\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:30:22.104246 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:22.104151 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a0321d1f-21e9-4d20-af13-ed8a1fa71638-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a0321d1f-21e9-4d20-af13-ed8a1fa71638\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:30:22.104246 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:22.104157 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a0321d1f-21e9-4d20-af13-ed8a1fa71638-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a0321d1f-21e9-4d20-af13-ed8a1fa71638\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:30:22.104808 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:22.104781 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a0321d1f-21e9-4d20-af13-ed8a1fa71638-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a0321d1f-21e9-4d20-af13-ed8a1fa71638\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:30:22.104910 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:22.104791 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a0321d1f-21e9-4d20-af13-ed8a1fa71638-web-config\") pod \"alertmanager-main-0\" (UID: \"a0321d1f-21e9-4d20-af13-ed8a1fa71638\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:30:22.105161 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:22.105142 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a0321d1f-21e9-4d20-af13-ed8a1fa71638-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a0321d1f-21e9-4d20-af13-ed8a1fa71638\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:30:22.105479 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:22.105448 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a0321d1f-21e9-4d20-af13-ed8a1fa71638-config-volume\") pod \"alertmanager-main-0\" (UID: \"a0321d1f-21e9-4d20-af13-ed8a1fa71638\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:30:22.105625 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:22.105610 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a0321d1f-21e9-4d20-af13-ed8a1fa71638-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a0321d1f-21e9-4d20-af13-ed8a1fa71638\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:30:22.109707 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:22.109685 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2gks\" (UniqueName: \"kubernetes.io/projected/a0321d1f-21e9-4d20-af13-ed8a1fa71638-kube-api-access-r2gks\") pod \"alertmanager-main-0\" (UID: \"a0321d1f-21e9-4d20-af13-ed8a1fa71638\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:30:22.265097 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:22.265058 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:30:22.408427 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:22.408396 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 08 19:30:22.412035 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:30:22.412004 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0321d1f_21e9_4d20_af13_ed8a1fa71638.slice/crio-f9f55aa9d6ef669d38174a02858cfcf2dca0643ce3460849992cdaf88725ef74 WatchSource:0}: Error finding container f9f55aa9d6ef669d38174a02858cfcf2dca0643ce3460849992cdaf88725ef74: Status 404 returned error can't find the container with id f9f55aa9d6ef669d38174a02858cfcf2dca0643ce3460849992cdaf88725ef74 Jul 08 19:30:22.623307 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:22.623275 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12079511-682f-45ae-b4c3-af244b463829" path="/var/lib/kubelet/pods/12079511-682f-45ae-b4c3-af244b463829/volumes" Jul 08 19:30:22.900357 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:22.900261 2566 generic.go:356] "Generic (PLEG): container finished" podID="a0321d1f-21e9-4d20-af13-ed8a1fa71638" containerID="924b589f77e90c1543c5b44c5d7eb6fc0233e40bed8e9fbb413941af739a68f0" exitCode=0 Jul 08 19:30:22.900814 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:22.900351 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a0321d1f-21e9-4d20-af13-ed8a1fa71638","Type":"ContainerDied","Data":"924b589f77e90c1543c5b44c5d7eb6fc0233e40bed8e9fbb413941af739a68f0"} Jul 08 19:30:22.900814 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:22.900442 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a0321d1f-21e9-4d20-af13-ed8a1fa71638","Type":"ContainerStarted","Data":"f9f55aa9d6ef669d38174a02858cfcf2dca0643ce3460849992cdaf88725ef74"} Jul 08 19:30:23.908606 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:23.908573 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a0321d1f-21e9-4d20-af13-ed8a1fa71638","Type":"ContainerStarted","Data":"06dde1620185302c9a21956d7dbfe36035d37809d45c1553a4792c457de69e4f"} Jul 08 19:30:23.909079 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:23.908635 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a0321d1f-21e9-4d20-af13-ed8a1fa71638","Type":"ContainerStarted","Data":"d8b46cc3d37cf7c08d74af60c69cb4477b9839447b062c624dda81ad5f791f55"} Jul 08 19:30:23.909079 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:23.908664 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a0321d1f-21e9-4d20-af13-ed8a1fa71638","Type":"ContainerStarted","Data":"225d694c1d33b2ce24e70783fd7c5e512c198221576e38c8460be5084b32ef13"} Jul 08 19:30:23.909079 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:23.908689 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a0321d1f-21e9-4d20-af13-ed8a1fa71638","Type":"ContainerStarted","Data":"7fb9ff11389f98961e44815757adc88dc0405e48e10c51f5071dbd7328a54180"} Jul 08 19:30:23.909079 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:23.908712 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a0321d1f-21e9-4d20-af13-ed8a1fa71638","Type":"ContainerStarted","Data":"041750bbdeeae5f0c3e94089177c8d7aff9ba801f6c9a93225ff5b469b5ee08e"} Jul 08 19:30:23.909079 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:23.908735 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a0321d1f-21e9-4d20-af13-ed8a1fa71638","Type":"ContainerStarted","Data":"47209d7e3aba357d24cae80f9a5cc57ebf3905cbe757c2322759a79f7f525476"} Jul 08 19:30:23.941011 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:23.940786 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.925379032 podStartE2EDuration="2.940764548s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.385516ms" isStatefulPod=false podCreationTimestamp="2026-07-08 19:30:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:30:23.938144392 +0000 UTC m=+389.824334439" watchObservedRunningTime="2026-07-08 19:30:23.940764548 +0000 UTC m=+389.826954596" Jul 08 19:30:27.044200 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:27.044165 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:30:27.146152 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:27.146118 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:30:27.245919 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:27.245889 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:30:27.434567 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:27.434529 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:30:27.494394 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:27.494361 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:30:32.494428 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:32.494389 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:30:32.595981 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:32.595940 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:30:32.696779 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:30:32.696747 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:31:09.547522 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:31:09.547487 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:31:09.743508 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:31:09.743481 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:31:12.320234 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:31:12.320203 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:31:12.417637 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:31:12.417606 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:31:14.713113 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:31:14.713082 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:31:14.773483 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:31:14.773457 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:31:44.145981 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:31:44.145893 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:31:44.935254 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:31:44.935224 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:31:47.568576 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:31:47.568539 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:31:47.671097 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:31:47.671052 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:31:48.516913 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:31:48.516884 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:31:48.620748 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:31:48.620718 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:31:48.721460 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:31:48.721434 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:31:50.703159 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:31:50.703123 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:31:51.107467 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:31:51.107440 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:31:51.672451 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:31:51.672423 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-hwp8p"] Jul 08 19:31:51.675368 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:31:51.675350 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-hwp8p" Jul 08 19:31:51.677832 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:31:51.677810 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 08 19:31:51.678059 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:31:51.678045 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 08 19:31:51.678800 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:31:51.678787 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-5xs9r\"" Jul 08 19:31:51.686025 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:31:51.686003 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-hwp8p"] Jul 08 19:31:51.784856 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:31:51.784829 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd5m4\" (UniqueName: \"kubernetes.io/projected/2cd2e7dc-d2f1-462a-85e8-9d07b3496a1e-kube-api-access-qd5m4\") pod \"cert-manager-webhook-6486ffd997-hwp8p\" (UID: \"2cd2e7dc-d2f1-462a-85e8-9d07b3496a1e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-hwp8p" Jul 08 19:31:51.785167 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:31:51.784877 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2cd2e7dc-d2f1-462a-85e8-9d07b3496a1e-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-hwp8p\" (UID: \"2cd2e7dc-d2f1-462a-85e8-9d07b3496a1e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-hwp8p" Jul 08 19:31:51.885235 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:31:51.885208 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd5m4\" (UniqueName: \"kubernetes.io/projected/2cd2e7dc-d2f1-462a-85e8-9d07b3496a1e-kube-api-access-qd5m4\") pod \"cert-manager-webhook-6486ffd997-hwp8p\" (UID: \"2cd2e7dc-d2f1-462a-85e8-9d07b3496a1e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-hwp8p" Jul 08 19:31:51.885235 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:31:51.885238 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2cd2e7dc-d2f1-462a-85e8-9d07b3496a1e-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-hwp8p\" (UID: \"2cd2e7dc-d2f1-462a-85e8-9d07b3496a1e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-hwp8p" Jul 08 19:31:51.894334 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:31:51.894302 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2cd2e7dc-d2f1-462a-85e8-9d07b3496a1e-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-hwp8p\" (UID: \"2cd2e7dc-d2f1-462a-85e8-9d07b3496a1e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-hwp8p" Jul 08 19:31:51.898152 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:31:51.898127 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd5m4\" (UniqueName: \"kubernetes.io/projected/2cd2e7dc-d2f1-462a-85e8-9d07b3496a1e-kube-api-access-qd5m4\") pod \"cert-manager-webhook-6486ffd997-hwp8p\" (UID: \"2cd2e7dc-d2f1-462a-85e8-9d07b3496a1e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-hwp8p" Jul 08 19:31:51.993111 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:31:51.993034 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-hwp8p" Jul 08 19:31:52.116263 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:31:52.116235 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-hwp8p"] Jul 08 19:31:52.118707 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:31:52.118674 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2cd2e7dc_d2f1_462a_85e8_9d07b3496a1e.slice/crio-582330c7ddb8c48f0832555fc4b3eb0e8ed4a26cb178a9601ecccf5a2dc7d4f3 WatchSource:0}: Error finding container 582330c7ddb8c48f0832555fc4b3eb0e8ed4a26cb178a9601ecccf5a2dc7d4f3: Status 404 returned error can't find the container with id 582330c7ddb8c48f0832555fc4b3eb0e8ed4a26cb178a9601ecccf5a2dc7d4f3 Jul 08 19:31:52.240213 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:31:52.240180 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-hwp8p" event={"ID":"2cd2e7dc-d2f1-462a-85e8-9d07b3496a1e","Type":"ContainerStarted","Data":"582330c7ddb8c48f0832555fc4b3eb0e8ed4a26cb178a9601ecccf5a2dc7d4f3"} Jul 08 19:31:54.155552 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:31:54.155524 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:31:54.565065 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:31:54.564984 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:31:55.259191 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:31:55.259151 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-hwp8p" event={"ID":"2cd2e7dc-d2f1-462a-85e8-9d07b3496a1e","Type":"ContainerStarted","Data":"888efec566262defb41d231e75694aa9b44af45d58da59b274c81788b08a31c9"} Jul 08 19:31:55.259543 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:31:55.259280 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-hwp8p" Jul 08 19:31:55.274649 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:31:55.274606 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-hwp8p" podStartSLOduration=1.309593115 podStartE2EDuration="4.274593231s" totalImagesPullingTime="2.965000116s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:31:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:31:55.273854366 +0000 UTC m=+481.160044414" watchObservedRunningTime="2026-07-08 19:31:55.274593231 +0000 UTC m=+481.160783277" Jul 08 19:31:56.597400 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:31:56.597363 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:31:56.702341 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:31:56.702316 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:31:58.503537 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:31:58.503505 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:01.237118 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:01.237085 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:01.265767 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:01.265743 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-hwp8p" Jul 08 19:32:01.336882 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:01.336840 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:07.099780 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:07.099749 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:07.253102 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:07.253070 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:09.125340 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:09.125308 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:23.415576 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:23.415540 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:23.478825 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:23.478788 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6bb9464976-444br"] Jul 08 19:32:23.481824 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:23.481810 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6bb9464976-444br" Jul 08 19:32:23.485106 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:23.485085 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 08 19:32:23.485106 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:23.485092 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 08 19:32:23.486152 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:23.486126 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 08 19:32:23.486261 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:23.486151 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-l7l9z\"" Jul 08 19:32:23.486261 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:23.486174 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 08 19:32:23.486261 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:23.486243 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 08 19:32:23.492169 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:23.492148 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6bb9464976-444br"] Jul 08 19:32:23.511248 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:23.511223 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:23.637726 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:23.637689 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8251066c-61ee-45a0-80b5-3ed028425362-metrics-cert\") pod \"lws-controller-manager-6bb9464976-444br\" (UID: \"8251066c-61ee-45a0-80b5-3ed028425362\") " pod="openshift-lws-operator/lws-controller-manager-6bb9464976-444br" Jul 08 19:32:23.637960 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:23.637801 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8251066c-61ee-45a0-80b5-3ed028425362-cert\") pod \"lws-controller-manager-6bb9464976-444br\" (UID: \"8251066c-61ee-45a0-80b5-3ed028425362\") " pod="openshift-lws-operator/lws-controller-manager-6bb9464976-444br" Jul 08 19:32:23.637960 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:23.637837 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6xr4\" (UniqueName: \"kubernetes.io/projected/8251066c-61ee-45a0-80b5-3ed028425362-kube-api-access-n6xr4\") pod \"lws-controller-manager-6bb9464976-444br\" (UID: \"8251066c-61ee-45a0-80b5-3ed028425362\") " pod="openshift-lws-operator/lws-controller-manager-6bb9464976-444br" Jul 08 19:32:23.637960 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:23.637891 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8251066c-61ee-45a0-80b5-3ed028425362-manager-config\") pod \"lws-controller-manager-6bb9464976-444br\" (UID: \"8251066c-61ee-45a0-80b5-3ed028425362\") " pod="openshift-lws-operator/lws-controller-manager-6bb9464976-444br" Jul 08 19:32:23.738754 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:23.738648 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8251066c-61ee-45a0-80b5-3ed028425362-cert\") pod \"lws-controller-manager-6bb9464976-444br\" (UID: \"8251066c-61ee-45a0-80b5-3ed028425362\") " pod="openshift-lws-operator/lws-controller-manager-6bb9464976-444br" Jul 08 19:32:23.738754 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:23.738685 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6xr4\" (UniqueName: \"kubernetes.io/projected/8251066c-61ee-45a0-80b5-3ed028425362-kube-api-access-n6xr4\") pod \"lws-controller-manager-6bb9464976-444br\" (UID: \"8251066c-61ee-45a0-80b5-3ed028425362\") " pod="openshift-lws-operator/lws-controller-manager-6bb9464976-444br" Jul 08 19:32:23.738754 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:23.738706 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8251066c-61ee-45a0-80b5-3ed028425362-manager-config\") pod \"lws-controller-manager-6bb9464976-444br\" (UID: \"8251066c-61ee-45a0-80b5-3ed028425362\") " pod="openshift-lws-operator/lws-controller-manager-6bb9464976-444br" Jul 08 19:32:23.739100 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:23.738834 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8251066c-61ee-45a0-80b5-3ed028425362-metrics-cert\") pod \"lws-controller-manager-6bb9464976-444br\" (UID: \"8251066c-61ee-45a0-80b5-3ed028425362\") " pod="openshift-lws-operator/lws-controller-manager-6bb9464976-444br" Jul 08 19:32:23.739473 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:23.739450 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8251066c-61ee-45a0-80b5-3ed028425362-manager-config\") pod \"lws-controller-manager-6bb9464976-444br\" (UID: \"8251066c-61ee-45a0-80b5-3ed028425362\") " pod="openshift-lws-operator/lws-controller-manager-6bb9464976-444br" Jul 08 19:32:23.741381 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:23.741352 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8251066c-61ee-45a0-80b5-3ed028425362-metrics-cert\") pod \"lws-controller-manager-6bb9464976-444br\" (UID: \"8251066c-61ee-45a0-80b5-3ed028425362\") " pod="openshift-lws-operator/lws-controller-manager-6bb9464976-444br" Jul 08 19:32:23.741469 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:23.741352 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8251066c-61ee-45a0-80b5-3ed028425362-cert\") pod \"lws-controller-manager-6bb9464976-444br\" (UID: \"8251066c-61ee-45a0-80b5-3ed028425362\") " pod="openshift-lws-operator/lws-controller-manager-6bb9464976-444br" Jul 08 19:32:23.746757 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:23.746736 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6xr4\" (UniqueName: \"kubernetes.io/projected/8251066c-61ee-45a0-80b5-3ed028425362-kube-api-access-n6xr4\") pod \"lws-controller-manager-6bb9464976-444br\" (UID: \"8251066c-61ee-45a0-80b5-3ed028425362\") " pod="openshift-lws-operator/lws-controller-manager-6bb9464976-444br" Jul 08 19:32:23.791729 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:23.791703 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6bb9464976-444br" Jul 08 19:32:23.935410 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:23.935381 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6bb9464976-444br"] Jul 08 19:32:23.936487 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:32:23.936463 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8251066c_61ee_45a0_80b5_3ed028425362.slice/crio-79eaf101f4e620da6cf88f98dfa0c2a1e51e8ba742ad9b1f13b5f3f546bb63ce WatchSource:0}: Error finding container 79eaf101f4e620da6cf88f98dfa0c2a1e51e8ba742ad9b1f13b5f3f546bb63ce: Status 404 returned error can't find the container with id 79eaf101f4e620da6cf88f98dfa0c2a1e51e8ba742ad9b1f13b5f3f546bb63ce Jul 08 19:32:24.376371 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:24.376341 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6bb9464976-444br" event={"ID":"8251066c-61ee-45a0-80b5-3ed028425362","Type":"ContainerStarted","Data":"79eaf101f4e620da6cf88f98dfa0c2a1e51e8ba742ad9b1f13b5f3f546bb63ce"} Jul 08 19:32:24.379421 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:24.379393 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:24.485226 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:24.485190 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:24.991298 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:24.991257 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:25.231732 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:25.231695 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:25.341055 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:25.340978 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:25.390619 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:25.390582 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-77ddc6fd87-m2dgt"] Jul 08 19:32:25.394159 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:25.394137 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-77ddc6fd87-m2dgt" Jul 08 19:32:25.397447 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:25.397090 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 08 19:32:25.397447 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:25.397183 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 08 19:32:25.397447 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:25.397168 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 08 19:32:25.397447 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:25.397214 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 08 19:32:25.397447 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:25.397259 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-g9lls\"" Jul 08 19:32:25.406102 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:25.406081 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-77ddc6fd87-m2dgt"] Jul 08 19:32:25.454331 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:25.454295 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/926adae9-5aea-4f33-8467-166b43bb6a0d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-77ddc6fd87-m2dgt\" (UID: \"926adae9-5aea-4f33-8467-166b43bb6a0d\") " pod="opendatahub/opendatahub-operator-controller-manager-77ddc6fd87-m2dgt" Jul 08 19:32:25.454331 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:25.454331 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/926adae9-5aea-4f33-8467-166b43bb6a0d-webhook-cert\") pod \"opendatahub-operator-controller-manager-77ddc6fd87-m2dgt\" (UID: \"926adae9-5aea-4f33-8467-166b43bb6a0d\") " pod="opendatahub/opendatahub-operator-controller-manager-77ddc6fd87-m2dgt" Jul 08 19:32:25.454512 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:25.454362 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82f6k\" (UniqueName: \"kubernetes.io/projected/926adae9-5aea-4f33-8467-166b43bb6a0d-kube-api-access-82f6k\") pod \"opendatahub-operator-controller-manager-77ddc6fd87-m2dgt\" (UID: \"926adae9-5aea-4f33-8467-166b43bb6a0d\") " pod="opendatahub/opendatahub-operator-controller-manager-77ddc6fd87-m2dgt" Jul 08 19:32:25.454512 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:25.454416 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/926adae9-5aea-4f33-8467-166b43bb6a0d-manifests\") pod \"opendatahub-operator-controller-manager-77ddc6fd87-m2dgt\" (UID: \"926adae9-5aea-4f33-8467-166b43bb6a0d\") " pod="opendatahub/opendatahub-operator-controller-manager-77ddc6fd87-m2dgt" Jul 08 19:32:25.454512 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:25.454454 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/926adae9-5aea-4f33-8467-166b43bb6a0d-tmp\") pod \"opendatahub-operator-controller-manager-77ddc6fd87-m2dgt\" (UID: \"926adae9-5aea-4f33-8467-166b43bb6a0d\") " pod="opendatahub/opendatahub-operator-controller-manager-77ddc6fd87-m2dgt" Jul 08 19:32:25.498741 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:25.498709 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:25.555727 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:25.555693 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/926adae9-5aea-4f33-8467-166b43bb6a0d-manifests\") pod \"opendatahub-operator-controller-manager-77ddc6fd87-m2dgt\" (UID: \"926adae9-5aea-4f33-8467-166b43bb6a0d\") " pod="opendatahub/opendatahub-operator-controller-manager-77ddc6fd87-m2dgt" Jul 08 19:32:25.555727 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:25.555726 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/926adae9-5aea-4f33-8467-166b43bb6a0d-tmp\") pod \"opendatahub-operator-controller-manager-77ddc6fd87-m2dgt\" (UID: \"926adae9-5aea-4f33-8467-166b43bb6a0d\") " pod="opendatahub/opendatahub-operator-controller-manager-77ddc6fd87-m2dgt" Jul 08 19:32:25.555941 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:25.555791 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/926adae9-5aea-4f33-8467-166b43bb6a0d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-77ddc6fd87-m2dgt\" (UID: \"926adae9-5aea-4f33-8467-166b43bb6a0d\") " pod="opendatahub/opendatahub-operator-controller-manager-77ddc6fd87-m2dgt" Jul 08 19:32:25.555941 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:25.555806 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/926adae9-5aea-4f33-8467-166b43bb6a0d-webhook-cert\") pod \"opendatahub-operator-controller-manager-77ddc6fd87-m2dgt\" (UID: \"926adae9-5aea-4f33-8467-166b43bb6a0d\") " pod="opendatahub/opendatahub-operator-controller-manager-77ddc6fd87-m2dgt" Jul 08 19:32:25.555941 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:25.555846 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82f6k\" (UniqueName: \"kubernetes.io/projected/926adae9-5aea-4f33-8467-166b43bb6a0d-kube-api-access-82f6k\") pod \"opendatahub-operator-controller-manager-77ddc6fd87-m2dgt\" (UID: \"926adae9-5aea-4f33-8467-166b43bb6a0d\") " pod="opendatahub/opendatahub-operator-controller-manager-77ddc6fd87-m2dgt" Jul 08 19:32:25.556207 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:25.556168 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/926adae9-5aea-4f33-8467-166b43bb6a0d-manifests\") pod \"opendatahub-operator-controller-manager-77ddc6fd87-m2dgt\" (UID: \"926adae9-5aea-4f33-8467-166b43bb6a0d\") " pod="opendatahub/opendatahub-operator-controller-manager-77ddc6fd87-m2dgt" Jul 08 19:32:25.558253 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:25.558217 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/926adae9-5aea-4f33-8467-166b43bb6a0d-tmp\") pod \"opendatahub-operator-controller-manager-77ddc6fd87-m2dgt\" (UID: \"926adae9-5aea-4f33-8467-166b43bb6a0d\") " pod="opendatahub/opendatahub-operator-controller-manager-77ddc6fd87-m2dgt" Jul 08 19:32:25.558401 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:25.558384 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/926adae9-5aea-4f33-8467-166b43bb6a0d-webhook-cert\") pod \"opendatahub-operator-controller-manager-77ddc6fd87-m2dgt\" (UID: \"926adae9-5aea-4f33-8467-166b43bb6a0d\") " pod="opendatahub/opendatahub-operator-controller-manager-77ddc6fd87-m2dgt" Jul 08 19:32:25.558448 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:25.558414 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/926adae9-5aea-4f33-8467-166b43bb6a0d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-77ddc6fd87-m2dgt\" (UID: \"926adae9-5aea-4f33-8467-166b43bb6a0d\") " pod="opendatahub/opendatahub-operator-controller-manager-77ddc6fd87-m2dgt" Jul 08 19:32:25.567100 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:25.567076 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82f6k\" (UniqueName: \"kubernetes.io/projected/926adae9-5aea-4f33-8467-166b43bb6a0d-kube-api-access-82f6k\") pod \"opendatahub-operator-controller-manager-77ddc6fd87-m2dgt\" (UID: \"926adae9-5aea-4f33-8467-166b43bb6a0d\") " pod="opendatahub/opendatahub-operator-controller-manager-77ddc6fd87-m2dgt" Jul 08 19:32:25.593835 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:25.593778 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:25.696061 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:25.696023 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:25.708037 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:25.708012 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-77ddc6fd87-m2dgt" Jul 08 19:32:25.796142 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:25.796112 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:25.864451 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:25.864377 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-77ddc6fd87-m2dgt"] Jul 08 19:32:25.867947 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:32:25.867913 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod926adae9_5aea_4f33_8467_166b43bb6a0d.slice/crio-188abe1f95ff37e75e5259dfb9da27e6b025f93e8defa11abe2210fcecd2988c WatchSource:0}: Error finding container 188abe1f95ff37e75e5259dfb9da27e6b025f93e8defa11abe2210fcecd2988c: Status 404 returned error can't find the container with id 188abe1f95ff37e75e5259dfb9da27e6b025f93e8defa11abe2210fcecd2988c Jul 08 19:32:25.900238 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:25.900209 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:26.015265 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:26.015234 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:26.104632 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:26.104600 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:26.197488 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:26.197455 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:26.387465 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:26.387434 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-77ddc6fd87-m2dgt" event={"ID":"926adae9-5aea-4f33-8467-166b43bb6a0d","Type":"ContainerStarted","Data":"188abe1f95ff37e75e5259dfb9da27e6b025f93e8defa11abe2210fcecd2988c"} Jul 08 19:32:29.404302 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:29.404270 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-77ddc6fd87-m2dgt" event={"ID":"926adae9-5aea-4f33-8467-166b43bb6a0d","Type":"ContainerStarted","Data":"95b10bcecdc4758cf03a8657ca1b40972d5650601baa069d6fdbbdb8c5cb5fb3"} Jul 08 19:32:30.410102 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:30.410070 2566 generic.go:356] "Generic (PLEG): container finished" podID="926adae9-5aea-4f33-8467-166b43bb6a0d" containerID="95b10bcecdc4758cf03a8657ca1b40972d5650601baa069d6fdbbdb8c5cb5fb3" exitCode=0 Jul 08 19:32:30.410541 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:30.410155 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-77ddc6fd87-m2dgt" event={"ID":"926adae9-5aea-4f33-8467-166b43bb6a0d","Type":"ContainerDied","Data":"95b10bcecdc4758cf03a8657ca1b40972d5650601baa069d6fdbbdb8c5cb5fb3"} Jul 08 19:32:31.416552 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:31.416511 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-77ddc6fd87-m2dgt" event={"ID":"926adae9-5aea-4f33-8467-166b43bb6a0d","Type":"ContainerStarted","Data":"9f2f923ca624427407bc5cbc7a86a114ebda5edac688ce8521c74ac95b94241e"} Jul 08 19:32:31.421936 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:31.417143 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-77ddc6fd87-m2dgt" Jul 08 19:32:31.441504 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:31.441446 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-77ddc6fd87-m2dgt" podStartSLOduration=1.8475708370000001 podStartE2EDuration="6.441430755s" totalImagesPullingTime="3.581418112s" totalInitContainerRuntime="1.012441806s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:32:25 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:32:31.439782121 +0000 UTC m=+517.325972168" watchObservedRunningTime="2026-07-08 19:32:31.441430755 +0000 UTC m=+517.327620801" Jul 08 19:32:39.458022 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:39.457986 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6bb9464976-444br" event={"ID":"8251066c-61ee-45a0-80b5-3ed028425362","Type":"ContainerStarted","Data":"77f828fb5ed3672cd56bcf89013624991e1e83889d9e25ca63bb9e80bb99173a"} Jul 08 19:32:39.458442 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:39.458101 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6bb9464976-444br" Jul 08 19:32:39.475433 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:39.475383 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6bb9464976-444br" podStartSLOduration=1.745967041 podStartE2EDuration="16.475367384s" totalImagesPullingTime="14.729400343s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:32:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:32:39.473789985 +0000 UTC m=+525.359980036" watchObservedRunningTime="2026-07-08 19:32:39.475367384 +0000 UTC m=+525.361557431" Jul 08 19:32:42.423067 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:42.423035 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-77ddc6fd87-m2dgt" Jul 08 19:32:44.986218 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:44.986189 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:45.129587 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:45.129552 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:45.360530 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:45.360454 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:45.460461 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:45.460434 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:45.721527 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:45.721490 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:45.821784 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:45.821758 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:45.926991 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:45.926966 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:46.648016 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:46.647990 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:46.708155 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:46.708130 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-5kkcw"] Jul 08 19:32:46.720213 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:46.720194 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-5kkcw" Jul 08 19:32:46.723045 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:46.723027 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 08 19:32:46.724027 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:46.724005 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-pc8wp\"" Jul 08 19:32:46.734083 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:46.734064 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-5kkcw"] Jul 08 19:32:46.751276 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:46.751259 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:46.841926 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:46.841895 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8spv\" (UniqueName: \"kubernetes.io/projected/8080820c-03c1-4be8-8b2c-f8bbf00d5182-kube-api-access-w8spv\") pod \"model-serving-api-54d89bd5fc-5kkcw\" (UID: \"8080820c-03c1-4be8-8b2c-f8bbf00d5182\") " pod="opendatahub/model-serving-api-54d89bd5fc-5kkcw" Jul 08 19:32:46.842096 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:46.841958 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8080820c-03c1-4be8-8b2c-f8bbf00d5182-tls-certs\") pod \"model-serving-api-54d89bd5fc-5kkcw\" (UID: \"8080820c-03c1-4be8-8b2c-f8bbf00d5182\") " pod="opendatahub/model-serving-api-54d89bd5fc-5kkcw" Jul 08 19:32:46.867475 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:46.867443 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:46.943097 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:46.943070 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8080820c-03c1-4be8-8b2c-f8bbf00d5182-tls-certs\") pod \"model-serving-api-54d89bd5fc-5kkcw\" (UID: \"8080820c-03c1-4be8-8b2c-f8bbf00d5182\") " pod="opendatahub/model-serving-api-54d89bd5fc-5kkcw" Jul 08 19:32:46.943238 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:46.943124 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8spv\" (UniqueName: \"kubernetes.io/projected/8080820c-03c1-4be8-8b2c-f8bbf00d5182-kube-api-access-w8spv\") pod \"model-serving-api-54d89bd5fc-5kkcw\" (UID: \"8080820c-03c1-4be8-8b2c-f8bbf00d5182\") " pod="opendatahub/model-serving-api-54d89bd5fc-5kkcw" Jul 08 19:32:46.945493 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:46.945465 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8080820c-03c1-4be8-8b2c-f8bbf00d5182-tls-certs\") pod \"model-serving-api-54d89bd5fc-5kkcw\" (UID: \"8080820c-03c1-4be8-8b2c-f8bbf00d5182\") " pod="opendatahub/model-serving-api-54d89bd5fc-5kkcw" Jul 08 19:32:46.952019 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:46.951993 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8spv\" (UniqueName: \"kubernetes.io/projected/8080820c-03c1-4be8-8b2c-f8bbf00d5182-kube-api-access-w8spv\") pod \"model-serving-api-54d89bd5fc-5kkcw\" (UID: \"8080820c-03c1-4be8-8b2c-f8bbf00d5182\") " pod="opendatahub/model-serving-api-54d89bd5fc-5kkcw" Jul 08 19:32:47.030703 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:47.030669 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-5kkcw" Jul 08 19:32:47.160924 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:47.160893 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-5kkcw"] Jul 08 19:32:47.165208 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:32:47.165174 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8080820c_03c1_4be8_8b2c_f8bbf00d5182.slice/crio-9bae36af2579eec1a7d1948ef6d3c1849c1b3388d9c62bd17a4e456d7b46804d WatchSource:0}: Error finding container 9bae36af2579eec1a7d1948ef6d3c1849c1b3388d9c62bd17a4e456d7b46804d: Status 404 returned error can't find the container with id 9bae36af2579eec1a7d1948ef6d3c1849c1b3388d9c62bd17a4e456d7b46804d Jul 08 19:32:47.492468 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:47.492433 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-5kkcw" event={"ID":"8080820c-03c1-4be8-8b2c-f8bbf00d5182","Type":"ContainerStarted","Data":"9bae36af2579eec1a7d1948ef6d3c1849c1b3388d9c62bd17a4e456d7b46804d"} Jul 08 19:32:47.801211 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:47.801129 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:49.502329 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:49.502295 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-5kkcw" event={"ID":"8080820c-03c1-4be8-8b2c-f8bbf00d5182","Type":"ContainerStarted","Data":"a6458ed53f3ddc8f624904719ba65a1f754a32b4f060c3f489473bdb0b25680f"} Jul 08 19:32:49.502729 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:49.502383 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-5kkcw" Jul 08 19:32:49.520358 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:49.520307 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-5kkcw" podStartSLOduration=1.413197729 podStartE2EDuration="3.520290967s" totalImagesPullingTime="2.107093238s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:32:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:32:49.519125994 +0000 UTC m=+535.405316042" watchObservedRunningTime="2026-07-08 19:32:49.520290967 +0000 UTC m=+535.406481014" Jul 08 19:32:50.464230 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:50.464201 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6bb9464976-444br" Jul 08 19:32:53.898496 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:53.898463 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j57db"] Jul 08 19:32:53.901758 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:53.901739 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j57db" Jul 08 19:32:53.904564 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:53.904540 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 08 19:32:53.904646 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:53.904561 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 08 19:32:53.904801 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:53.904786 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-6cs98\"" Jul 08 19:32:53.904894 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:53.904786 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 08 19:32:53.907529 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:53.907514 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 08 19:32:53.919574 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:53.919551 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j57db"] Jul 08 19:32:53.955490 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:53.955466 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:54.004590 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:54.004561 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/13b2effd-726c-4425-aff2-45d8df9443ef-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-j57db\" (UID: \"13b2effd-726c-4425-aff2-45d8df9443ef\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j57db" Jul 08 19:32:54.004590 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:54.004593 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/13b2effd-726c-4425-aff2-45d8df9443ef-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-j57db\" (UID: \"13b2effd-726c-4425-aff2-45d8df9443ef\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j57db" Jul 08 19:32:54.004742 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:54.004611 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/13b2effd-726c-4425-aff2-45d8df9443ef-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-j57db\" (UID: \"13b2effd-726c-4425-aff2-45d8df9443ef\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j57db" Jul 08 19:32:54.004742 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:54.004651 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws9qf\" (UniqueName: \"kubernetes.io/projected/13b2effd-726c-4425-aff2-45d8df9443ef-kube-api-access-ws9qf\") pod \"istiod-openshift-gateway-94bb8fbfd-j57db\" (UID: \"13b2effd-726c-4425-aff2-45d8df9443ef\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j57db" Jul 08 19:32:54.004742 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:54.004675 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/13b2effd-726c-4425-aff2-45d8df9443ef-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-j57db\" (UID: \"13b2effd-726c-4425-aff2-45d8df9443ef\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j57db" Jul 08 19:32:54.004910 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:54.004760 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13b2effd-726c-4425-aff2-45d8df9443ef-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-j57db\" (UID: \"13b2effd-726c-4425-aff2-45d8df9443ef\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j57db" Jul 08 19:32:54.004910 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:54.004789 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/13b2effd-726c-4425-aff2-45d8df9443ef-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-j57db\" (UID: \"13b2effd-726c-4425-aff2-45d8df9443ef\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j57db" Jul 08 19:32:54.035851 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:54.035824 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:54.105711 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:54.105679 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13b2effd-726c-4425-aff2-45d8df9443ef-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-j57db\" (UID: \"13b2effd-726c-4425-aff2-45d8df9443ef\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j57db" Jul 08 19:32:54.105886 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:54.105718 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/13b2effd-726c-4425-aff2-45d8df9443ef-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-j57db\" (UID: \"13b2effd-726c-4425-aff2-45d8df9443ef\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j57db" Jul 08 19:32:54.105886 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:54.105753 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/13b2effd-726c-4425-aff2-45d8df9443ef-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-j57db\" (UID: \"13b2effd-726c-4425-aff2-45d8df9443ef\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j57db" Jul 08 19:32:54.105886 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:54.105773 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/13b2effd-726c-4425-aff2-45d8df9443ef-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-j57db\" (UID: \"13b2effd-726c-4425-aff2-45d8df9443ef\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j57db" Jul 08 19:32:54.105886 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:54.105793 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/13b2effd-726c-4425-aff2-45d8df9443ef-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-j57db\" (UID: \"13b2effd-726c-4425-aff2-45d8df9443ef\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j57db" Jul 08 19:32:54.105886 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:54.105814 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws9qf\" (UniqueName: \"kubernetes.io/projected/13b2effd-726c-4425-aff2-45d8df9443ef-kube-api-access-ws9qf\") pod \"istiod-openshift-gateway-94bb8fbfd-j57db\" (UID: \"13b2effd-726c-4425-aff2-45d8df9443ef\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j57db" Jul 08 19:32:54.105886 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:54.105832 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/13b2effd-726c-4425-aff2-45d8df9443ef-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-j57db\" (UID: \"13b2effd-726c-4425-aff2-45d8df9443ef\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j57db" Jul 08 19:32:54.106584 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:54.106549 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/13b2effd-726c-4425-aff2-45d8df9443ef-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-j57db\" (UID: \"13b2effd-726c-4425-aff2-45d8df9443ef\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j57db" Jul 08 19:32:54.108307 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:54.108282 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/13b2effd-726c-4425-aff2-45d8df9443ef-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-j57db\" (UID: \"13b2effd-726c-4425-aff2-45d8df9443ef\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j57db" Jul 08 19:32:54.108413 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:54.108287 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/13b2effd-726c-4425-aff2-45d8df9443ef-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-j57db\" (UID: \"13b2effd-726c-4425-aff2-45d8df9443ef\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j57db" Jul 08 19:32:54.108413 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:54.108375 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/13b2effd-726c-4425-aff2-45d8df9443ef-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-j57db\" (UID: \"13b2effd-726c-4425-aff2-45d8df9443ef\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j57db" Jul 08 19:32:54.108525 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:54.108445 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/13b2effd-726c-4425-aff2-45d8df9443ef-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-j57db\" (UID: \"13b2effd-726c-4425-aff2-45d8df9443ef\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j57db" Jul 08 19:32:54.113187 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:54.113166 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13b2effd-726c-4425-aff2-45d8df9443ef-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-j57db\" (UID: \"13b2effd-726c-4425-aff2-45d8df9443ef\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j57db" Jul 08 19:32:54.113728 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:54.113706 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws9qf\" (UniqueName: \"kubernetes.io/projected/13b2effd-726c-4425-aff2-45d8df9443ef-kube-api-access-ws9qf\") pod \"istiod-openshift-gateway-94bb8fbfd-j57db\" (UID: \"13b2effd-726c-4425-aff2-45d8df9443ef\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j57db" Jul 08 19:32:54.210847 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:54.210821 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j57db" Jul 08 19:32:54.352449 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:54.352417 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j57db"] Jul 08 19:32:54.354906 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:32:54.354844 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13b2effd_726c_4425_aff2_45d8df9443ef.slice/crio-d6de628152561e78a24122bb2d046615cc406048f479a8bd9a3280b786a1e7f5 WatchSource:0}: Error finding container d6de628152561e78a24122bb2d046615cc406048f479a8bd9a3280b786a1e7f5: Status 404 returned error can't find the container with id d6de628152561e78a24122bb2d046615cc406048f479a8bd9a3280b786a1e7f5 Jul 08 19:32:54.403992 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:54.403965 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:54.526429 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:54.526355 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j57db" event={"ID":"13b2effd-726c-4425-aff2-45d8df9443ef","Type":"ContainerStarted","Data":"d6de628152561e78a24122bb2d046615cc406048f479a8bd9a3280b786a1e7f5"} Jul 08 19:32:54.653095 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:54.653068 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:54.753834 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:54.753806 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:56.624752 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:56.624719 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 08 19:32:56.625017 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:56.624783 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 08 19:32:56.729023 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:56.728971 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:56.819073 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:56.819035 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:56.887100 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:56.887024 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:57.012065 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:57.012031 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:57.542622 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:57.542583 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j57db" event={"ID":"13b2effd-726c-4425-aff2-45d8df9443ef","Type":"ContainerStarted","Data":"5eaccfe1f200db593a2cab1a3d3bd7e1c26ae449243dddf7508c03c20da5e7f3"} Jul 08 19:32:57.542806 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:57.542766 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j57db" Jul 08 19:32:57.544455 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:57.544428 2566 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-j57db container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 08 19:32:57.544561 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:57.544502 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j57db" podUID="13b2effd-726c-4425-aff2-45d8df9443ef" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 08 19:32:57.577873 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:57.577817 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j57db" podStartSLOduration=2.310273005 podStartE2EDuration="4.577802472s" totalImagesPullingTime="2.267529467s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:32:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:32:57.575415288 +0000 UTC m=+543.461605507" watchObservedRunningTime="2026-07-08 19:32:57.577802472 +0000 UTC m=+543.463992518" Jul 08 19:32:58.056199 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:58.056168 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:58.156966 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:58.156933 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:58.548046 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:58.548022 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j57db" Jul 08 19:32:59.103343 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:59.103306 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:59.345519 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:32:59.345491 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:33:00.511536 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:33:00.511509 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-5kkcw" Jul 08 19:33:02.223376 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:33:02.223344 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:33:02.323967 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:33:02.323940 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:33:04.083934 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:33:04.083893 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:33:04.158579 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:33:04.158542 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:33:38.711685 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:33:38.711642 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:33:46.564738 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:33:46.564685 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:33:46.656673 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:33:46.656643 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:33:46.777775 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:33:46.777736 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:34:21.140667 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:34:21.140573 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:34:22.910255 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:34:22.910223 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:34:26.351017 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:34:26.350987 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:34:26.653092 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:34:26.653067 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:34:28.976138 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:34:28.976105 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:34:29.092823 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:34:29.092796 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:34:29.187534 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:34:29.187499 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:34:30.982688 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:34:30.982659 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:34:32.402032 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:34:32.401998 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:34:33.955044 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:34:33.955015 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:34:34.115856 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:34:34.115825 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:34:37.259128 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:34:37.259098 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:34:37.359748 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:34:37.359716 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:34:37.459137 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:34:37.459100 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:34:37.559304 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:34:37.559238 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:34:37.659846 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:34:37.659819 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:34:38.557970 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:34:38.557935 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:34:38.653139 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:34:38.653114 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:34:40.954215 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:34:40.954190 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:34:41.020434 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:34:41.020387 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:34:43.069146 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:34:43.069115 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:34:43.261066 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:34:43.261035 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:34:46.775035 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:34:46.775000 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:34:46.875276 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:34:46.875243 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:34:50.062259 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:34:50.062227 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:34:54.577795 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:34:54.577765 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:34:54.690741 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:34:54.690710 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:34:57.154984 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:34:57.154950 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:34:57.254247 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:34:57.254213 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:34:57.355390 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:34:57.355361 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:34:57.456555 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:34:57.456490 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:35:01.156358 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:35:01.156328 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:35:01.255622 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:35:01.255594 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:35:01.414885 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:35:01.414797 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:35:01.497910 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:35:01.497885 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:35:06.559294 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:35:06.559261 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:35:06.659369 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:35:06.659338 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:35:14.360606 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:35:14.360538 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:35:14.461052 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:35:14.461026 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:35:40.612433 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:35:40.612394 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:35:40.685301 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:35:40.685268 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:35:42.599958 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:35:42.599924 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:35:42.741256 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:35:42.741226 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:35:42.841813 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:35:42.841770 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:35:42.914112 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:35:42.914088 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:35:42.917550 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:35:42.917533 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-9vx6x" Jul 08 19:35:42.920368 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:35:42.920343 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 08 19:35:42.920368 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:35:42.920363 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-78nk4\"" Jul 08 19:35:42.920508 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:35:42.920408 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 08 19:35:42.921064 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:35:42.921051 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 08 19:35:42.925653 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:35:42.925632 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:35:42.946940 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:35:42.946914 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgv6r\" (UniqueName: \"kubernetes.io/projected/4a97688b-b2d6-4c40-ad27-ada055b7f320-kube-api-access-fgv6r\") pod \"limitador-limitador-54d6d988d5-9vx6x\" (UID: \"4a97688b-b2d6-4c40-ad27-ada055b7f320\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-9vx6x" Jul 08 19:35:42.947022 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:35:42.946979 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4a97688b-b2d6-4c40-ad27-ada055b7f320-config-file\") pod \"limitador-limitador-54d6d988d5-9vx6x\" (UID: \"4a97688b-b2d6-4c40-ad27-ada055b7f320\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-9vx6x" Jul 08 19:35:42.952747 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:35:42.952729 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:35:43.048270 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:35:43.048247 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgv6r\" (UniqueName: \"kubernetes.io/projected/4a97688b-b2d6-4c40-ad27-ada055b7f320-kube-api-access-fgv6r\") pod \"limitador-limitador-54d6d988d5-9vx6x\" (UID: \"4a97688b-b2d6-4c40-ad27-ada055b7f320\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-9vx6x" Jul 08 19:35:43.048346 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:35:43.048283 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4a97688b-b2d6-4c40-ad27-ada055b7f320-config-file\") pod \"limitador-limitador-54d6d988d5-9vx6x\" (UID: \"4a97688b-b2d6-4c40-ad27-ada055b7f320\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-9vx6x" Jul 08 19:35:43.048808 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:35:43.048793 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4a97688b-b2d6-4c40-ad27-ada055b7f320-config-file\") pod \"limitador-limitador-54d6d988d5-9vx6x\" (UID: \"4a97688b-b2d6-4c40-ad27-ada055b7f320\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-9vx6x" Jul 08 19:35:43.059974 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:35:43.059947 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgv6r\" (UniqueName: \"kubernetes.io/projected/4a97688b-b2d6-4c40-ad27-ada055b7f320-kube-api-access-fgv6r\") pod \"limitador-limitador-54d6d988d5-9vx6x\" (UID: \"4a97688b-b2d6-4c40-ad27-ada055b7f320\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-9vx6x" Jul 08 19:35:43.228988 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:35:43.228916 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-9vx6x" Jul 08 19:35:43.353881 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:35:43.353845 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:35:43.355884 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:35:43.355848 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a97688b_b2d6_4c40_ad27_ada055b7f320.slice/crio-6a15a6e1587494b6fe3cfe2012365ae2980a27ffb8cfe307dd18f6a9013a221b WatchSource:0}: Error finding container 6a15a6e1587494b6fe3cfe2012365ae2980a27ffb8cfe307dd18f6a9013a221b: Status 404 returned error can't find the container with id 6a15a6e1587494b6fe3cfe2012365ae2980a27ffb8cfe307dd18f6a9013a221b Jul 08 19:35:43.357924 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:35:43.357903 2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 19:35:43.426011 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:35:43.425990 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:35:43.530683 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:35:43.530616 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:35:44.253436 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:35:44.253398 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-9vx6x" event={"ID":"4a97688b-b2d6-4c40-ad27-ada055b7f320","Type":"ContainerStarted","Data":"6a15a6e1587494b6fe3cfe2012365ae2980a27ffb8cfe307dd18f6a9013a221b"} Jul 08 19:35:46.264890 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:35:46.264833 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-9vx6x" event={"ID":"4a97688b-b2d6-4c40-ad27-ada055b7f320","Type":"ContainerStarted","Data":"c674cfefb5f7f5d104cbf6e104e142e7d3ca34002854467f1fb0f35a727d110a"} Jul 08 19:35:46.265259 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:35:46.264925 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-9vx6x" Jul 08 19:35:46.283135 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:35:46.283084 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-9vx6x" podStartSLOduration=1.798036578 podStartE2EDuration="4.283069786s" totalImagesPullingTime="2.485033208s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:35:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:35:46.282226611 +0000 UTC m=+712.168416659" watchObservedRunningTime="2026-07-08 19:35:46.283069786 +0000 UTC m=+712.169259832" Jul 08 19:35:57.271907 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:35:57.271854 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-9vx6x" Jul 08 19:35:59.088672 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:35:59.088643 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:36:04.821329 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:36:04.821295 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:36:10.690440 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:36:10.690414 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:36:10.920216 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:36:10.920185 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:36:17.199099 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:36:17.199060 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:36:17.271681 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:36:17.271645 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:36:17.335566 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:36:17.335541 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:36:17.770402 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:36:17.770373 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:36:20.852581 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:36:20.852548 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:36:20.952370 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:36:20.952340 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:36:52.281661 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:36:52.281629 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:36:52.653731 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:36:52.653689 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:36:53.031268 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:36:53.031193 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:36:53.124657 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:36:53.124631 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:36:53.217973 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:36:53.217948 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:36:53.218768 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:36:53.218749 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:37:03.663960 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:37:03.663905 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:37:03.775163 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:37:03.775119 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:37:17.923385 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:37:17.923284 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:37:18.409835 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:37:18.409804 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:37:29.115660 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:37:29.115616 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:37:29.780080 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:37:29.780045 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:37:34.014078 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:37:34.014045 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:37:35.096959 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:37:35.096915 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:37:43.216418 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:37:43.216369 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:37:44.790525 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:37:44.790494 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:38:02.567931 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:38:02.567891 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-zs5q6"] Jul 08 19:38:02.571220 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:38:02.571200 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zs5q6" Jul 08 19:38:02.574118 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:38:02.574089 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 08 19:38:02.574824 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:38:02.574803 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 08 19:38:02.574946 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:38:02.574823 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-m5fhc\"" Jul 08 19:38:02.574946 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:38:02.574843 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 08 19:38:02.581583 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:38:02.581563 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-zs5q6"] Jul 08 19:38:02.610285 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:38:02.610257 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:38:02.661470 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:38:02.661445 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3854bf68-0ce0-401b-9aa7-8bc92c2ccf89-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zs5q6\" (UID: \"3854bf68-0ce0-401b-9aa7-8bc92c2ccf89\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zs5q6" Jul 08 19:38:02.661587 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:38:02.661477 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv5sw\" (UniqueName: \"kubernetes.io/projected/3854bf68-0ce0-401b-9aa7-8bc92c2ccf89-kube-api-access-tv5sw\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zs5q6\" (UID: \"3854bf68-0ce0-401b-9aa7-8bc92c2ccf89\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zs5q6" Jul 08 19:38:02.661587 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:38:02.661525 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3854bf68-0ce0-401b-9aa7-8bc92c2ccf89-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zs5q6\" (UID: \"3854bf68-0ce0-401b-9aa7-8bc92c2ccf89\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zs5q6" Jul 08 19:38:02.661587 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:38:02.661579 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3854bf68-0ce0-401b-9aa7-8bc92c2ccf89-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zs5q6\" (UID: \"3854bf68-0ce0-401b-9aa7-8bc92c2ccf89\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zs5q6" Jul 08 19:38:02.661683 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:38:02.661608 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3854bf68-0ce0-401b-9aa7-8bc92c2ccf89-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zs5q6\" (UID: \"3854bf68-0ce0-401b-9aa7-8bc92c2ccf89\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zs5q6" Jul 08 19:38:02.661683 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:38:02.661622 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3854bf68-0ce0-401b-9aa7-8bc92c2ccf89-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zs5q6\" (UID: \"3854bf68-0ce0-401b-9aa7-8bc92c2ccf89\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zs5q6" Jul 08 19:38:02.762443 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:38:02.762400 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3854bf68-0ce0-401b-9aa7-8bc92c2ccf89-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zs5q6\" (UID: \"3854bf68-0ce0-401b-9aa7-8bc92c2ccf89\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zs5q6" Jul 08 19:38:02.762656 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:38:02.762465 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3854bf68-0ce0-401b-9aa7-8bc92c2ccf89-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zs5q6\" (UID: \"3854bf68-0ce0-401b-9aa7-8bc92c2ccf89\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zs5q6" Jul 08 19:38:02.762656 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:38:02.762487 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3854bf68-0ce0-401b-9aa7-8bc92c2ccf89-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zs5q6\" (UID: \"3854bf68-0ce0-401b-9aa7-8bc92c2ccf89\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zs5q6" Jul 08 19:38:02.762656 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:38:02.762510 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3854bf68-0ce0-401b-9aa7-8bc92c2ccf89-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zs5q6\" (UID: \"3854bf68-0ce0-401b-9aa7-8bc92c2ccf89\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zs5q6" Jul 08 19:38:02.762656 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:38:02.762571 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3854bf68-0ce0-401b-9aa7-8bc92c2ccf89-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zs5q6\" (UID: \"3854bf68-0ce0-401b-9aa7-8bc92c2ccf89\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zs5q6" Jul 08 19:38:02.762656 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:38:02.762608 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv5sw\" (UniqueName: \"kubernetes.io/projected/3854bf68-0ce0-401b-9aa7-8bc92c2ccf89-kube-api-access-tv5sw\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zs5q6\" (UID: \"3854bf68-0ce0-401b-9aa7-8bc92c2ccf89\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zs5q6" Jul 08 19:38:02.762949 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:38:02.762926 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3854bf68-0ce0-401b-9aa7-8bc92c2ccf89-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zs5q6\" (UID: \"3854bf68-0ce0-401b-9aa7-8bc92c2ccf89\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zs5q6" Jul 08 19:38:02.763004 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:38:02.762967 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3854bf68-0ce0-401b-9aa7-8bc92c2ccf89-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zs5q6\" (UID: \"3854bf68-0ce0-401b-9aa7-8bc92c2ccf89\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zs5q6" Jul 08 19:38:02.763004 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:38:02.762998 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3854bf68-0ce0-401b-9aa7-8bc92c2ccf89-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zs5q6\" (UID: \"3854bf68-0ce0-401b-9aa7-8bc92c2ccf89\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zs5q6" Jul 08 19:38:02.764676 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:38:02.764647 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3854bf68-0ce0-401b-9aa7-8bc92c2ccf89-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zs5q6\" (UID: \"3854bf68-0ce0-401b-9aa7-8bc92c2ccf89\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zs5q6" Jul 08 19:38:02.764974 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:38:02.764959 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3854bf68-0ce0-401b-9aa7-8bc92c2ccf89-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zs5q6\" (UID: \"3854bf68-0ce0-401b-9aa7-8bc92c2ccf89\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zs5q6" Jul 08 19:38:02.771699 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:38:02.771673 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv5sw\" (UniqueName: \"kubernetes.io/projected/3854bf68-0ce0-401b-9aa7-8bc92c2ccf89-kube-api-access-tv5sw\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zs5q6\" (UID: \"3854bf68-0ce0-401b-9aa7-8bc92c2ccf89\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zs5q6" Jul 08 19:38:02.883167 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:38:02.883085 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zs5q6" Jul 08 19:38:03.010470 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:38:03.010295 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-zs5q6"] Jul 08 19:38:03.013196 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:38:03.013172 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3854bf68_0ce0_401b_9aa7_8bc92c2ccf89.slice/crio-ba053e3d905a4aa217457f877f940036a88e278a2f1982d12ce4f66eb05d211f WatchSource:0}: Error finding container ba053e3d905a4aa217457f877f940036a88e278a2f1982d12ce4f66eb05d211f: Status 404 returned error can't find the container with id ba053e3d905a4aa217457f877f940036a88e278a2f1982d12ce4f66eb05d211f Jul 08 19:38:03.864010 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:38:03.863978 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zs5q6" event={"ID":"3854bf68-0ce0-401b-9aa7-8bc92c2ccf89","Type":"ContainerStarted","Data":"ba053e3d905a4aa217457f877f940036a88e278a2f1982d12ce4f66eb05d211f"} Jul 08 19:38:04.389011 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:38:04.388968 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:38:04.869263 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:38:04.869228 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zs5q6" event={"ID":"3854bf68-0ce0-401b-9aa7-8bc92c2ccf89","Type":"ContainerStarted","Data":"c41e9848558e05f2ae15e6f160f7cf6a7f659aea9ce4db72b8ffa4433a075b49"} Jul 08 19:38:04.887074 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:38:04.887017 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zs5q6" podStartSLOduration=1.873085873 podStartE2EDuration="2.886996533s" totalImagesPullingTime="1.01391066s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:38:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:38:04.886249811 +0000 UTC m=+850.772439853" watchObservedRunningTime="2026-07-08 19:38:04.886996533 +0000 UTC m=+850.773186561" Jul 08 19:38:12.884077 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:38:12.884031 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zs5q6" Jul 08 19:38:12.884456 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:38:12.884105 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zs5q6" Jul 08 19:38:12.896267 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:38:12.896243 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zs5q6" Jul 08 19:38:12.915802 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:38:12.915776 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zs5q6" Jul 08 19:38:20.811106 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:38:20.811072 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:38:23.281532 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:38:23.281497 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:38:57.064268 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:38:57.064237 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:38:57.142309 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:38:57.142277 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:38:58.342077 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:38:58.342049 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:38:58.441133 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:38:58.441102 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:39:26.886403 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:39:26.886371 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:39:36.085343 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:39:36.085309 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:39:48.083670 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:39:48.083632 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:39:54.693407 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:39:54.693377 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:40:03.593237 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:40:03.593200 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:40:09.602378 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:40:09.602344 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:40:11.384013 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:40:11.383985 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:40:12.274305 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:40:12.274272 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:40:34.893130 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:40:34.893053 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:40:37.488489 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:40:37.488456 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:40:37.586448 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:40:37.586418 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:40:47.491523 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:40:47.491487 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:40:52.041972 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:40:52.041939 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:40:52.138348 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:40:52.138314 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:40:52.239627 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:40:52.239600 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:40:58.630024 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:40:58.629990 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:40:58.734267 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:40:58.734231 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:40:58.830739 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:40:58.830707 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:40:58.990659 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:40:58.990572 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:41:05.186633 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:41:05.186592 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:41:05.286799 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:41:05.286759 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:41:22.184620 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:41:22.184584 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:41:31.587938 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:41:31.587903 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:41:40.179803 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:41:40.179766 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:41:45.737039 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:41:45.737005 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:41:49.717318 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:41:49.717284 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:41:49.793291 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:41:49.793252 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:41:49.796388 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:41:49.796360 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:41:58.787591 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:41:58.787550 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:42:09.187479 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:42:09.187441 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:42:21.285368 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:42:21.285333 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:42:30.794317 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:42:30.794277 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:42:31.256674 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:42:31.256643 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:42:31.357927 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:42:31.357898 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:42:33.555389 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:42:33.555352 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:42:41.435208 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:42:41.435171 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:42:53.792006 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:42:53.791973 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:43:03.901345 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:43:03.901313 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:43:29.833679 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:43:29.833594 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:43:42.286498 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:43:42.286463 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:43:50.449391 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:43:50.449354 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:43:50.549413 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:43:50.549384 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:43:54.677561 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:43:54.677525 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:43:57.085806 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:43:57.085771 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:44:08.989877 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:44:08.989823 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:44:18.195752 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:44:18.195712 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:44:27.883925 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:44:27.883888 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:44:37.191116 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:44:37.191083 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:46:24.690191 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:46:24.690108 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:46:35.713034 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:46:35.712993 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:46:45.887195 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:46:45.887158 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:46:57.086151 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:46:57.086121 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:47:09.026337 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:47:09.026296 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:47:20.080957 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:47:20.080920 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:47:28.584441 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:47:28.584405 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:47:38.887292 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:47:38.887256 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:47:50.291358 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:47:50.291269 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:47:58.691778 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:47:58.691738 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:48:09.186422 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:48:09.186385 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:48:14.681543 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:48:14.681507 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:48:40.889840 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:48:40.889807 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:48:48.081618 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:48:48.081580 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:48:56.081922 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:48:56.081884 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:48:57.186145 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:48:57.186114 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:49:09.488504 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:49:09.488469 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:49:24.880879 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:49:24.880786 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:49:36.284512 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:49:36.284478 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:49:45.587127 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:49:45.587088 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:49:55.584623 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:49:55.584589 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:50:05.080891 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:50:05.080845 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:50:14.578770 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:50:14.578733 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:50:27.983887 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:50:27.983828 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:50:36.994909 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:50:36.994874 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:50:46.486060 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:50:46.485974 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:50:58.783792 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:50:58.783762 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:51:03.421942 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:51:03.421897 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:51:03.511643 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:51:03.511612 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:51:06.646160 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:51:06.646124 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:51:19.076831 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:51:19.076791 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:51:36.189483 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:51:36.189442 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:51:48.388601 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:51:48.388555 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:52:01.586329 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:01.586291 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:52:04.146654 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:04.146579 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:52:10.189059 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:10.189026 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:52:13.864294 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:13.864261 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:52:22.130777 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:22.130697 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:52:22.296643 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:22.296610 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:52:27.261445 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:27.261400 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:52:30.187673 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:30.187635 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:52:30.287076 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:30.287047 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:52:30.688994 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:30.688959 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:52:31.779422 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:31.779388 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:52:36.586912 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:36.586851 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:52:36.652411 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:36.652377 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:52:36.808551 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:36.808518 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:52:36.899452 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:36.899374 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:52:37.977734 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:37.977700 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:52:38.045729 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:38.045694 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-5a9c03f1-fbdb854d-pf7bn"] Jul 08 19:52:38.050453 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:38.050421 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-5a9c03f1-fbdb854d-pf7bn" Jul 08 19:52:38.053426 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:38.053400 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-7kmg8\"" Jul 08 19:52:38.055643 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:38.053636 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-5a9c03f1-serving-cert\"" Jul 08 19:52:38.061172 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:38.060940 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-5a9c03f1-fbdb854d-pf7bn"] Jul 08 19:52:38.077039 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:38.077007 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:52:38.175170 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:38.175137 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89b8fa04-5f70-440a-87a1-c76f3be2bb5a-openshift-service-ca\") pod \"maas-api-e2e-ait-5a9c03f1-fbdb854d-pf7bn\" (UID: \"89b8fa04-5f70-440a-87a1-c76f3be2bb5a\") " pod="opendatahub/maas-api-e2e-ait-5a9c03f1-fbdb854d-pf7bn" Jul 08 19:52:38.175668 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:38.175638 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89b8fa04-5f70-440a-87a1-c76f3be2bb5a-maas-api-tls\") pod \"maas-api-e2e-ait-5a9c03f1-fbdb854d-pf7bn\" (UID: \"89b8fa04-5f70-440a-87a1-c76f3be2bb5a\") " pod="opendatahub/maas-api-e2e-ait-5a9c03f1-fbdb854d-pf7bn" Jul 08 19:52:38.175783 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:38.175742 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9p7mh\" (UniqueName: \"kubernetes.io/projected/89b8fa04-5f70-440a-87a1-c76f3be2bb5a-kube-api-access-9p7mh\") pod \"maas-api-e2e-ait-5a9c03f1-fbdb854d-pf7bn\" (UID: \"89b8fa04-5f70-440a-87a1-c76f3be2bb5a\") " pod="opendatahub/maas-api-e2e-ait-5a9c03f1-fbdb854d-pf7bn" Jul 08 19:52:38.178669 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:38.178644 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:52:38.276750 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:38.276666 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89b8fa04-5f70-440a-87a1-c76f3be2bb5a-openshift-service-ca\") pod \"maas-api-e2e-ait-5a9c03f1-fbdb854d-pf7bn\" (UID: \"89b8fa04-5f70-440a-87a1-c76f3be2bb5a\") " pod="opendatahub/maas-api-e2e-ait-5a9c03f1-fbdb854d-pf7bn" Jul 08 19:52:38.276750 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:38.276706 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89b8fa04-5f70-440a-87a1-c76f3be2bb5a-maas-api-tls\") pod \"maas-api-e2e-ait-5a9c03f1-fbdb854d-pf7bn\" (UID: \"89b8fa04-5f70-440a-87a1-c76f3be2bb5a\") " pod="opendatahub/maas-api-e2e-ait-5a9c03f1-fbdb854d-pf7bn" Jul 08 19:52:38.276750 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:38.276734 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p7mh\" (UniqueName: \"kubernetes.io/projected/89b8fa04-5f70-440a-87a1-c76f3be2bb5a-kube-api-access-9p7mh\") pod \"maas-api-e2e-ait-5a9c03f1-fbdb854d-pf7bn\" (UID: \"89b8fa04-5f70-440a-87a1-c76f3be2bb5a\") " pod="opendatahub/maas-api-e2e-ait-5a9c03f1-fbdb854d-pf7bn" Jul 08 19:52:38.277534 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:38.277475 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89b8fa04-5f70-440a-87a1-c76f3be2bb5a-openshift-service-ca\") pod \"maas-api-e2e-ait-5a9c03f1-fbdb854d-pf7bn\" (UID: \"89b8fa04-5f70-440a-87a1-c76f3be2bb5a\") " pod="opendatahub/maas-api-e2e-ait-5a9c03f1-fbdb854d-pf7bn" Jul 08 19:52:38.277700 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:38.277678 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:52:38.279347 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:38.279322 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89b8fa04-5f70-440a-87a1-c76f3be2bb5a-maas-api-tls\") pod \"maas-api-e2e-ait-5a9c03f1-fbdb854d-pf7bn\" (UID: \"89b8fa04-5f70-440a-87a1-c76f3be2bb5a\") " pod="opendatahub/maas-api-e2e-ait-5a9c03f1-fbdb854d-pf7bn" Jul 08 19:52:38.286213 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:38.286160 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9p7mh\" (UniqueName: \"kubernetes.io/projected/89b8fa04-5f70-440a-87a1-c76f3be2bb5a-kube-api-access-9p7mh\") pod \"maas-api-e2e-ait-5a9c03f1-fbdb854d-pf7bn\" (UID: \"89b8fa04-5f70-440a-87a1-c76f3be2bb5a\") " pod="opendatahub/maas-api-e2e-ait-5a9c03f1-fbdb854d-pf7bn" Jul 08 19:52:38.364517 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:38.364483 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-5a9c03f1-fbdb854d-pf7bn" Jul 08 19:52:38.499983 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:38.499959 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-5a9c03f1-fbdb854d-pf7bn"] Jul 08 19:52:38.502887 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:52:38.502821 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89b8fa04_5f70_440a_87a1_c76f3be2bb5a.slice/crio-022ee967a6e121580afc8e697b1cbff7b95e167261637e755e13b1ac0a5c8ed5 WatchSource:0}: Error finding container 022ee967a6e121580afc8e697b1cbff7b95e167261637e755e13b1ac0a5c8ed5: Status 404 returned error can't find the container with id 022ee967a6e121580afc8e697b1cbff7b95e167261637e755e13b1ac0a5c8ed5 Jul 08 19:52:38.504282 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:38.504263 2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 19:52:38.546242 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:38.546159 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:52:38.647348 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:38.647315 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:52:38.691241 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:38.691211 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-5a9c03f1-fbdb854d-pf7bn" event={"ID":"89b8fa04-5f70-440a-87a1-c76f3be2bb5a","Type":"ContainerStarted","Data":"022ee967a6e121580afc8e697b1cbff7b95e167261637e755e13b1ac0a5c8ed5"} Jul 08 19:52:40.702573 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:40.702531 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-5a9c03f1-fbdb854d-pf7bn" event={"ID":"89b8fa04-5f70-440a-87a1-c76f3be2bb5a","Type":"ContainerStarted","Data":"3ca6359b26d79fe7e151a9accd9711f418c90f5871e34125070f382abd6cd3d4"} Jul 08 19:52:40.702961 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:40.702599 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-5a9c03f1-fbdb854d-pf7bn" Jul 08 19:52:40.720978 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:40.720932 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-5a9c03f1-fbdb854d-pf7bn" podStartSLOduration=0.99052854 podStartE2EDuration="2.720915432s" totalImagesPullingTime="1.730386892s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:52:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:52:40.719708472 +0000 UTC m=+1726.605898522" watchObservedRunningTime="2026-07-08 19:52:40.720915432 +0000 UTC m=+1726.607105479" Jul 08 19:52:40.884274 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:40.884236 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:52:41.179318 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:41.179287 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:52:43.069238 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:43.069160 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:52:43.270242 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:43.270208 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:52:43.359527 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:43.359431 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:52:43.407756 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:43.407720 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-5a9c03f1-fbdb854d-pf7bn"] Jul 08 19:52:43.407966 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:43.407933 2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-5a9c03f1-fbdb854d-pf7bn" podUID="89b8fa04-5f70-440a-87a1-c76f3be2bb5a" containerName="maas-api" containerID="cri-o://3ca6359b26d79fe7e151a9accd9711f418c90f5871e34125070f382abd6cd3d4" gracePeriod=30 Jul 08 19:52:43.429354 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:43.429328 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:52:43.655094 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:43.655033 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-5a9c03f1-fbdb854d-pf7bn" Jul 08 19:52:43.719017 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:43.718986 2566 generic.go:356] "Generic (PLEG): container finished" podID="89b8fa04-5f70-440a-87a1-c76f3be2bb5a" containerID="3ca6359b26d79fe7e151a9accd9711f418c90f5871e34125070f382abd6cd3d4" exitCode=0 Jul 08 19:52:43.719190 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:43.719035 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-5a9c03f1-fbdb854d-pf7bn" Jul 08 19:52:43.719190 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:43.719073 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-5a9c03f1-fbdb854d-pf7bn" event={"ID":"89b8fa04-5f70-440a-87a1-c76f3be2bb5a","Type":"ContainerDied","Data":"3ca6359b26d79fe7e151a9accd9711f418c90f5871e34125070f382abd6cd3d4"} Jul 08 19:52:43.719190 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:43.719154 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-5a9c03f1-fbdb854d-pf7bn" event={"ID":"89b8fa04-5f70-440a-87a1-c76f3be2bb5a","Type":"ContainerDied","Data":"022ee967a6e121580afc8e697b1cbff7b95e167261637e755e13b1ac0a5c8ed5"} Jul 08 19:52:43.719190 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:43.719182 2566 scope.go:117] "RemoveContainer" containerID="3ca6359b26d79fe7e151a9accd9711f418c90f5871e34125070f382abd6cd3d4" Jul 08 19:52:43.725502 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:43.725483 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89b8fa04-5f70-440a-87a1-c76f3be2bb5a-maas-api-tls\") pod \"89b8fa04-5f70-440a-87a1-c76f3be2bb5a\" (UID: \"89b8fa04-5f70-440a-87a1-c76f3be2bb5a\") " Jul 08 19:52:43.725610 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:43.725591 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89b8fa04-5f70-440a-87a1-c76f3be2bb5a-openshift-service-ca\") pod \"89b8fa04-5f70-440a-87a1-c76f3be2bb5a\" (UID: \"89b8fa04-5f70-440a-87a1-c76f3be2bb5a\") " Jul 08 19:52:43.725658 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:43.725625 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9p7mh\" (UniqueName: \"kubernetes.io/projected/89b8fa04-5f70-440a-87a1-c76f3be2bb5a-kube-api-access-9p7mh\") pod \"89b8fa04-5f70-440a-87a1-c76f3be2bb5a\" (UID: \"89b8fa04-5f70-440a-87a1-c76f3be2bb5a\") " Jul 08 19:52:43.726033 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:43.726006 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89b8fa04-5f70-440a-87a1-c76f3be2bb5a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "89b8fa04-5f70-440a-87a1-c76f3be2bb5a" (UID: "89b8fa04-5f70-440a-87a1-c76f3be2bb5a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:52:43.727711 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:43.727686 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89b8fa04-5f70-440a-87a1-c76f3be2bb5a-kube-api-access-9p7mh" (OuterVolumeSpecName: "kube-api-access-9p7mh") pod "89b8fa04-5f70-440a-87a1-c76f3be2bb5a" (UID: "89b8fa04-5f70-440a-87a1-c76f3be2bb5a"). InnerVolumeSpecName "kube-api-access-9p7mh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:52:43.727711 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:43.727702 2566 scope.go:117] "RemoveContainer" containerID="3ca6359b26d79fe7e151a9accd9711f418c90f5871e34125070f382abd6cd3d4" Jul 08 19:52:43.727847 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:43.727754 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89b8fa04-5f70-440a-87a1-c76f3be2bb5a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "89b8fa04-5f70-440a-87a1-c76f3be2bb5a" (UID: "89b8fa04-5f70-440a-87a1-c76f3be2bb5a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:52:43.728016 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:52:43.727998 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ca6359b26d79fe7e151a9accd9711f418c90f5871e34125070f382abd6cd3d4\": container with ID starting with 3ca6359b26d79fe7e151a9accd9711f418c90f5871e34125070f382abd6cd3d4 not found: ID does not exist" containerID="3ca6359b26d79fe7e151a9accd9711f418c90f5871e34125070f382abd6cd3d4" Jul 08 19:52:43.728063 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:43.728023 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ca6359b26d79fe7e151a9accd9711f418c90f5871e34125070f382abd6cd3d4"} err="failed to get container status \"3ca6359b26d79fe7e151a9accd9711f418c90f5871e34125070f382abd6cd3d4\": rpc error: code = NotFound desc = could not find container \"3ca6359b26d79fe7e151a9accd9711f418c90f5871e34125070f382abd6cd3d4\": container with ID starting with 3ca6359b26d79fe7e151a9accd9711f418c90f5871e34125070f382abd6cd3d4 not found: ID does not exist" Jul 08 19:52:43.826670 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:43.826641 2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89b8fa04-5f70-440a-87a1-c76f3be2bb5a-openshift-service-ca\") on node \"ip-10-0-142-195.ec2.internal\" DevicePath \"\"" Jul 08 19:52:43.826670 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:43.826666 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9p7mh\" (UniqueName: \"kubernetes.io/projected/89b8fa04-5f70-440a-87a1-c76f3be2bb5a-kube-api-access-9p7mh\") on node \"ip-10-0-142-195.ec2.internal\" DevicePath \"\"" Jul 08 19:52:43.826670 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:43.826675 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89b8fa04-5f70-440a-87a1-c76f3be2bb5a-maas-api-tls\") on node \"ip-10-0-142-195.ec2.internal\" DevicePath \"\"" Jul 08 19:52:44.044451 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:44.044423 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-5a9c03f1-fbdb854d-pf7bn"] Jul 08 19:52:44.045956 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:44.045936 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-5a9c03f1-fbdb854d-pf7bn"] Jul 08 19:52:44.623082 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:44.623048 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89b8fa04-5f70-440a-87a1-c76f3be2bb5a" path="/var/lib/kubelet/pods/89b8fa04-5f70-440a-87a1-c76f3be2bb5a/volumes" Jul 08 19:52:46.965823 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:46.965791 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:52:49.404346 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:49.404302 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:52:49.468663 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:49.468625 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:52:49.625560 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:49.625529 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:52:49.701815 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:49.701718 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:52:50.825395 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:50.825357 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:52:50.893945 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:50.893914 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:52:50.898175 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:50.898147 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-8e698564-598c5687d7-t5gpp"] Jul 08 19:52:50.898639 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:50.898617 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89b8fa04-5f70-440a-87a1-c76f3be2bb5a" containerName="maas-api" Jul 08 19:52:50.898639 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:50.898639 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="89b8fa04-5f70-440a-87a1-c76f3be2bb5a" containerName="maas-api" Jul 08 19:52:50.898796 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:50.898723 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="89b8fa04-5f70-440a-87a1-c76f3be2bb5a" containerName="maas-api" Jul 08 19:52:50.904521 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:50.904503 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-8e698564-598c5687d7-t5gpp" Jul 08 19:52:50.907024 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:50.907004 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-8e698564-serving-cert\"" Jul 08 19:52:50.907024 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:50.907018 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-7kmg8\"" Jul 08 19:52:50.911848 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:50.911819 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-8e698564-598c5687d7-t5gpp"] Jul 08 19:52:50.954249 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:50.954209 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:52:50.989371 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:50.989316 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsbdb\" (UniqueName: \"kubernetes.io/projected/2afa5960-2121-4f27-a643-e41f7ea2d7fb-kube-api-access-rsbdb\") pod \"maas-api-e2e-ait-8e698564-598c5687d7-t5gpp\" (UID: \"2afa5960-2121-4f27-a643-e41f7ea2d7fb\") " pod="opendatahub/maas-api-e2e-ait-8e698564-598c5687d7-t5gpp" Jul 08 19:52:50.989576 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:50.989470 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2afa5960-2121-4f27-a643-e41f7ea2d7fb-maas-api-tls\") pod \"maas-api-e2e-ait-8e698564-598c5687d7-t5gpp\" (UID: \"2afa5960-2121-4f27-a643-e41f7ea2d7fb\") " pod="opendatahub/maas-api-e2e-ait-8e698564-598c5687d7-t5gpp" Jul 08 19:52:50.989576 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:50.989499 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2afa5960-2121-4f27-a643-e41f7ea2d7fb-openshift-service-ca\") pod \"maas-api-e2e-ait-8e698564-598c5687d7-t5gpp\" (UID: \"2afa5960-2121-4f27-a643-e41f7ea2d7fb\") " pod="opendatahub/maas-api-e2e-ait-8e698564-598c5687d7-t5gpp" Jul 08 19:52:51.090376 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:51.090256 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2afa5960-2121-4f27-a643-e41f7ea2d7fb-maas-api-tls\") pod \"maas-api-e2e-ait-8e698564-598c5687d7-t5gpp\" (UID: \"2afa5960-2121-4f27-a643-e41f7ea2d7fb\") " pod="opendatahub/maas-api-e2e-ait-8e698564-598c5687d7-t5gpp" Jul 08 19:52:51.090376 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:51.090312 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2afa5960-2121-4f27-a643-e41f7ea2d7fb-openshift-service-ca\") pod \"maas-api-e2e-ait-8e698564-598c5687d7-t5gpp\" (UID: \"2afa5960-2121-4f27-a643-e41f7ea2d7fb\") " pod="opendatahub/maas-api-e2e-ait-8e698564-598c5687d7-t5gpp" Jul 08 19:52:51.090376 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:51.090351 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsbdb\" (UniqueName: \"kubernetes.io/projected/2afa5960-2121-4f27-a643-e41f7ea2d7fb-kube-api-access-rsbdb\") pod \"maas-api-e2e-ait-8e698564-598c5687d7-t5gpp\" (UID: \"2afa5960-2121-4f27-a643-e41f7ea2d7fb\") " pod="opendatahub/maas-api-e2e-ait-8e698564-598c5687d7-t5gpp" Jul 08 19:52:51.090702 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:52:51.090447 2566 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-ait-8e698564-serving-cert: secret "maas-api-e2e-ait-8e698564-serving-cert" not found Jul 08 19:52:51.090702 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:52:51.090552 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2afa5960-2121-4f27-a643-e41f7ea2d7fb-maas-api-tls podName:2afa5960-2121-4f27-a643-e41f7ea2d7fb nodeName:}" failed. No retries permitted until 2026-07-08 19:52:51.590526441 +0000 UTC m=+1737.476716482 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2afa5960-2121-4f27-a643-e41f7ea2d7fb-maas-api-tls") pod "maas-api-e2e-ait-8e698564-598c5687d7-t5gpp" (UID: "2afa5960-2121-4f27-a643-e41f7ea2d7fb") : secret "maas-api-e2e-ait-8e698564-serving-cert" not found Jul 08 19:52:51.091077 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:51.091059 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2afa5960-2121-4f27-a643-e41f7ea2d7fb-openshift-service-ca\") pod \"maas-api-e2e-ait-8e698564-598c5687d7-t5gpp\" (UID: \"2afa5960-2121-4f27-a643-e41f7ea2d7fb\") " pod="opendatahub/maas-api-e2e-ait-8e698564-598c5687d7-t5gpp" Jul 08 19:52:51.101599 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:51.101575 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsbdb\" (UniqueName: \"kubernetes.io/projected/2afa5960-2121-4f27-a643-e41f7ea2d7fb-kube-api-access-rsbdb\") pod \"maas-api-e2e-ait-8e698564-598c5687d7-t5gpp\" (UID: \"2afa5960-2121-4f27-a643-e41f7ea2d7fb\") " pod="opendatahub/maas-api-e2e-ait-8e698564-598c5687d7-t5gpp" Jul 08 19:52:51.282604 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:51.282567 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:52:51.385501 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:51.385415 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:52:51.489293 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:51.489236 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:52:51.595257 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:51.595217 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2afa5960-2121-4f27-a643-e41f7ea2d7fb-maas-api-tls\") pod \"maas-api-e2e-ait-8e698564-598c5687d7-t5gpp\" (UID: \"2afa5960-2121-4f27-a643-e41f7ea2d7fb\") " pod="opendatahub/maas-api-e2e-ait-8e698564-598c5687d7-t5gpp" Jul 08 19:52:51.597667 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:51.597635 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2afa5960-2121-4f27-a643-e41f7ea2d7fb-maas-api-tls\") pod \"maas-api-e2e-ait-8e698564-598c5687d7-t5gpp\" (UID: \"2afa5960-2121-4f27-a643-e41f7ea2d7fb\") " pod="opendatahub/maas-api-e2e-ait-8e698564-598c5687d7-t5gpp" Jul 08 19:52:51.818275 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:51.818239 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-8e698564-598c5687d7-t5gpp" Jul 08 19:52:51.956706 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:51.956657 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-8e698564-598c5687d7-t5gpp"] Jul 08 19:52:51.960228 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:52:51.960198 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2afa5960_2121_4f27_a643_e41f7ea2d7fb.slice/crio-d0247b0b5db62c27c4993dfae4393f918744e584b13b86a2946f02a24f654438 WatchSource:0}: Error finding container d0247b0b5db62c27c4993dfae4393f918744e584b13b86a2946f02a24f654438: Status 404 returned error can't find the container with id d0247b0b5db62c27c4993dfae4393f918744e584b13b86a2946f02a24f654438 Jul 08 19:52:52.764217 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:52.764121 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-8e698564-598c5687d7-t5gpp" event={"ID":"2afa5960-2121-4f27-a643-e41f7ea2d7fb","Type":"ContainerStarted","Data":"25aede249fcd0b3b207815ca78b452f0f4e4406c95b8067109614cc150808f39"} Jul 08 19:52:52.764217 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:52.764180 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-8e698564-598c5687d7-t5gpp" event={"ID":"2afa5960-2121-4f27-a643-e41f7ea2d7fb","Type":"ContainerStarted","Data":"d0247b0b5db62c27c4993dfae4393f918744e584b13b86a2946f02a24f654438"} Jul 08 19:52:52.764217 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:52.764212 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-8e698564-598c5687d7-t5gpp" Jul 08 19:52:52.781785 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:52.781731 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-8e698564-598c5687d7-t5gpp" podStartSLOduration=2.328776122 podStartE2EDuration="2.781712123s" totalImagesPullingTime="452.936001ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:52:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:52:52.780634417 +0000 UTC m=+1738.666824464" watchObservedRunningTime="2026-07-08 19:52:52.781712123 +0000 UTC m=+1738.667902169" Jul 08 19:52:52.985724 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:52.985687 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:52:53.074116 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:53.074068 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:52:56.123513 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:56.123477 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:52:56.482296 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:56.482262 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-8e698564-598c5687d7-t5gpp"] Jul 08 19:52:56.482491 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:56.482468 2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-8e698564-598c5687d7-t5gpp" podUID="2afa5960-2121-4f27-a643-e41f7ea2d7fb" containerName="maas-api" containerID="cri-o://25aede249fcd0b3b207815ca78b452f0f4e4406c95b8067109614cc150808f39" gracePeriod=30 Jul 08 19:52:56.510382 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:56.510351 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:52:56.724480 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:56.724451 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-8e698564-598c5687d7-t5gpp" Jul 08 19:52:56.782959 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:56.782882 2566 generic.go:356] "Generic (PLEG): container finished" podID="2afa5960-2121-4f27-a643-e41f7ea2d7fb" containerID="25aede249fcd0b3b207815ca78b452f0f4e4406c95b8067109614cc150808f39" exitCode=0 Jul 08 19:52:56.782959 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:56.782928 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-8e698564-598c5687d7-t5gpp" Jul 08 19:52:56.782959 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:56.782952 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-8e698564-598c5687d7-t5gpp" event={"ID":"2afa5960-2121-4f27-a643-e41f7ea2d7fb","Type":"ContainerDied","Data":"25aede249fcd0b3b207815ca78b452f0f4e4406c95b8067109614cc150808f39"} Jul 08 19:52:56.783208 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:56.782999 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-8e698564-598c5687d7-t5gpp" event={"ID":"2afa5960-2121-4f27-a643-e41f7ea2d7fb","Type":"ContainerDied","Data":"d0247b0b5db62c27c4993dfae4393f918744e584b13b86a2946f02a24f654438"} Jul 08 19:52:56.783208 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:56.783027 2566 scope.go:117] "RemoveContainer" containerID="25aede249fcd0b3b207815ca78b452f0f4e4406c95b8067109614cc150808f39" Jul 08 19:52:56.791264 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:56.791238 2566 scope.go:117] "RemoveContainer" containerID="25aede249fcd0b3b207815ca78b452f0f4e4406c95b8067109614cc150808f39" Jul 08 19:52:56.791511 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:52:56.791493 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25aede249fcd0b3b207815ca78b452f0f4e4406c95b8067109614cc150808f39\": container with ID starting with 25aede249fcd0b3b207815ca78b452f0f4e4406c95b8067109614cc150808f39 not found: ID does not exist" containerID="25aede249fcd0b3b207815ca78b452f0f4e4406c95b8067109614cc150808f39" Jul 08 19:52:56.791555 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:56.791520 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25aede249fcd0b3b207815ca78b452f0f4e4406c95b8067109614cc150808f39"} err="failed to get container status \"25aede249fcd0b3b207815ca78b452f0f4e4406c95b8067109614cc150808f39\": rpc error: code = NotFound desc = could not find container \"25aede249fcd0b3b207815ca78b452f0f4e4406c95b8067109614cc150808f39\": container with ID starting with 25aede249fcd0b3b207815ca78b452f0f4e4406c95b8067109614cc150808f39 not found: ID does not exist" Jul 08 19:52:56.844252 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:56.844230 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2afa5960-2121-4f27-a643-e41f7ea2d7fb-openshift-service-ca\") pod \"2afa5960-2121-4f27-a643-e41f7ea2d7fb\" (UID: \"2afa5960-2121-4f27-a643-e41f7ea2d7fb\") " Jul 08 19:52:56.844357 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:56.844318 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rsbdb\" (UniqueName: \"kubernetes.io/projected/2afa5960-2121-4f27-a643-e41f7ea2d7fb-kube-api-access-rsbdb\") pod \"2afa5960-2121-4f27-a643-e41f7ea2d7fb\" (UID: \"2afa5960-2121-4f27-a643-e41f7ea2d7fb\") " Jul 08 19:52:56.844357 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:56.844338 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2afa5960-2121-4f27-a643-e41f7ea2d7fb-maas-api-tls\") pod \"2afa5960-2121-4f27-a643-e41f7ea2d7fb\" (UID: \"2afa5960-2121-4f27-a643-e41f7ea2d7fb\") " Jul 08 19:52:56.844634 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:56.844611 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2afa5960-2121-4f27-a643-e41f7ea2d7fb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2afa5960-2121-4f27-a643-e41f7ea2d7fb" (UID: "2afa5960-2121-4f27-a643-e41f7ea2d7fb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:52:56.846294 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:56.846269 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2afa5960-2121-4f27-a643-e41f7ea2d7fb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2afa5960-2121-4f27-a643-e41f7ea2d7fb" (UID: "2afa5960-2121-4f27-a643-e41f7ea2d7fb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:52:56.846383 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:56.846333 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2afa5960-2121-4f27-a643-e41f7ea2d7fb-kube-api-access-rsbdb" (OuterVolumeSpecName: "kube-api-access-rsbdb") pod "2afa5960-2121-4f27-a643-e41f7ea2d7fb" (UID: "2afa5960-2121-4f27-a643-e41f7ea2d7fb"). InnerVolumeSpecName "kube-api-access-rsbdb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:52:56.945030 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:56.945003 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsbdb\" (UniqueName: \"kubernetes.io/projected/2afa5960-2121-4f27-a643-e41f7ea2d7fb-kube-api-access-rsbdb\") on node \"ip-10-0-142-195.ec2.internal\" DevicePath \"\"" Jul 08 19:52:56.945030 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:56.945027 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2afa5960-2121-4f27-a643-e41f7ea2d7fb-maas-api-tls\") on node \"ip-10-0-142-195.ec2.internal\" DevicePath \"\"" Jul 08 19:52:56.945030 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:56.945036 2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2afa5960-2121-4f27-a643-e41f7ea2d7fb-openshift-service-ca\") on node \"ip-10-0-142-195.ec2.internal\" DevicePath \"\"" Jul 08 19:52:57.104543 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:57.104510 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-8e698564-598c5687d7-t5gpp"] Jul 08 19:52:57.108785 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:57.108766 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-8e698564-598c5687d7-t5gpp"] Jul 08 19:52:58.622817 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:52:58.622783 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2afa5960-2121-4f27-a643-e41f7ea2d7fb" path="/var/lib/kubelet/pods/2afa5960-2121-4f27-a643-e41f7ea2d7fb/volumes" Jul 08 19:53:02.301403 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:02.301367 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:53:02.405749 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:02.405712 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:53:04.070100 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:04.070066 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:53:04.388291 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:04.388207 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:53:08.432418 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:08.432377 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:53:08.489996 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:08.489942 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:53:08.634876 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:08.634833 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:53:08.736095 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:08.736011 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:53:09.815766 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:09.815728 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:53:09.881813 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:09.881779 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-derive-1d4d50e0-79cd69fcc4-w4nq8"] Jul 08 19:53:09.882164 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:09.882149 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2afa5960-2121-4f27-a643-e41f7ea2d7fb" containerName="maas-api" Jul 08 19:53:09.882164 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:09.882165 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2afa5960-2121-4f27-a643-e41f7ea2d7fb" containerName="maas-api" Jul 08 19:53:09.882276 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:09.882266 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2afa5960-2121-4f27-a643-e41f7ea2d7fb" containerName="maas-api" Jul 08 19:53:09.886690 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:09.886668 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-1d4d50e0-79cd69fcc4-w4nq8" Jul 08 19:53:09.889107 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:09.889087 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-derive-1d4d50e0-serving-cert\"" Jul 08 19:53:09.889213 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:09.889113 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-7kmg8\"" Jul 08 19:53:09.893880 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:09.893840 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:53:09.898646 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:09.895633 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-1d4d50e0-79cd69fcc4-w4nq8"] Jul 08 19:53:09.957808 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:09.957775 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:53:09.962903 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:09.962881 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp5bk\" (UniqueName: \"kubernetes.io/projected/58eb9dc0-e546-4067-960e-b0459904e76d-kube-api-access-jp5bk\") pod \"maas-api-e2e-derive-1d4d50e0-79cd69fcc4-w4nq8\" (UID: \"58eb9dc0-e546-4067-960e-b0459904e76d\") " pod="opendatahub/maas-api-e2e-derive-1d4d50e0-79cd69fcc4-w4nq8" Jul 08 19:53:09.963001 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:09.962937 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58eb9dc0-e546-4067-960e-b0459904e76d-maas-api-tls\") pod \"maas-api-e2e-derive-1d4d50e0-79cd69fcc4-w4nq8\" (UID: \"58eb9dc0-e546-4067-960e-b0459904e76d\") " pod="opendatahub/maas-api-e2e-derive-1d4d50e0-79cd69fcc4-w4nq8" Jul 08 19:53:09.963001 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:09.962983 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58eb9dc0-e546-4067-960e-b0459904e76d-openshift-service-ca\") pod \"maas-api-e2e-derive-1d4d50e0-79cd69fcc4-w4nq8\" (UID: \"58eb9dc0-e546-4067-960e-b0459904e76d\") " pod="opendatahub/maas-api-e2e-derive-1d4d50e0-79cd69fcc4-w4nq8" Jul 08 19:53:10.063767 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:10.063733 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp5bk\" (UniqueName: \"kubernetes.io/projected/58eb9dc0-e546-4067-960e-b0459904e76d-kube-api-access-jp5bk\") pod \"maas-api-e2e-derive-1d4d50e0-79cd69fcc4-w4nq8\" (UID: \"58eb9dc0-e546-4067-960e-b0459904e76d\") " pod="opendatahub/maas-api-e2e-derive-1d4d50e0-79cd69fcc4-w4nq8" Jul 08 19:53:10.063956 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:10.063792 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58eb9dc0-e546-4067-960e-b0459904e76d-maas-api-tls\") pod \"maas-api-e2e-derive-1d4d50e0-79cd69fcc4-w4nq8\" (UID: \"58eb9dc0-e546-4067-960e-b0459904e76d\") " pod="opendatahub/maas-api-e2e-derive-1d4d50e0-79cd69fcc4-w4nq8" Jul 08 19:53:10.063956 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:10.063819 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58eb9dc0-e546-4067-960e-b0459904e76d-openshift-service-ca\") pod \"maas-api-e2e-derive-1d4d50e0-79cd69fcc4-w4nq8\" (UID: \"58eb9dc0-e546-4067-960e-b0459904e76d\") " pod="opendatahub/maas-api-e2e-derive-1d4d50e0-79cd69fcc4-w4nq8" Jul 08 19:53:10.064094 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:53:10.063963 2566 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-derive-1d4d50e0-serving-cert: secret "maas-api-e2e-derive-1d4d50e0-serving-cert" not found Jul 08 19:53:10.064094 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:53:10.064054 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58eb9dc0-e546-4067-960e-b0459904e76d-maas-api-tls podName:58eb9dc0-e546-4067-960e-b0459904e76d nodeName:}" failed. No retries permitted until 2026-07-08 19:53:10.564031028 +0000 UTC m=+1756.450221053 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/58eb9dc0-e546-4067-960e-b0459904e76d-maas-api-tls") pod "maas-api-e2e-derive-1d4d50e0-79cd69fcc4-w4nq8" (UID: "58eb9dc0-e546-4067-960e-b0459904e76d") : secret "maas-api-e2e-derive-1d4d50e0-serving-cert" not found Jul 08 19:53:10.064473 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:10.064452 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58eb9dc0-e546-4067-960e-b0459904e76d-openshift-service-ca\") pod \"maas-api-e2e-derive-1d4d50e0-79cd69fcc4-w4nq8\" (UID: \"58eb9dc0-e546-4067-960e-b0459904e76d\") " pod="opendatahub/maas-api-e2e-derive-1d4d50e0-79cd69fcc4-w4nq8" Jul 08 19:53:10.071786 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:10.071731 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp5bk\" (UniqueName: \"kubernetes.io/projected/58eb9dc0-e546-4067-960e-b0459904e76d-kube-api-access-jp5bk\") pod \"maas-api-e2e-derive-1d4d50e0-79cd69fcc4-w4nq8\" (UID: \"58eb9dc0-e546-4067-960e-b0459904e76d\") " pod="opendatahub/maas-api-e2e-derive-1d4d50e0-79cd69fcc4-w4nq8" Jul 08 19:53:10.506619 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:10.506583 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:53:10.568552 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:10.568521 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58eb9dc0-e546-4067-960e-b0459904e76d-maas-api-tls\") pod \"maas-api-e2e-derive-1d4d50e0-79cd69fcc4-w4nq8\" (UID: \"58eb9dc0-e546-4067-960e-b0459904e76d\") " pod="opendatahub/maas-api-e2e-derive-1d4d50e0-79cd69fcc4-w4nq8" Jul 08 19:53:10.570844 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:10.570821 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58eb9dc0-e546-4067-960e-b0459904e76d-maas-api-tls\") pod \"maas-api-e2e-derive-1d4d50e0-79cd69fcc4-w4nq8\" (UID: \"58eb9dc0-e546-4067-960e-b0459904e76d\") " pod="opendatahub/maas-api-e2e-derive-1d4d50e0-79cd69fcc4-w4nq8" Jul 08 19:53:10.609556 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:10.609530 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:53:10.707697 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:10.707659 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:53:10.804527 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:10.804451 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-1d4d50e0-79cd69fcc4-w4nq8" Jul 08 19:53:10.932727 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:10.932703 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-1d4d50e0-79cd69fcc4-w4nq8"] Jul 08 19:53:10.934788 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:53:10.934748 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58eb9dc0_e546_4067_960e_b0459904e76d.slice/crio-6b450577c6c82d1b9536cafd3a3568a00c2a325f5e104146fe834d704967e4ff WatchSource:0}: Error finding container 6b450577c6c82d1b9536cafd3a3568a00c2a325f5e104146fe834d704967e4ff: Status 404 returned error can't find the container with id 6b450577c6c82d1b9536cafd3a3568a00c2a325f5e104146fe834d704967e4ff Jul 08 19:53:11.852286 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:11.852252 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-1d4d50e0-79cd69fcc4-w4nq8" event={"ID":"58eb9dc0-e546-4067-960e-b0459904e76d","Type":"ContainerStarted","Data":"3404a06372f05636e47830e098f5fbcdd4ef464f7db016bd4af369da3d27227e"} Jul 08 19:53:11.852478 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:11.852305 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-1d4d50e0-79cd69fcc4-w4nq8" event={"ID":"58eb9dc0-e546-4067-960e-b0459904e76d","Type":"ContainerStarted","Data":"6b450577c6c82d1b9536cafd3a3568a00c2a325f5e104146fe834d704967e4ff"} Jul 08 19:53:11.852478 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:11.852388 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-derive-1d4d50e0-79cd69fcc4-w4nq8" Jul 08 19:53:11.870930 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:11.870882 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-derive-1d4d50e0-79cd69fcc4-w4nq8" podStartSLOduration=2.505776581 podStartE2EDuration="2.870846016s" totalImagesPullingTime="365.069435ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:53:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:53:11.869843991 +0000 UTC m=+1757.756034039" watchObservedRunningTime="2026-07-08 19:53:11.870846016 +0000 UTC m=+1757.757036062" Jul 08 19:53:12.043085 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:12.043047 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:53:13.784187 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:13.784152 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:53:14.180391 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:14.180356 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:53:14.367935 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:14.367901 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:53:14.469061 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:14.468984 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:53:14.481093 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:14.481057 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-1d4d50e0-79cd69fcc4-w4nq8"] Jul 08 19:53:14.481369 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:14.481330 2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-derive-1d4d50e0-79cd69fcc4-w4nq8" podUID="58eb9dc0-e546-4067-960e-b0459904e76d" containerName="maas-api" containerID="cri-o://3404a06372f05636e47830e098f5fbcdd4ef464f7db016bd4af369da3d27227e" gracePeriod=30 Jul 08 19:53:14.526201 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:14.526167 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:53:14.724336 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:14.724275 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-1d4d50e0-79cd69fcc4-w4nq8" Jul 08 19:53:14.806932 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:14.806904 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jp5bk\" (UniqueName: \"kubernetes.io/projected/58eb9dc0-e546-4067-960e-b0459904e76d-kube-api-access-jp5bk\") pod \"58eb9dc0-e546-4067-960e-b0459904e76d\" (UID: \"58eb9dc0-e546-4067-960e-b0459904e76d\") " Jul 08 19:53:14.807293 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:14.806948 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58eb9dc0-e546-4067-960e-b0459904e76d-openshift-service-ca\") pod \"58eb9dc0-e546-4067-960e-b0459904e76d\" (UID: \"58eb9dc0-e546-4067-960e-b0459904e76d\") " Jul 08 19:53:14.807293 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:14.806988 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58eb9dc0-e546-4067-960e-b0459904e76d-maas-api-tls\") pod \"58eb9dc0-e546-4067-960e-b0459904e76d\" (UID: \"58eb9dc0-e546-4067-960e-b0459904e76d\") " Jul 08 19:53:14.807373 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:14.807344 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58eb9dc0-e546-4067-960e-b0459904e76d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "58eb9dc0-e546-4067-960e-b0459904e76d" (UID: "58eb9dc0-e546-4067-960e-b0459904e76d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:53:14.808970 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:14.808951 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58eb9dc0-e546-4067-960e-b0459904e76d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "58eb9dc0-e546-4067-960e-b0459904e76d" (UID: "58eb9dc0-e546-4067-960e-b0459904e76d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:53:14.809091 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:14.808988 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58eb9dc0-e546-4067-960e-b0459904e76d-kube-api-access-jp5bk" (OuterVolumeSpecName: "kube-api-access-jp5bk") pod "58eb9dc0-e546-4067-960e-b0459904e76d" (UID: "58eb9dc0-e546-4067-960e-b0459904e76d"). InnerVolumeSpecName "kube-api-access-jp5bk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:53:14.867184 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:14.867153 2566 generic.go:356] "Generic (PLEG): container finished" podID="58eb9dc0-e546-4067-960e-b0459904e76d" containerID="3404a06372f05636e47830e098f5fbcdd4ef464f7db016bd4af369da3d27227e" exitCode=0 Jul 08 19:53:14.867359 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:14.867196 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-1d4d50e0-79cd69fcc4-w4nq8" Jul 08 19:53:14.867359 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:14.867205 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-1d4d50e0-79cd69fcc4-w4nq8" event={"ID":"58eb9dc0-e546-4067-960e-b0459904e76d","Type":"ContainerDied","Data":"3404a06372f05636e47830e098f5fbcdd4ef464f7db016bd4af369da3d27227e"} Jul 08 19:53:14.867359 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:14.867246 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-1d4d50e0-79cd69fcc4-w4nq8" event={"ID":"58eb9dc0-e546-4067-960e-b0459904e76d","Type":"ContainerDied","Data":"6b450577c6c82d1b9536cafd3a3568a00c2a325f5e104146fe834d704967e4ff"} Jul 08 19:53:14.867359 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:14.867266 2566 scope.go:117] "RemoveContainer" containerID="3404a06372f05636e47830e098f5fbcdd4ef464f7db016bd4af369da3d27227e" Jul 08 19:53:14.876881 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:14.876841 2566 scope.go:117] "RemoveContainer" containerID="3404a06372f05636e47830e098f5fbcdd4ef464f7db016bd4af369da3d27227e" Jul 08 19:53:14.877133 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:53:14.877115 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3404a06372f05636e47830e098f5fbcdd4ef464f7db016bd4af369da3d27227e\": container with ID starting with 3404a06372f05636e47830e098f5fbcdd4ef464f7db016bd4af369da3d27227e not found: ID does not exist" containerID="3404a06372f05636e47830e098f5fbcdd4ef464f7db016bd4af369da3d27227e" Jul 08 19:53:14.877186 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:14.877142 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3404a06372f05636e47830e098f5fbcdd4ef464f7db016bd4af369da3d27227e"} err="failed to get container status \"3404a06372f05636e47830e098f5fbcdd4ef464f7db016bd4af369da3d27227e\": rpc error: code = NotFound desc = could not find container \"3404a06372f05636e47830e098f5fbcdd4ef464f7db016bd4af369da3d27227e\": container with ID starting with 3404a06372f05636e47830e098f5fbcdd4ef464f7db016bd4af369da3d27227e not found: ID does not exist" Jul 08 19:53:14.887736 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:14.887713 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-1d4d50e0-79cd69fcc4-w4nq8"] Jul 08 19:53:14.891799 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:14.891778 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-derive-1d4d50e0-79cd69fcc4-w4nq8"] Jul 08 19:53:14.908327 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:14.908308 2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58eb9dc0-e546-4067-960e-b0459904e76d-openshift-service-ca\") on node \"ip-10-0-142-195.ec2.internal\" DevicePath \"\"" Jul 08 19:53:14.908384 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:14.908328 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58eb9dc0-e546-4067-960e-b0459904e76d-maas-api-tls\") on node \"ip-10-0-142-195.ec2.internal\" DevicePath \"\"" Jul 08 19:53:14.908384 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:14.908338 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jp5bk\" (UniqueName: \"kubernetes.io/projected/58eb9dc0-e546-4067-960e-b0459904e76d-kube-api-access-jp5bk\") on node \"ip-10-0-142-195.ec2.internal\" DevicePath \"\"" Jul 08 19:53:16.623452 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:16.623419 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58eb9dc0-e546-4067-960e-b0459904e76d" path="/var/lib/kubelet/pods/58eb9dc0-e546-4067-960e-b0459904e76d/volumes" Jul 08 19:53:17.606177 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:17.606138 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:53:20.310185 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:20.310155 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:53:20.372724 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:20.372692 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:53:20.776940 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:20.776907 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:53:20.872296 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:20.872263 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:53:20.979219 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:20.979186 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:53:22.372959 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:22.372923 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:53:24.584714 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:24.584676 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:53:28.323290 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:28.323257 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:53:28.377222 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:28.377187 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:53:28.448170 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:28.448129 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:53:28.543593 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:28.543556 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:53:28.599977 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:28.599890 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:53:29.022135 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:29.022104 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:53:55.015367 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:53:55.015284 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:54:00.934267 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:54:00.934227 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:54:03.215219 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:54:03.215182 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:54:17.056355 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:54:17.056322 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:54:22.994163 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:54:22.994128 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:54:23.097120 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:54:23.097086 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:54:23.194490 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:54:23.194450 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:54:33.715485 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:54:33.715447 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:54:34.510007 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:54:34.509971 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:54:40.790753 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:54:40.790722 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:54:40.892225 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:54:40.892191 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:54:40.995456 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:54:40.995408 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:54:41.133485 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:54:41.133401 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:54:41.893168 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:54:41.893133 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:54:41.992266 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:54:41.992232 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:54:42.094139 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:54:42.094109 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:54:43.011323 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:54:43.011287 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:54:43.902811 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:54:43.902780 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:54:49.570260 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:54:49.570222 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:54:49.624335 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:54:49.624299 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:54:50.241356 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:54:50.241326 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:54:50.490260 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:54:50.490229 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:54:56.201238 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:54:56.201201 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:54:56.302094 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:54:56.302065 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:54:56.440073 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:54:56.440034 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:54:56.772238 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:54:56.772207 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:54:56.982124 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:54:56.982092 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:54:59.189808 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:54:59.189773 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:54:59.763175 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:54:59.763138 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:54:59.863397 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:54:59.863361 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:54:59.889646 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:54:59.889614 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-e9d7beea-f667cb85f-44vgg"] Jul 08 19:54:59.889991 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:54:59.889970 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58eb9dc0-e546-4067-960e-b0459904e76d" containerName="maas-api" Jul 08 19:54:59.889991 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:54:59.889989 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="58eb9dc0-e546-4067-960e-b0459904e76d" containerName="maas-api" Jul 08 19:54:59.890141 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:54:59.890070 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="58eb9dc0-e546-4067-960e-b0459904e76d" containerName="maas-api" Jul 08 19:54:59.899621 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:54:59.899593 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-e9d7beea-f667cb85f-44vgg" Jul 08 19:54:59.903060 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:54:59.903037 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-7kmg8\"" Jul 08 19:54:59.903355 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:54:59.903069 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-e9d7beea-serving-cert\"" Jul 08 19:54:59.903513 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:54:59.903494 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-e9d7beea-f667cb85f-44vgg"] Jul 08 19:54:59.965141 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:54:59.965114 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:55:00.039178 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:00.039085 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ab7b6de-ada8-4387-917a-940b9735aed8-maas-api-tls\") pod \"maas-api-e2e-mt-e9d7beea-f667cb85f-44vgg\" (UID: \"1ab7b6de-ada8-4387-917a-940b9735aed8\") " pod="opendatahub/maas-api-e2e-mt-e9d7beea-f667cb85f-44vgg" Jul 08 19:55:00.039178 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:00.039156 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ab7b6de-ada8-4387-917a-940b9735aed8-openshift-service-ca\") pod \"maas-api-e2e-mt-e9d7beea-f667cb85f-44vgg\" (UID: \"1ab7b6de-ada8-4387-917a-940b9735aed8\") " pod="opendatahub/maas-api-e2e-mt-e9d7beea-f667cb85f-44vgg" Jul 08 19:55:00.039374 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:00.039210 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps6dl\" (UniqueName: \"kubernetes.io/projected/1ab7b6de-ada8-4387-917a-940b9735aed8-kube-api-access-ps6dl\") pod \"maas-api-e2e-mt-e9d7beea-f667cb85f-44vgg\" (UID: \"1ab7b6de-ada8-4387-917a-940b9735aed8\") " pod="opendatahub/maas-api-e2e-mt-e9d7beea-f667cb85f-44vgg" Jul 08 19:55:00.063105 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:00.063065 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:55:00.140229 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:00.140195 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ab7b6de-ada8-4387-917a-940b9735aed8-maas-api-tls\") pod \"maas-api-e2e-mt-e9d7beea-f667cb85f-44vgg\" (UID: \"1ab7b6de-ada8-4387-917a-940b9735aed8\") " pod="opendatahub/maas-api-e2e-mt-e9d7beea-f667cb85f-44vgg" Jul 08 19:55:00.140386 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:00.140257 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ab7b6de-ada8-4387-917a-940b9735aed8-openshift-service-ca\") pod \"maas-api-e2e-mt-e9d7beea-f667cb85f-44vgg\" (UID: \"1ab7b6de-ada8-4387-917a-940b9735aed8\") " pod="opendatahub/maas-api-e2e-mt-e9d7beea-f667cb85f-44vgg" Jul 08 19:55:00.140386 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:00.140306 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps6dl\" (UniqueName: \"kubernetes.io/projected/1ab7b6de-ada8-4387-917a-940b9735aed8-kube-api-access-ps6dl\") pod \"maas-api-e2e-mt-e9d7beea-f667cb85f-44vgg\" (UID: \"1ab7b6de-ada8-4387-917a-940b9735aed8\") " pod="opendatahub/maas-api-e2e-mt-e9d7beea-f667cb85f-44vgg" Jul 08 19:55:00.141020 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:00.141002 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ab7b6de-ada8-4387-917a-940b9735aed8-openshift-service-ca\") pod \"maas-api-e2e-mt-e9d7beea-f667cb85f-44vgg\" (UID: \"1ab7b6de-ada8-4387-917a-940b9735aed8\") " pod="opendatahub/maas-api-e2e-mt-e9d7beea-f667cb85f-44vgg" Jul 08 19:55:00.142562 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:00.142545 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ab7b6de-ada8-4387-917a-940b9735aed8-maas-api-tls\") pod \"maas-api-e2e-mt-e9d7beea-f667cb85f-44vgg\" (UID: \"1ab7b6de-ada8-4387-917a-940b9735aed8\") " pod="opendatahub/maas-api-e2e-mt-e9d7beea-f667cb85f-44vgg" Jul 08 19:55:00.148003 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:00.147981 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps6dl\" (UniqueName: \"kubernetes.io/projected/1ab7b6de-ada8-4387-917a-940b9735aed8-kube-api-access-ps6dl\") pod \"maas-api-e2e-mt-e9d7beea-f667cb85f-44vgg\" (UID: \"1ab7b6de-ada8-4387-917a-940b9735aed8\") " pod="opendatahub/maas-api-e2e-mt-e9d7beea-f667cb85f-44vgg" Jul 08 19:55:00.213044 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:00.213015 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-e9d7beea-f667cb85f-44vgg" Jul 08 19:55:00.343575 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:00.343503 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-e9d7beea-f667cb85f-44vgg"] Jul 08 19:55:00.345928 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:55:00.345899 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ab7b6de_ada8_4387_917a_940b9735aed8.slice/crio-b573011869d085d60a6107605b6262e1ca3a4607fedc5e364f2b16f51d34d3cd WatchSource:0}: Error finding container b573011869d085d60a6107605b6262e1ca3a4607fedc5e364f2b16f51d34d3cd: Status 404 returned error can't find the container with id b573011869d085d60a6107605b6262e1ca3a4607fedc5e364f2b16f51d34d3cd Jul 08 19:55:00.439967 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:00.439932 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:55:00.540407 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:00.540364 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:55:01.336483 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:01.336450 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-e9d7beea-f667cb85f-44vgg" event={"ID":"1ab7b6de-ada8-4387-917a-940b9735aed8","Type":"ContainerStarted","Data":"b6908a4092585c60d84b9bbbfca41d35d0d593313ec2f612d92796c72ea28d8c"} Jul 08 19:55:01.336892 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:01.336496 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-e9d7beea-f667cb85f-44vgg" event={"ID":"1ab7b6de-ada8-4387-917a-940b9735aed8","Type":"ContainerStarted","Data":"b573011869d085d60a6107605b6262e1ca3a4607fedc5e364f2b16f51d34d3cd"} Jul 08 19:55:01.336892 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:01.336526 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-e9d7beea-f667cb85f-44vgg" Jul 08 19:55:01.354740 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:01.354676 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-e9d7beea-f667cb85f-44vgg" podStartSLOduration=1.985013914 podStartE2EDuration="2.354655597s" totalImagesPullingTime="369.641683ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:54:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:55:01.354180202 +0000 UTC m=+1867.240370249" watchObservedRunningTime="2026-07-08 19:55:01.354655597 +0000 UTC m=+1867.240845641" Jul 08 19:55:04.272581 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:04.272550 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:55:04.916292 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:04.916257 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:55:05.016370 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:05.016340 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:55:05.418840 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:05.418805 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:55:07.347284 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:07.347200 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-e9d7beea-f667cb85f-44vgg" Jul 08 19:55:10.496462 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:10.496427 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:55:11.448650 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:11.448611 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:55:11.549261 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:11.549224 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:55:11.868266 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:11.868165 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:55:12.292401 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:12.292369 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:55:12.315313 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:12.315287 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-e9d7beea-f667cb85f-44vgg"] Jul 08 19:55:12.315498 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:12.315475 2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-e9d7beea-f667cb85f-44vgg" podUID="1ab7b6de-ada8-4387-917a-940b9735aed8" containerName="maas-api" containerID="cri-o://b6908a4092585c60d84b9bbbfca41d35d0d593313ec2f612d92796c72ea28d8c" gracePeriod=30 Jul 08 19:55:12.341576 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:12.341544 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-api-e2e-mt-e9d7beea-f667cb85f-44vgg" podUID="1ab7b6de-ada8-4387-917a-940b9735aed8" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.31:8443/health\": dial tcp 10.133.0.31:8443: connect: connection refused" Jul 08 19:55:12.386306 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:12.386255 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:55:12.560119 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:12.560092 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-e9d7beea-f667cb85f-44vgg" Jul 08 19:55:12.645150 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:12.645123 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ab7b6de-ada8-4387-917a-940b9735aed8-maas-api-tls\") pod \"1ab7b6de-ada8-4387-917a-940b9735aed8\" (UID: \"1ab7b6de-ada8-4387-917a-940b9735aed8\") " Jul 08 19:55:12.645287 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:12.645179 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ab7b6de-ada8-4387-917a-940b9735aed8-openshift-service-ca\") pod \"1ab7b6de-ada8-4387-917a-940b9735aed8\" (UID: \"1ab7b6de-ada8-4387-917a-940b9735aed8\") " Jul 08 19:55:12.645287 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:12.645243 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ps6dl\" (UniqueName: \"kubernetes.io/projected/1ab7b6de-ada8-4387-917a-940b9735aed8-kube-api-access-ps6dl\") pod \"1ab7b6de-ada8-4387-917a-940b9735aed8\" (UID: \"1ab7b6de-ada8-4387-917a-940b9735aed8\") " Jul 08 19:55:12.645535 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:12.645506 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ab7b6de-ada8-4387-917a-940b9735aed8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1ab7b6de-ada8-4387-917a-940b9735aed8" (UID: "1ab7b6de-ada8-4387-917a-940b9735aed8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:55:12.647241 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:12.647218 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ab7b6de-ada8-4387-917a-940b9735aed8-kube-api-access-ps6dl" (OuterVolumeSpecName: "kube-api-access-ps6dl") pod "1ab7b6de-ada8-4387-917a-940b9735aed8" (UID: "1ab7b6de-ada8-4387-917a-940b9735aed8"). InnerVolumeSpecName "kube-api-access-ps6dl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:55:12.647320 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:12.647263 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ab7b6de-ada8-4387-917a-940b9735aed8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1ab7b6de-ada8-4387-917a-940b9735aed8" (UID: "1ab7b6de-ada8-4387-917a-940b9735aed8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:55:12.746152 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:12.746124 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ab7b6de-ada8-4387-917a-940b9735aed8-maas-api-tls\") on node \"ip-10-0-142-195.ec2.internal\" DevicePath \"\"" Jul 08 19:55:12.746152 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:12.746153 2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ab7b6de-ada8-4387-917a-940b9735aed8-openshift-service-ca\") on node \"ip-10-0-142-195.ec2.internal\" DevicePath \"\"" Jul 08 19:55:12.746357 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:12.746169 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ps6dl\" (UniqueName: \"kubernetes.io/projected/1ab7b6de-ada8-4387-917a-940b9735aed8-kube-api-access-ps6dl\") on node \"ip-10-0-142-195.ec2.internal\" DevicePath \"\"" Jul 08 19:55:13.392379 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:13.392339 2566 generic.go:356] "Generic (PLEG): container finished" podID="1ab7b6de-ada8-4387-917a-940b9735aed8" containerID="b6908a4092585c60d84b9bbbfca41d35d0d593313ec2f612d92796c72ea28d8c" exitCode=0 Jul 08 19:55:13.392523 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:13.392401 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-e9d7beea-f667cb85f-44vgg" Jul 08 19:55:13.392523 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:13.392403 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-e9d7beea-f667cb85f-44vgg" event={"ID":"1ab7b6de-ada8-4387-917a-940b9735aed8","Type":"ContainerDied","Data":"b6908a4092585c60d84b9bbbfca41d35d0d593313ec2f612d92796c72ea28d8c"} Jul 08 19:55:13.392523 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:13.392454 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-e9d7beea-f667cb85f-44vgg" event={"ID":"1ab7b6de-ada8-4387-917a-940b9735aed8","Type":"ContainerDied","Data":"b573011869d085d60a6107605b6262e1ca3a4607fedc5e364f2b16f51d34d3cd"} Jul 08 19:55:13.392523 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:13.392484 2566 scope.go:117] "RemoveContainer" containerID="b6908a4092585c60d84b9bbbfca41d35d0d593313ec2f612d92796c72ea28d8c" Jul 08 19:55:13.401487 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:13.401468 2566 scope.go:117] "RemoveContainer" containerID="b6908a4092585c60d84b9bbbfca41d35d0d593313ec2f612d92796c72ea28d8c" Jul 08 19:55:13.401742 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:55:13.401721 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6908a4092585c60d84b9bbbfca41d35d0d593313ec2f612d92796c72ea28d8c\": container with ID starting with b6908a4092585c60d84b9bbbfca41d35d0d593313ec2f612d92796c72ea28d8c not found: ID does not exist" containerID="b6908a4092585c60d84b9bbbfca41d35d0d593313ec2f612d92796c72ea28d8c" Jul 08 19:55:13.401803 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:13.401751 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6908a4092585c60d84b9bbbfca41d35d0d593313ec2f612d92796c72ea28d8c"} err="failed to get container status \"b6908a4092585c60d84b9bbbfca41d35d0d593313ec2f612d92796c72ea28d8c\": rpc error: code = NotFound desc = could not find container \"b6908a4092585c60d84b9bbbfca41d35d0d593313ec2f612d92796c72ea28d8c\": container with ID starting with b6908a4092585c60d84b9bbbfca41d35d0d593313ec2f612d92796c72ea28d8c not found: ID does not exist" Jul 08 19:55:13.413425 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:13.413403 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-e9d7beea-f667cb85f-44vgg"] Jul 08 19:55:13.416731 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:13.416713 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-e9d7beea-f667cb85f-44vgg"] Jul 08 19:55:14.623287 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:14.623258 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ab7b6de-ada8-4387-917a-940b9735aed8" path="/var/lib/kubelet/pods/1ab7b6de-ada8-4387-917a-940b9735aed8/volumes" Jul 08 19:55:18.082037 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:18.081952 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:55:18.139622 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:18.139590 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:55:18.350996 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:18.350914 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:55:18.599438 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:18.599403 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:55:18.723126 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:18.723062 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:55:18.823198 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:18.823167 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:55:18.970279 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:18.970242 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:55:20.588425 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:20.588381 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:55:22.947632 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:22.947598 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:55:24.912393 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:24.912359 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:55:24.970332 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:24.970305 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:55:25.046207 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:25.046171 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:55:25.146524 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:25.146497 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:55:25.289591 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:25.289493 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:55:31.189295 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:31.189237 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:55:31.261961 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:31.261933 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:55:33.179619 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:33.179585 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:55:33.283246 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:33.283208 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:55:38.145354 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:38.145313 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:55:38.470173 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:38.470144 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:55:38.625473 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:38.625437 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:55:41.045245 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:41.045210 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:55:41.147733 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:41.147696 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:55:41.202072 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:41.202045 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-c4f984b0-5979f6dff9-kskfv"] Jul 08 19:55:41.202399 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:41.202387 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ab7b6de-ada8-4387-917a-940b9735aed8" containerName="maas-api" Jul 08 19:55:41.202447 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:41.202402 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ab7b6de-ada8-4387-917a-940b9735aed8" containerName="maas-api" Jul 08 19:55:41.202481 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:41.202469 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ab7b6de-ada8-4387-917a-940b9735aed8" containerName="maas-api" Jul 08 19:55:41.205522 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:41.205502 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-c4f984b0-5979f6dff9-kskfv" Jul 08 19:55:41.207935 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:41.207913 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-c4f984b0-serving-cert\"" Jul 08 19:55:41.208061 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:41.208039 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-7kmg8\"" Jul 08 19:55:41.217458 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:41.217435 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-c4f984b0-5979f6dff9-kskfv"] Jul 08 19:55:41.239515 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:41.239483 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:55:41.300248 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:41.300175 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2d8f\" (UniqueName: \"kubernetes.io/projected/438f70da-fb5c-4e55-b14c-b1eacb3301d2-kube-api-access-x2d8f\") pod \"maas-api-e2e-mt-c4f984b0-5979f6dff9-kskfv\" (UID: \"438f70da-fb5c-4e55-b14c-b1eacb3301d2\") " pod="opendatahub/maas-api-e2e-mt-c4f984b0-5979f6dff9-kskfv" Jul 08 19:55:41.300248 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:41.300230 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/438f70da-fb5c-4e55-b14c-b1eacb3301d2-maas-api-tls\") pod \"maas-api-e2e-mt-c4f984b0-5979f6dff9-kskfv\" (UID: \"438f70da-fb5c-4e55-b14c-b1eacb3301d2\") " pod="opendatahub/maas-api-e2e-mt-c4f984b0-5979f6dff9-kskfv" Jul 08 19:55:41.300406 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:41.300259 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/438f70da-fb5c-4e55-b14c-b1eacb3301d2-openshift-service-ca\") pod \"maas-api-e2e-mt-c4f984b0-5979f6dff9-kskfv\" (UID: \"438f70da-fb5c-4e55-b14c-b1eacb3301d2\") " pod="opendatahub/maas-api-e2e-mt-c4f984b0-5979f6dff9-kskfv" Jul 08 19:55:41.401373 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:41.401346 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2d8f\" (UniqueName: \"kubernetes.io/projected/438f70da-fb5c-4e55-b14c-b1eacb3301d2-kube-api-access-x2d8f\") pod \"maas-api-e2e-mt-c4f984b0-5979f6dff9-kskfv\" (UID: \"438f70da-fb5c-4e55-b14c-b1eacb3301d2\") " pod="opendatahub/maas-api-e2e-mt-c4f984b0-5979f6dff9-kskfv" Jul 08 19:55:41.401543 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:41.401388 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/438f70da-fb5c-4e55-b14c-b1eacb3301d2-maas-api-tls\") pod \"maas-api-e2e-mt-c4f984b0-5979f6dff9-kskfv\" (UID: \"438f70da-fb5c-4e55-b14c-b1eacb3301d2\") " pod="opendatahub/maas-api-e2e-mt-c4f984b0-5979f6dff9-kskfv" Jul 08 19:55:41.401543 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:41.401413 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/438f70da-fb5c-4e55-b14c-b1eacb3301d2-openshift-service-ca\") pod \"maas-api-e2e-mt-c4f984b0-5979f6dff9-kskfv\" (UID: \"438f70da-fb5c-4e55-b14c-b1eacb3301d2\") " pod="opendatahub/maas-api-e2e-mt-c4f984b0-5979f6dff9-kskfv" Jul 08 19:55:41.401543 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:55:41.401526 2566 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-mt-c4f984b0-serving-cert: secret "maas-api-e2e-mt-c4f984b0-serving-cert" not found Jul 08 19:55:41.401669 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:55:41.401590 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/438f70da-fb5c-4e55-b14c-b1eacb3301d2-maas-api-tls podName:438f70da-fb5c-4e55-b14c-b1eacb3301d2 nodeName:}" failed. No retries permitted until 2026-07-08 19:55:41.901571374 +0000 UTC m=+1907.787761398 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/438f70da-fb5c-4e55-b14c-b1eacb3301d2-maas-api-tls") pod "maas-api-e2e-mt-c4f984b0-5979f6dff9-kskfv" (UID: "438f70da-fb5c-4e55-b14c-b1eacb3301d2") : secret "maas-api-e2e-mt-c4f984b0-serving-cert" not found Jul 08 19:55:41.402103 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:41.402085 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/438f70da-fb5c-4e55-b14c-b1eacb3301d2-openshift-service-ca\") pod \"maas-api-e2e-mt-c4f984b0-5979f6dff9-kskfv\" (UID: \"438f70da-fb5c-4e55-b14c-b1eacb3301d2\") " pod="opendatahub/maas-api-e2e-mt-c4f984b0-5979f6dff9-kskfv" Jul 08 19:55:41.410455 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:41.410431 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2d8f\" (UniqueName: \"kubernetes.io/projected/438f70da-fb5c-4e55-b14c-b1eacb3301d2-kube-api-access-x2d8f\") pod \"maas-api-e2e-mt-c4f984b0-5979f6dff9-kskfv\" (UID: \"438f70da-fb5c-4e55-b14c-b1eacb3301d2\") " pod="opendatahub/maas-api-e2e-mt-c4f984b0-5979f6dff9-kskfv" Jul 08 19:55:41.698470 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:41.698438 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:55:41.801677 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:41.801649 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:55:41.858874 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:41.858818 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:55:41.906617 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:41.906586 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/438f70da-fb5c-4e55-b14c-b1eacb3301d2-maas-api-tls\") pod \"maas-api-e2e-mt-c4f984b0-5979f6dff9-kskfv\" (UID: \"438f70da-fb5c-4e55-b14c-b1eacb3301d2\") " pod="opendatahub/maas-api-e2e-mt-c4f984b0-5979f6dff9-kskfv" Jul 08 19:55:41.908991 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:41.908974 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/438f70da-fb5c-4e55-b14c-b1eacb3301d2-maas-api-tls\") pod \"maas-api-e2e-mt-c4f984b0-5979f6dff9-kskfv\" (UID: \"438f70da-fb5c-4e55-b14c-b1eacb3301d2\") " pod="opendatahub/maas-api-e2e-mt-c4f984b0-5979f6dff9-kskfv" Jul 08 19:55:42.116528 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:42.116441 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-c4f984b0-5979f6dff9-kskfv" Jul 08 19:55:42.245692 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:42.245668 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-c4f984b0-5979f6dff9-kskfv"] Jul 08 19:55:42.247902 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:55:42.247876 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod438f70da_fb5c_4e55_b14c_b1eacb3301d2.slice/crio-1892c723ca0075f60029e2f4416bfc5321c49f2eae903092a77c3b6d70eb9c09 WatchSource:0}: Error finding container 1892c723ca0075f60029e2f4416bfc5321c49f2eae903092a77c3b6d70eb9c09: Status 404 returned error can't find the container with id 1892c723ca0075f60029e2f4416bfc5321c49f2eae903092a77c3b6d70eb9c09 Jul 08 19:55:42.529461 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:42.529431 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-c4f984b0-5979f6dff9-kskfv" event={"ID":"438f70da-fb5c-4e55-b14c-b1eacb3301d2","Type":"ContainerStarted","Data":"1892c723ca0075f60029e2f4416bfc5321c49f2eae903092a77c3b6d70eb9c09"} Jul 08 19:55:43.534922 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:43.534880 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-c4f984b0-5979f6dff9-kskfv" event={"ID":"438f70da-fb5c-4e55-b14c-b1eacb3301d2","Type":"ContainerStarted","Data":"07e4b25249e2814df35369c7cf099863715223027521f7025e312013199c9845"} Jul 08 19:55:43.535332 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:43.534947 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-c4f984b0-5979f6dff9-kskfv" Jul 08 19:55:43.552771 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:43.552724 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-c4f984b0-5979f6dff9-kskfv" podStartSLOduration=2.136238995 podStartE2EDuration="2.552707809s" totalImagesPullingTime="416.468814ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:55:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:55:43.551227169 +0000 UTC m=+1909.437417243" watchObservedRunningTime="2026-07-08 19:55:43.552707809 +0000 UTC m=+1909.438897855" Jul 08 19:55:43.687646 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:43.687610 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:55:45.610546 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:45.610517 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:55:49.544808 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:49.544778 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-c4f984b0-5979f6dff9-kskfv" Jul 08 19:55:59.113309 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:55:59.113275 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:56:02.387516 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:56:02.387482 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:56:16.491773 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:56:16.491733 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:56:18.057046 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:56:18.057004 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:56:18.436694 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:56:18.436660 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-c4f984b0-5979f6dff9-kskfv"] Jul 08 19:56:18.437006 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:56:18.436974 2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-c4f984b0-5979f6dff9-kskfv" podUID="438f70da-fb5c-4e55-b14c-b1eacb3301d2" containerName="maas-api" containerID="cri-o://07e4b25249e2814df35369c7cf099863715223027521f7025e312013199c9845" gracePeriod=30 Jul 08 19:56:18.556604 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:56:18.556573 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:56:18.678409 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:56:18.678383 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-c4f984b0-5979f6dff9-kskfv" Jul 08 19:56:18.691120 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:56:18.691062 2566 generic.go:356] "Generic (PLEG): container finished" podID="438f70da-fb5c-4e55-b14c-b1eacb3301d2" containerID="07e4b25249e2814df35369c7cf099863715223027521f7025e312013199c9845" exitCode=0 Jul 08 19:56:18.691120 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:56:18.691099 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-c4f984b0-5979f6dff9-kskfv" Jul 08 19:56:18.691120 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:56:18.691114 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-c4f984b0-5979f6dff9-kskfv" event={"ID":"438f70da-fb5c-4e55-b14c-b1eacb3301d2","Type":"ContainerDied","Data":"07e4b25249e2814df35369c7cf099863715223027521f7025e312013199c9845"} Jul 08 19:56:18.691300 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:56:18.691154 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-c4f984b0-5979f6dff9-kskfv" event={"ID":"438f70da-fb5c-4e55-b14c-b1eacb3301d2","Type":"ContainerDied","Data":"1892c723ca0075f60029e2f4416bfc5321c49f2eae903092a77c3b6d70eb9c09"} Jul 08 19:56:18.691300 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:56:18.691175 2566 scope.go:117] "RemoveContainer" containerID="07e4b25249e2814df35369c7cf099863715223027521f7025e312013199c9845" Jul 08 19:56:18.701310 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:56:18.701290 2566 scope.go:117] "RemoveContainer" containerID="07e4b25249e2814df35369c7cf099863715223027521f7025e312013199c9845" Jul 08 19:56:18.701576 ip-10-0-142-195 kubenswrapper[2566]: E0708 19:56:18.701558 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07e4b25249e2814df35369c7cf099863715223027521f7025e312013199c9845\": container with ID starting with 07e4b25249e2814df35369c7cf099863715223027521f7025e312013199c9845 not found: ID does not exist" containerID="07e4b25249e2814df35369c7cf099863715223027521f7025e312013199c9845" Jul 08 19:56:18.701635 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:56:18.701586 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07e4b25249e2814df35369c7cf099863715223027521f7025e312013199c9845"} err="failed to get container status \"07e4b25249e2814df35369c7cf099863715223027521f7025e312013199c9845\": rpc error: code = NotFound desc = could not find container \"07e4b25249e2814df35369c7cf099863715223027521f7025e312013199c9845\": container with ID starting with 07e4b25249e2814df35369c7cf099863715223027521f7025e312013199c9845 not found: ID does not exist" Jul 08 19:56:18.824160 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:56:18.824129 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x2d8f\" (UniqueName: \"kubernetes.io/projected/438f70da-fb5c-4e55-b14c-b1eacb3301d2-kube-api-access-x2d8f\") pod \"438f70da-fb5c-4e55-b14c-b1eacb3301d2\" (UID: \"438f70da-fb5c-4e55-b14c-b1eacb3301d2\") " Jul 08 19:56:18.824327 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:56:18.824177 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/438f70da-fb5c-4e55-b14c-b1eacb3301d2-openshift-service-ca\") pod \"438f70da-fb5c-4e55-b14c-b1eacb3301d2\" (UID: \"438f70da-fb5c-4e55-b14c-b1eacb3301d2\") " Jul 08 19:56:18.824327 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:56:18.824198 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/438f70da-fb5c-4e55-b14c-b1eacb3301d2-maas-api-tls\") pod \"438f70da-fb5c-4e55-b14c-b1eacb3301d2\" (UID: \"438f70da-fb5c-4e55-b14c-b1eacb3301d2\") " Jul 08 19:56:18.824635 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:56:18.824599 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/438f70da-fb5c-4e55-b14c-b1eacb3301d2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "438f70da-fb5c-4e55-b14c-b1eacb3301d2" (UID: "438f70da-fb5c-4e55-b14c-b1eacb3301d2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:56:18.826310 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:56:18.826286 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/438f70da-fb5c-4e55-b14c-b1eacb3301d2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "438f70da-fb5c-4e55-b14c-b1eacb3301d2" (UID: "438f70da-fb5c-4e55-b14c-b1eacb3301d2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:56:18.826469 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:56:18.826447 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/438f70da-fb5c-4e55-b14c-b1eacb3301d2-kube-api-access-x2d8f" (OuterVolumeSpecName: "kube-api-access-x2d8f") pod "438f70da-fb5c-4e55-b14c-b1eacb3301d2" (UID: "438f70da-fb5c-4e55-b14c-b1eacb3301d2"). InnerVolumeSpecName "kube-api-access-x2d8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:56:18.925150 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:56:18.925119 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2d8f\" (UniqueName: \"kubernetes.io/projected/438f70da-fb5c-4e55-b14c-b1eacb3301d2-kube-api-access-x2d8f\") on node \"ip-10-0-142-195.ec2.internal\" DevicePath \"\"" Jul 08 19:56:18.925150 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:56:18.925144 2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/438f70da-fb5c-4e55-b14c-b1eacb3301d2-openshift-service-ca\") on node \"ip-10-0-142-195.ec2.internal\" DevicePath \"\"" Jul 08 19:56:18.925150 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:56:18.925154 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/438f70da-fb5c-4e55-b14c-b1eacb3301d2-maas-api-tls\") on node \"ip-10-0-142-195.ec2.internal\" DevicePath \"\"" Jul 08 19:56:19.011904 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:56:19.011877 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-c4f984b0-5979f6dff9-kskfv"] Jul 08 19:56:19.015854 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:56:19.015832 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-c4f984b0-5979f6dff9-kskfv"] Jul 08 19:56:20.622849 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:56:20.622819 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="438f70da-fb5c-4e55-b14c-b1eacb3301d2" path="/var/lib/kubelet/pods/438f70da-fb5c-4e55-b14c-b1eacb3301d2/volumes" Jul 08 19:56:24.199606 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:56:24.199563 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:56:28.999927 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:56:28.999887 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:56:29.258681 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:56:29.258600 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:57:06.280696 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:57:06.280651 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:57:06.550466 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:57:06.550388 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:57:08.795781 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:57:08.795746 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:57:12.690653 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:57:12.690619 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:57:36.641590 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:57:36.641549 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:57:36.742579 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:57:36.742542 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:57:36.944296 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:57:36.944263 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:57:37.222064 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:57:37.221970 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:57:38.341402 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:57:38.341367 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:57:38.435542 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:57:38.435506 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:57:38.532711 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:57:38.532674 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:57:39.345847 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:57:39.345812 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:57:39.439509 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:57:39.439472 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:57:39.539555 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:57:39.539517 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:57:40.601218 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:57:40.601183 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:57:41.056685 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:57:41.056641 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:57:47.112651 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:57:47.112617 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:57:47.174619 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:57:47.174594 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:57:47.819348 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:57:47.819310 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:57:53.851597 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:57:53.851563 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:57:53.958971 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:57:53.958936 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:57:54.053103 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:57:54.053075 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:57:54.889567 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:57:54.889530 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:58:04.587586 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:58:04.587551 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:58:04.949016 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:58:04.948976 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:58:25.919879 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:58:25.919787 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:58:29.968075 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:58:29.968032 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:58:32.151138 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:58:32.151103 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:58:32.247166 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:58:32.247132 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:58:32.478424 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:58:32.478392 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:58:32.863018 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:58:32.862939 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:58:33.008685 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:58:33.008651 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:58:36.039890 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:58:36.039833 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:58:36.138632 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:58:36.138590 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:58:36.209402 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:58:36.209371 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-a-d7248b-558577c4fd-fdd78"] Jul 08 19:58:36.209784 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:58:36.209766 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="438f70da-fb5c-4e55-b14c-b1eacb3301d2" containerName="maas-api" Jul 08 19:58:36.209901 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:58:36.209787 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="438f70da-fb5c-4e55-b14c-b1eacb3301d2" containerName="maas-api" Jul 08 19:58:36.209967 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:58:36.209921 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="438f70da-fb5c-4e55-b14c-b1eacb3301d2" containerName="maas-api" Jul 08 19:58:36.212830 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:58:36.212810 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-d7248b-558577c4fd-fdd78" Jul 08 19:58:36.215343 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:58:36.215325 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-a-d7248b-serving-cert\"" Jul 08 19:58:36.215565 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:58:36.215552 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-7kmg8\"" Jul 08 19:58:36.226170 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:58:36.226139 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-d7248b-558577c4fd-fdd78"] Jul 08 19:58:36.255364 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:58:36.255340 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:58:36.330022 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:58:36.329953 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c7041e6-8d1a-42c3-8a7a-ca027d924a4b-maas-api-tls\") pod \"maas-api-e2e-inf-a-d7248b-558577c4fd-fdd78\" (UID: \"7c7041e6-8d1a-42c3-8a7a-ca027d924a4b\") " pod="opendatahub/maas-api-e2e-inf-a-d7248b-558577c4fd-fdd78" Jul 08 19:58:36.330022 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:58:36.330015 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wd8p\" (UniqueName: \"kubernetes.io/projected/7c7041e6-8d1a-42c3-8a7a-ca027d924a4b-kube-api-access-8wd8p\") pod \"maas-api-e2e-inf-a-d7248b-558577c4fd-fdd78\" (UID: \"7c7041e6-8d1a-42c3-8a7a-ca027d924a4b\") " pod="opendatahub/maas-api-e2e-inf-a-d7248b-558577c4fd-fdd78" Jul 08 19:58:36.330176 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:58:36.330104 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c7041e6-8d1a-42c3-8a7a-ca027d924a4b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d7248b-558577c4fd-fdd78\" (UID: \"7c7041e6-8d1a-42c3-8a7a-ca027d924a4b\") " pod="opendatahub/maas-api-e2e-inf-a-d7248b-558577c4fd-fdd78" Jul 08 19:58:36.353468 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:58:36.353437 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:58:36.431287 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:58:36.431256 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c7041e6-8d1a-42c3-8a7a-ca027d924a4b-maas-api-tls\") pod \"maas-api-e2e-inf-a-d7248b-558577c4fd-fdd78\" (UID: \"7c7041e6-8d1a-42c3-8a7a-ca027d924a4b\") " pod="opendatahub/maas-api-e2e-inf-a-d7248b-558577c4fd-fdd78" Jul 08 19:58:36.431432 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:58:36.431309 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wd8p\" (UniqueName: \"kubernetes.io/projected/7c7041e6-8d1a-42c3-8a7a-ca027d924a4b-kube-api-access-8wd8p\") pod \"maas-api-e2e-inf-a-d7248b-558577c4fd-fdd78\" (UID: \"7c7041e6-8d1a-42c3-8a7a-ca027d924a4b\") " pod="opendatahub/maas-api-e2e-inf-a-d7248b-558577c4fd-fdd78" Jul 08 19:58:36.431432 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:58:36.431344 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c7041e6-8d1a-42c3-8a7a-ca027d924a4b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d7248b-558577c4fd-fdd78\" (UID: \"7c7041e6-8d1a-42c3-8a7a-ca027d924a4b\") " pod="opendatahub/maas-api-e2e-inf-a-d7248b-558577c4fd-fdd78" Jul 08 19:58:36.431954 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:58:36.431935 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c7041e6-8d1a-42c3-8a7a-ca027d924a4b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d7248b-558577c4fd-fdd78\" (UID: \"7c7041e6-8d1a-42c3-8a7a-ca027d924a4b\") " pod="opendatahub/maas-api-e2e-inf-a-d7248b-558577c4fd-fdd78" Jul 08 19:58:36.433688 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:58:36.433665 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c7041e6-8d1a-42c3-8a7a-ca027d924a4b-maas-api-tls\") pod \"maas-api-e2e-inf-a-d7248b-558577c4fd-fdd78\" (UID: \"7c7041e6-8d1a-42c3-8a7a-ca027d924a4b\") " pod="opendatahub/maas-api-e2e-inf-a-d7248b-558577c4fd-fdd78" Jul 08 19:58:36.440098 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:58:36.440077 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wd8p\" (UniqueName: \"kubernetes.io/projected/7c7041e6-8d1a-42c3-8a7a-ca027d924a4b-kube-api-access-8wd8p\") pod \"maas-api-e2e-inf-a-d7248b-558577c4fd-fdd78\" (UID: \"7c7041e6-8d1a-42c3-8a7a-ca027d924a4b\") " pod="opendatahub/maas-api-e2e-inf-a-d7248b-558577c4fd-fdd78" Jul 08 19:58:36.523282 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:58:36.523253 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-d7248b-558577c4fd-fdd78" Jul 08 19:58:36.660364 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:58:36.660335 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-d7248b-558577c4fd-fdd78"] Jul 08 19:58:36.660893 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:58:36.660846 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c7041e6_8d1a_42c3_8a7a_ca027d924a4b.slice/crio-60cab00057e6163dfe6ea2b187e45520061bffd8bffef21cce39113d3ef317b9 WatchSource:0}: Error finding container 60cab00057e6163dfe6ea2b187e45520061bffd8bffef21cce39113d3ef317b9: Status 404 returned error can't find the container with id 60cab00057e6163dfe6ea2b187e45520061bffd8bffef21cce39113d3ef317b9 Jul 08 19:58:36.662244 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:58:36.662225 2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 19:58:36.758610 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:58:36.758573 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:58:36.861509 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:58:36.861425 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:58:37.316503 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:58:37.316460 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-d7248b-558577c4fd-fdd78" event={"ID":"7c7041e6-8d1a-42c3-8a7a-ca027d924a4b","Type":"ContainerStarted","Data":"4ad36e6c45c8291e3533d0aa52915d5866cf69c32db2119852ac971985dc9fce"} Jul 08 19:58:37.316854 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:58:37.316512 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-d7248b-558577c4fd-fdd78" event={"ID":"7c7041e6-8d1a-42c3-8a7a-ca027d924a4b","Type":"ContainerStarted","Data":"60cab00057e6163dfe6ea2b187e45520061bffd8bffef21cce39113d3ef317b9"} Jul 08 19:58:37.316854 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:58:37.316590 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-a-d7248b-558577c4fd-fdd78" Jul 08 19:58:37.337674 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:58:37.337630 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-a-d7248b-558577c4fd-fdd78" podStartSLOduration=0.86534026 podStartE2EDuration="1.337617139s" totalImagesPullingTime="472.276879ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:58:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:58:37.335681023 +0000 UTC m=+2083.221871101" watchObservedRunningTime="2026-07-08 19:58:37.337617139 +0000 UTC m=+2083.223807186" Jul 08 19:58:40.535291 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:58:40.535258 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:58:40.650746 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:58:40.650710 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:58:40.892276 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:58:40.892199 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:58:41.215344 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:58:41.215315 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:58:41.341496 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:58:41.341454 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:58:43.327084 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:58:43.327056 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-a-d7248b-558577c4fd-fdd78" Jul 08 19:58:43.496090 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:58:43.496058 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:58:44.087903 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:58:44.087846 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:58:44.167811 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:58:44.167777 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-b-deba89-9bfb4c795-2jv94"] Jul 08 19:58:44.172139 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:58:44.172117 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-deba89-9bfb4c795-2jv94" Jul 08 19:58:44.174947 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:58:44.174923 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-b-deba89-serving-cert\"" Jul 08 19:58:44.177606 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:58:44.177583 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:58:44.183666 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:58:44.183560 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-deba89-9bfb4c795-2jv94"] Jul 08 19:58:44.255916 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:58:44.255885 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:58:44.301440 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:58:44.301409 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg9fz\" (UniqueName: \"kubernetes.io/projected/13afbd86-76cb-4d29-915b-f18bcd3a58df-kube-api-access-wg9fz\") pod \"maas-api-e2e-inf-b-deba89-9bfb4c795-2jv94\" (UID: \"13afbd86-76cb-4d29-915b-f18bcd3a58df\") " pod="opendatahub/maas-api-e2e-inf-b-deba89-9bfb4c795-2jv94" Jul 08 19:58:44.301649 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:58:44.301461 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13afbd86-76cb-4d29-915b-f18bcd3a58df-maas-api-tls\") pod \"maas-api-e2e-inf-b-deba89-9bfb4c795-2jv94\" (UID: \"13afbd86-76cb-4d29-915b-f18bcd3a58df\") " pod="opendatahub/maas-api-e2e-inf-b-deba89-9bfb4c795-2jv94" Jul 08 19:58:44.301649 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:58:44.301533 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13afbd86-76cb-4d29-915b-f18bcd3a58df-openshift-service-ca\") pod \"maas-api-e2e-inf-b-deba89-9bfb4c795-2jv94\" (UID: \"13afbd86-76cb-4d29-915b-f18bcd3a58df\") " pod="opendatahub/maas-api-e2e-inf-b-deba89-9bfb4c795-2jv94" Jul 08 19:58:44.354305 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:58:44.354229 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:58:44.402584 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:58:44.402557 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13afbd86-76cb-4d29-915b-f18bcd3a58df-maas-api-tls\") pod \"maas-api-e2e-inf-b-deba89-9bfb4c795-2jv94\" (UID: \"13afbd86-76cb-4d29-915b-f18bcd3a58df\") " pod="opendatahub/maas-api-e2e-inf-b-deba89-9bfb4c795-2jv94" Jul 08 19:58:44.402697 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:58:44.402623 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13afbd86-76cb-4d29-915b-f18bcd3a58df-openshift-service-ca\") pod \"maas-api-e2e-inf-b-deba89-9bfb4c795-2jv94\" (UID: \"13afbd86-76cb-4d29-915b-f18bcd3a58df\") " pod="opendatahub/maas-api-e2e-inf-b-deba89-9bfb4c795-2jv94" Jul 08 19:58:44.402697 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:58:44.402686 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg9fz\" (UniqueName: \"kubernetes.io/projected/13afbd86-76cb-4d29-915b-f18bcd3a58df-kube-api-access-wg9fz\") pod \"maas-api-e2e-inf-b-deba89-9bfb4c795-2jv94\" (UID: \"13afbd86-76cb-4d29-915b-f18bcd3a58df\") " pod="opendatahub/maas-api-e2e-inf-b-deba89-9bfb4c795-2jv94" Jul 08 19:58:44.403329 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:58:44.403310 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13afbd86-76cb-4d29-915b-f18bcd3a58df-openshift-service-ca\") pod \"maas-api-e2e-inf-b-deba89-9bfb4c795-2jv94\" (UID: \"13afbd86-76cb-4d29-915b-f18bcd3a58df\") " pod="opendatahub/maas-api-e2e-inf-b-deba89-9bfb4c795-2jv94" Jul 08 19:58:44.404981 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:58:44.404959 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13afbd86-76cb-4d29-915b-f18bcd3a58df-maas-api-tls\") pod \"maas-api-e2e-inf-b-deba89-9bfb4c795-2jv94\" (UID: \"13afbd86-76cb-4d29-915b-f18bcd3a58df\") " pod="opendatahub/maas-api-e2e-inf-b-deba89-9bfb4c795-2jv94" Jul 08 19:58:44.411567 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:58:44.411532 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg9fz\" (UniqueName: \"kubernetes.io/projected/13afbd86-76cb-4d29-915b-f18bcd3a58df-kube-api-access-wg9fz\") pod \"maas-api-e2e-inf-b-deba89-9bfb4c795-2jv94\" (UID: \"13afbd86-76cb-4d29-915b-f18bcd3a58df\") " pod="opendatahub/maas-api-e2e-inf-b-deba89-9bfb4c795-2jv94" Jul 08 19:58:44.485611 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:58:44.485565 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-deba89-9bfb4c795-2jv94" Jul 08 19:58:44.623089 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:58:44.623057 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13afbd86_76cb_4d29_915b_f18bcd3a58df.slice/crio-51ff97876a45a19d021d3a5ffff16dfc533fa76e62f3711b248c928c7f4d0490 WatchSource:0}: Error finding container 51ff97876a45a19d021d3a5ffff16dfc533fa76e62f3711b248c928c7f4d0490: Status 404 returned error can't find the container with id 51ff97876a45a19d021d3a5ffff16dfc533fa76e62f3711b248c928c7f4d0490 Jul 08 19:58:44.625414 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:58:44.625394 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-deba89-9bfb4c795-2jv94"] Jul 08 19:58:44.659776 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:58:44.659750 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:58:44.760235 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:58:44.760195 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:58:45.361137 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:58:45.361056 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-deba89-9bfb4c795-2jv94" event={"ID":"13afbd86-76cb-4d29-915b-f18bcd3a58df","Type":"ContainerStarted","Data":"2004a6e58d3bd318c188d8a4bacbb6ab94f832da6cf66413b52e4aae8e54f08c"} Jul 08 19:58:45.361137 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:58:45.361107 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-deba89-9bfb4c795-2jv94" event={"ID":"13afbd86-76cb-4d29-915b-f18bcd3a58df","Type":"ContainerStarted","Data":"51ff97876a45a19d021d3a5ffff16dfc533fa76e62f3711b248c928c7f4d0490"} Jul 08 19:58:45.361524 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:58:45.361143 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-b-deba89-9bfb4c795-2jv94" Jul 08 19:58:45.379541 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:58:45.379493 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-b-deba89-9bfb4c795-2jv94" podStartSLOduration=1.067608292 podStartE2EDuration="1.37947902s" totalImagesPullingTime="311.870728ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:58:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:58:45.378523964 +0000 UTC m=+2091.264713993" watchObservedRunningTime="2026-07-08 19:58:45.37947902 +0000 UTC m=+2091.265669057" Jul 08 19:58:48.592955 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:58:48.592903 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:58:51.370761 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:58:51.370732 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-b-deba89-9bfb4c795-2jv94" Jul 08 19:58:57.939567 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:58:57.939532 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:58:58.710090 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:58:58.710055 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:58:58.805373 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:58:58.805341 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:59:05.510881 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:59:05.510840 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:59:06.810834 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:59:06.810794 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:59:24.234089 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:59:24.234051 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:59:27.791260 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:59:27.791223 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 19:59:53.993120 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:59:53.993025 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-deba89/test-model-deba89-kserve-9d886787f-kwptp"] Jul 08 19:59:53.996506 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:59:53.996486 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-deba89/test-model-deba89-kserve-9d886787f-kwptp" Jul 08 19:59:53.999245 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:59:53.999220 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-deba89\"/\"openshift-service-ca.crt\"" Jul 08 19:59:54.000259 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:59:54.000233 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-deba89\"/\"default-dockercfg-fdlv7\"" Jul 08 19:59:54.000413 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:59:54.000383 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-deba89\"/\"test-model-deba89-kserve-self-signed-certs\"" Jul 08 19:59:54.000552 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:59:54.000534 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-deba89\"/\"kube-root-ca.crt\"" Jul 08 19:59:54.006823 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:59:54.006797 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-deba89/test-model-deba89-kserve-9d886787f-kwptp"] Jul 08 19:59:54.023529 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:59:54.023509 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 19:59:54.116082 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:59:54.116047 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0ed9d11-b667-44ab-a844-58b9e3105a9f-tmp-dir\") pod \"test-model-deba89-kserve-9d886787f-kwptp\" (UID: \"e0ed9d11-b667-44ab-a844-58b9e3105a9f\") " pod="ai-tenant-e2e-inf-b-deba89/test-model-deba89-kserve-9d886787f-kwptp" Jul 08 19:59:54.116232 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:59:54.116093 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0ed9d11-b667-44ab-a844-58b9e3105a9f-home\") pod \"test-model-deba89-kserve-9d886787f-kwptp\" (UID: \"e0ed9d11-b667-44ab-a844-58b9e3105a9f\") " pod="ai-tenant-e2e-inf-b-deba89/test-model-deba89-kserve-9d886787f-kwptp" Jul 08 19:59:54.116232 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:59:54.116113 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0ed9d11-b667-44ab-a844-58b9e3105a9f-model-cache\") pod \"test-model-deba89-kserve-9d886787f-kwptp\" (UID: \"e0ed9d11-b667-44ab-a844-58b9e3105a9f\") " pod="ai-tenant-e2e-inf-b-deba89/test-model-deba89-kserve-9d886787f-kwptp" Jul 08 19:59:54.116232 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:59:54.116144 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0ed9d11-b667-44ab-a844-58b9e3105a9f-tls-certs\") pod \"test-model-deba89-kserve-9d886787f-kwptp\" (UID: \"e0ed9d11-b667-44ab-a844-58b9e3105a9f\") " pod="ai-tenant-e2e-inf-b-deba89/test-model-deba89-kserve-9d886787f-kwptp" Jul 08 19:59:54.116232 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:59:54.116177 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0ed9d11-b667-44ab-a844-58b9e3105a9f-dshm\") pod \"test-model-deba89-kserve-9d886787f-kwptp\" (UID: \"e0ed9d11-b667-44ab-a844-58b9e3105a9f\") " pod="ai-tenant-e2e-inf-b-deba89/test-model-deba89-kserve-9d886787f-kwptp" Jul 08 19:59:54.116232 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:59:54.116207 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4kc2\" (UniqueName: \"kubernetes.io/projected/e0ed9d11-b667-44ab-a844-58b9e3105a9f-kube-api-access-f4kc2\") pod \"test-model-deba89-kserve-9d886787f-kwptp\" (UID: \"e0ed9d11-b667-44ab-a844-58b9e3105a9f\") " pod="ai-tenant-e2e-inf-b-deba89/test-model-deba89-kserve-9d886787f-kwptp" Jul 08 19:59:54.217479 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:59:54.217449 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0ed9d11-b667-44ab-a844-58b9e3105a9f-dshm\") pod \"test-model-deba89-kserve-9d886787f-kwptp\" (UID: \"e0ed9d11-b667-44ab-a844-58b9e3105a9f\") " pod="ai-tenant-e2e-inf-b-deba89/test-model-deba89-kserve-9d886787f-kwptp" Jul 08 19:59:54.217632 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:59:54.217491 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4kc2\" (UniqueName: \"kubernetes.io/projected/e0ed9d11-b667-44ab-a844-58b9e3105a9f-kube-api-access-f4kc2\") pod \"test-model-deba89-kserve-9d886787f-kwptp\" (UID: \"e0ed9d11-b667-44ab-a844-58b9e3105a9f\") " pod="ai-tenant-e2e-inf-b-deba89/test-model-deba89-kserve-9d886787f-kwptp" Jul 08 19:59:54.217632 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:59:54.217610 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0ed9d11-b667-44ab-a844-58b9e3105a9f-tmp-dir\") pod \"test-model-deba89-kserve-9d886787f-kwptp\" (UID: \"e0ed9d11-b667-44ab-a844-58b9e3105a9f\") " pod="ai-tenant-e2e-inf-b-deba89/test-model-deba89-kserve-9d886787f-kwptp" Jul 08 19:59:54.217710 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:59:54.217654 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0ed9d11-b667-44ab-a844-58b9e3105a9f-home\") pod \"test-model-deba89-kserve-9d886787f-kwptp\" (UID: \"e0ed9d11-b667-44ab-a844-58b9e3105a9f\") " pod="ai-tenant-e2e-inf-b-deba89/test-model-deba89-kserve-9d886787f-kwptp" Jul 08 19:59:54.217710 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:59:54.217678 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0ed9d11-b667-44ab-a844-58b9e3105a9f-model-cache\") pod \"test-model-deba89-kserve-9d886787f-kwptp\" (UID: \"e0ed9d11-b667-44ab-a844-58b9e3105a9f\") " pod="ai-tenant-e2e-inf-b-deba89/test-model-deba89-kserve-9d886787f-kwptp" Jul 08 19:59:54.217776 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:59:54.217709 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0ed9d11-b667-44ab-a844-58b9e3105a9f-tls-certs\") pod \"test-model-deba89-kserve-9d886787f-kwptp\" (UID: \"e0ed9d11-b667-44ab-a844-58b9e3105a9f\") " pod="ai-tenant-e2e-inf-b-deba89/test-model-deba89-kserve-9d886787f-kwptp" Jul 08 19:59:54.218015 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:59:54.217994 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0ed9d11-b667-44ab-a844-58b9e3105a9f-tmp-dir\") pod \"test-model-deba89-kserve-9d886787f-kwptp\" (UID: \"e0ed9d11-b667-44ab-a844-58b9e3105a9f\") " pod="ai-tenant-e2e-inf-b-deba89/test-model-deba89-kserve-9d886787f-kwptp" Jul 08 19:59:54.218098 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:59:54.218050 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0ed9d11-b667-44ab-a844-58b9e3105a9f-home\") pod \"test-model-deba89-kserve-9d886787f-kwptp\" (UID: \"e0ed9d11-b667-44ab-a844-58b9e3105a9f\") " pod="ai-tenant-e2e-inf-b-deba89/test-model-deba89-kserve-9d886787f-kwptp" Jul 08 19:59:54.218098 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:59:54.218066 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0ed9d11-b667-44ab-a844-58b9e3105a9f-model-cache\") pod \"test-model-deba89-kserve-9d886787f-kwptp\" (UID: \"e0ed9d11-b667-44ab-a844-58b9e3105a9f\") " pod="ai-tenant-e2e-inf-b-deba89/test-model-deba89-kserve-9d886787f-kwptp" Jul 08 19:59:54.219899 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:59:54.219849 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0ed9d11-b667-44ab-a844-58b9e3105a9f-dshm\") pod \"test-model-deba89-kserve-9d886787f-kwptp\" (UID: \"e0ed9d11-b667-44ab-a844-58b9e3105a9f\") " pod="ai-tenant-e2e-inf-b-deba89/test-model-deba89-kserve-9d886787f-kwptp" Jul 08 19:59:54.220098 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:59:54.220080 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0ed9d11-b667-44ab-a844-58b9e3105a9f-tls-certs\") pod \"test-model-deba89-kserve-9d886787f-kwptp\" (UID: \"e0ed9d11-b667-44ab-a844-58b9e3105a9f\") " pod="ai-tenant-e2e-inf-b-deba89/test-model-deba89-kserve-9d886787f-kwptp" Jul 08 19:59:54.229946 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:59:54.229924 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4kc2\" (UniqueName: \"kubernetes.io/projected/e0ed9d11-b667-44ab-a844-58b9e3105a9f-kube-api-access-f4kc2\") pod \"test-model-deba89-kserve-9d886787f-kwptp\" (UID: \"e0ed9d11-b667-44ab-a844-58b9e3105a9f\") " pod="ai-tenant-e2e-inf-b-deba89/test-model-deba89-kserve-9d886787f-kwptp" Jul 08 19:59:54.307920 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:59:54.307833 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-deba89/test-model-deba89-kserve-9d886787f-kwptp" Jul 08 19:59:54.439648 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:59:54.439627 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-deba89/test-model-deba89-kserve-9d886787f-kwptp"] Jul 08 19:59:54.441560 ip-10-0-142-195 kubenswrapper[2566]: W0708 19:59:54.441536 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode0ed9d11_b667_44ab_a844_58b9e3105a9f.slice/crio-3496858adcff1d70a636be59c4ecd2ffcaccd881dfa4448874fc1649dc0012fa WatchSource:0}: Error finding container 3496858adcff1d70a636be59c4ecd2ffcaccd881dfa4448874fc1649dc0012fa: Status 404 returned error can't find the container with id 3496858adcff1d70a636be59c4ecd2ffcaccd881dfa4448874fc1649dc0012fa Jul 08 19:59:54.679957 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:59:54.679926 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-deba89/test-model-deba89-kserve-9d886787f-kwptp" event={"ID":"e0ed9d11-b667-44ab-a844-58b9e3105a9f","Type":"ContainerStarted","Data":"10a74edcb3a795d74c0f8c9a4bcfebc77d0f74ae7a17d0ba66c5a729e09b2a14"} Jul 08 19:59:54.680107 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:59:54.679968 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-deba89/test-model-deba89-kserve-9d886787f-kwptp" event={"ID":"e0ed9d11-b667-44ab-a844-58b9e3105a9f","Type":"ContainerStarted","Data":"3496858adcff1d70a636be59c4ecd2ffcaccd881dfa4448874fc1649dc0012fa"} Jul 08 19:59:54.701044 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:59:54.700996 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-deba89/test-model-deba89-kserve-9d886787f-kwptp" podStartSLOduration=1.700980665 podStartE2EDuration="1.700980665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:59:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:59:54.698991337 +0000 UTC m=+2160.585181384" watchObservedRunningTime="2026-07-08 19:59:54.700980665 +0000 UTC m=+2160.587170711" Jul 08 19:59:58.103836 ip-10-0-142-195 kubenswrapper[2566]: I0708 19:59:58.103801 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 20:00:00.135247 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:00.135202 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-inf-a-d7248b-29725680-bdzhj"] Jul 08 20:00:00.138645 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:00.138627 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-inf-a-d7248b-29725680-bdzhj" Jul 08 20:00:00.149520 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:00.149494 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-inf-a-d7248b-29725680-bdzhj"] Jul 08 20:00:00.170303 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:00.170278 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x45j9\" (UniqueName: \"kubernetes.io/projected/3a70713e-2dc0-45d4-9660-ef52241d3684-kube-api-access-x45j9\") pod \"maas-api-key-cleanup-e2e-inf-a-d7248b-29725680-bdzhj\" (UID: \"3a70713e-2dc0-45d4-9660-ef52241d3684\") " pod="opendatahub/maas-api-key-cleanup-e2e-inf-a-d7248b-29725680-bdzhj" Jul 08 20:00:00.271321 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:00.271282 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x45j9\" (UniqueName: \"kubernetes.io/projected/3a70713e-2dc0-45d4-9660-ef52241d3684-kube-api-access-x45j9\") pod \"maas-api-key-cleanup-e2e-inf-a-d7248b-29725680-bdzhj\" (UID: \"3a70713e-2dc0-45d4-9660-ef52241d3684\") " pod="opendatahub/maas-api-key-cleanup-e2e-inf-a-d7248b-29725680-bdzhj" Jul 08 20:00:00.280566 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:00.280532 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x45j9\" (UniqueName: \"kubernetes.io/projected/3a70713e-2dc0-45d4-9660-ef52241d3684-kube-api-access-x45j9\") pod \"maas-api-key-cleanup-e2e-inf-a-d7248b-29725680-bdzhj\" (UID: \"3a70713e-2dc0-45d4-9660-ef52241d3684\") " pod="opendatahub/maas-api-key-cleanup-e2e-inf-a-d7248b-29725680-bdzhj" Jul 08 20:00:00.450399 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:00.450367 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-inf-a-d7248b-29725680-bdzhj" Jul 08 20:00:00.582760 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:00.582735 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-inf-a-d7248b-29725680-bdzhj"] Jul 08 20:00:00.585299 ip-10-0-142-195 kubenswrapper[2566]: W0708 20:00:00.585274 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a70713e_2dc0_45d4_9660_ef52241d3684.slice/crio-f3e463a95476df9ee084ba1babd0ba55d1e65121668c291cdaf1e62193139fbd WatchSource:0}: Error finding container f3e463a95476df9ee084ba1babd0ba55d1e65121668c291cdaf1e62193139fbd: Status 404 returned error can't find the container with id f3e463a95476df9ee084ba1babd0ba55d1e65121668c291cdaf1e62193139fbd Jul 08 20:00:00.707494 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:00.707415 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-inf-a-d7248b-29725680-bdzhj" event={"ID":"3a70713e-2dc0-45d4-9660-ef52241d3684","Type":"ContainerStarted","Data":"f3e463a95476df9ee084ba1babd0ba55d1e65121668c291cdaf1e62193139fbd"} Jul 08 20:00:04.308607 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:04.308573 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-deba89/test-model-deba89-kserve-9d886787f-kwptp" Jul 08 20:00:04.309209 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:04.308631 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-deba89/test-model-deba89-kserve-9d886787f-kwptp" Jul 08 20:00:04.322845 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:04.322821 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-deba89/test-model-deba89-kserve-9d886787f-kwptp" Jul 08 20:00:04.738711 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:04.738683 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-deba89/test-model-deba89-kserve-9d886787f-kwptp" Jul 08 20:00:05.732879 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:05.732833 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-inf-a-d7248b-29725680-bdzhj" event={"ID":"3a70713e-2dc0-45d4-9660-ef52241d3684","Type":"ContainerStarted","Data":"b036c48d0f65af812dd710329be586dd0e413292f9621a85d429a042270074b5"} Jul 08 20:00:05.753483 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:05.753433 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-key-cleanup-e2e-inf-a-d7248b-29725680-bdzhj" podStartSLOduration=1.150601338 podStartE2EDuration="5.753418499s" totalImagesPullingTime="4.602817161s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 20:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:00:05.752347715 +0000 UTC m=+2171.638537764" watchObservedRunningTime="2026-07-08 20:00:05.753418499 +0000 UTC m=+2171.639608544" Jul 08 20:00:06.742766 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:06.742728 2566 generic.go:356] "Generic (PLEG): container finished" podID="3a70713e-2dc0-45d4-9660-ef52241d3684" containerID="b036c48d0f65af812dd710329be586dd0e413292f9621a85d429a042270074b5" exitCode=0 Jul 08 20:00:06.743147 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:06.742804 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-inf-a-d7248b-29725680-bdzhj" event={"ID":"3a70713e-2dc0-45d4-9660-ef52241d3684","Type":"ContainerDied","Data":"b036c48d0f65af812dd710329be586dd0e413292f9621a85d429a042270074b5"} Jul 08 20:00:07.874706 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:07.874680 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-inf-a-d7248b-29725680-bdzhj" Jul 08 20:00:07.942724 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:07.942690 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x45j9\" (UniqueName: \"kubernetes.io/projected/3a70713e-2dc0-45d4-9660-ef52241d3684-kube-api-access-x45j9\") pod \"3a70713e-2dc0-45d4-9660-ef52241d3684\" (UID: \"3a70713e-2dc0-45d4-9660-ef52241d3684\") " Jul 08 20:00:07.944650 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:07.944621 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a70713e-2dc0-45d4-9660-ef52241d3684-kube-api-access-x45j9" (OuterVolumeSpecName: "kube-api-access-x45j9") pod "3a70713e-2dc0-45d4-9660-ef52241d3684" (UID: "3a70713e-2dc0-45d4-9660-ef52241d3684"). InnerVolumeSpecName "kube-api-access-x45j9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 20:00:08.043705 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:08.043635 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x45j9\" (UniqueName: \"kubernetes.io/projected/3a70713e-2dc0-45d4-9660-ef52241d3684-kube-api-access-x45j9\") on node \"ip-10-0-142-195.ec2.internal\" DevicePath \"\"" Jul 08 20:00:08.753246 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:08.753219 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-inf-a-d7248b-29725680-bdzhj" Jul 08 20:00:08.753400 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:08.753215 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-inf-a-d7248b-29725680-bdzhj" event={"ID":"3a70713e-2dc0-45d4-9660-ef52241d3684","Type":"ContainerDied","Data":"f3e463a95476df9ee084ba1babd0ba55d1e65121668c291cdaf1e62193139fbd"} Jul 08 20:00:08.753400 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:08.753340 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3e463a95476df9ee084ba1babd0ba55d1e65121668c291cdaf1e62193139fbd" Jul 08 20:00:12.954031 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:12.954000 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:00:13.887645 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:13.887610 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-deba89/test-model-deba89-kserve-9d886787f-kwptp"] Jul 08 20:00:13.912343 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:13.912300 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:00:14.123790 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:14.123755 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:00:14.499812 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:14.499781 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:00:14.605364 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:14.605327 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:00:14.705678 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:14.705631 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:00:14.781004 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:14.780928 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-d7248b-558577c4fd-fdd78"] Jul 08 20:00:14.781442 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:14.781408 2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-a-d7248b-558577c4fd-fdd78" podUID="7c7041e6-8d1a-42c3-8a7a-ca027d924a4b" containerName="maas-api" containerID="cri-o://4ad36e6c45c8291e3533d0aa52915d5866cf69c32db2119852ac971985dc9fce" gracePeriod=30 Jul 08 20:00:14.809117 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:14.809091 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:00:14.810460 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:14.810437 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-inf-a-d7248b-29725680-bdzhj"] Jul 08 20:00:14.812051 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:14.812030 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-inf-a-d7248b-29725680-bdzhj"] Jul 08 20:00:15.024042 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:15.024012 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-d7248b-558577c4fd-fdd78" Jul 08 20:00:15.108018 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:15.107939 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c7041e6-8d1a-42c3-8a7a-ca027d924a4b-openshift-service-ca\") pod \"7c7041e6-8d1a-42c3-8a7a-ca027d924a4b\" (UID: \"7c7041e6-8d1a-42c3-8a7a-ca027d924a4b\") " Jul 08 20:00:15.108150 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:15.108057 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c7041e6-8d1a-42c3-8a7a-ca027d924a4b-maas-api-tls\") pod \"7c7041e6-8d1a-42c3-8a7a-ca027d924a4b\" (UID: \"7c7041e6-8d1a-42c3-8a7a-ca027d924a4b\") " Jul 08 20:00:15.108150 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:15.108086 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8wd8p\" (UniqueName: \"kubernetes.io/projected/7c7041e6-8d1a-42c3-8a7a-ca027d924a4b-kube-api-access-8wd8p\") pod \"7c7041e6-8d1a-42c3-8a7a-ca027d924a4b\" (UID: \"7c7041e6-8d1a-42c3-8a7a-ca027d924a4b\") " Jul 08 20:00:15.108355 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:15.108334 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c7041e6-8d1a-42c3-8a7a-ca027d924a4b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7c7041e6-8d1a-42c3-8a7a-ca027d924a4b" (UID: "7c7041e6-8d1a-42c3-8a7a-ca027d924a4b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 20:00:15.110079 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:15.110057 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c7041e6-8d1a-42c3-8a7a-ca027d924a4b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7c7041e6-8d1a-42c3-8a7a-ca027d924a4b" (UID: "7c7041e6-8d1a-42c3-8a7a-ca027d924a4b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 20:00:15.110249 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:15.110230 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c7041e6-8d1a-42c3-8a7a-ca027d924a4b-kube-api-access-8wd8p" (OuterVolumeSpecName: "kube-api-access-8wd8p") pod "7c7041e6-8d1a-42c3-8a7a-ca027d924a4b" (UID: "7c7041e6-8d1a-42c3-8a7a-ca027d924a4b"). InnerVolumeSpecName "kube-api-access-8wd8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 20:00:15.209372 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:15.209329 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c7041e6-8d1a-42c3-8a7a-ca027d924a4b-maas-api-tls\") on node \"ip-10-0-142-195.ec2.internal\" DevicePath \"\"" Jul 08 20:00:15.209372 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:15.209359 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wd8p\" (UniqueName: \"kubernetes.io/projected/7c7041e6-8d1a-42c3-8a7a-ca027d924a4b-kube-api-access-8wd8p\") on node \"ip-10-0-142-195.ec2.internal\" DevicePath \"\"" Jul 08 20:00:15.209372 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:15.209374 2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c7041e6-8d1a-42c3-8a7a-ca027d924a4b-openshift-service-ca\") on node \"ip-10-0-142-195.ec2.internal\" DevicePath \"\"" Jul 08 20:00:15.597452 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:15.597409 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 20:00:15.787728 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:15.787696 2566 generic.go:356] "Generic (PLEG): container finished" podID="7c7041e6-8d1a-42c3-8a7a-ca027d924a4b" containerID="4ad36e6c45c8291e3533d0aa52915d5866cf69c32db2119852ac971985dc9fce" exitCode=0 Jul 08 20:00:15.787926 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:15.787739 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-d7248b-558577c4fd-fdd78" Jul 08 20:00:15.787926 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:15.787772 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-d7248b-558577c4fd-fdd78" event={"ID":"7c7041e6-8d1a-42c3-8a7a-ca027d924a4b","Type":"ContainerDied","Data":"4ad36e6c45c8291e3533d0aa52915d5866cf69c32db2119852ac971985dc9fce"} Jul 08 20:00:15.787926 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:15.787810 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-d7248b-558577c4fd-fdd78" event={"ID":"7c7041e6-8d1a-42c3-8a7a-ca027d924a4b","Type":"ContainerDied","Data":"60cab00057e6163dfe6ea2b187e45520061bffd8bffef21cce39113d3ef317b9"} Jul 08 20:00:15.787926 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:15.787832 2566 scope.go:117] "RemoveContainer" containerID="4ad36e6c45c8291e3533d0aa52915d5866cf69c32db2119852ac971985dc9fce" Jul 08 20:00:15.797142 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:15.797125 2566 scope.go:117] "RemoveContainer" containerID="4ad36e6c45c8291e3533d0aa52915d5866cf69c32db2119852ac971985dc9fce" Jul 08 20:00:15.797400 ip-10-0-142-195 kubenswrapper[2566]: E0708 20:00:15.797382 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ad36e6c45c8291e3533d0aa52915d5866cf69c32db2119852ac971985dc9fce\": container with ID starting with 4ad36e6c45c8291e3533d0aa52915d5866cf69c32db2119852ac971985dc9fce not found: ID does not exist" containerID="4ad36e6c45c8291e3533d0aa52915d5866cf69c32db2119852ac971985dc9fce" Jul 08 20:00:15.797453 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:15.797409 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ad36e6c45c8291e3533d0aa52915d5866cf69c32db2119852ac971985dc9fce"} err="failed to get container status \"4ad36e6c45c8291e3533d0aa52915d5866cf69c32db2119852ac971985dc9fce\": rpc error: code = NotFound desc = could not find container \"4ad36e6c45c8291e3533d0aa52915d5866cf69c32db2119852ac971985dc9fce\": container with ID starting with 4ad36e6c45c8291e3533d0aa52915d5866cf69c32db2119852ac971985dc9fce not found: ID does not exist" Jul 08 20:00:15.810647 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:15.810626 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-d7248b-558577c4fd-fdd78"] Jul 08 20:00:15.815079 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:15.815061 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-d7248b-558577c4fd-fdd78"] Jul 08 20:00:16.623590 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:16.623554 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a70713e-2dc0-45d4-9660-ef52241d3684" path="/var/lib/kubelet/pods/3a70713e-2dc0-45d4-9660-ef52241d3684/volumes" Jul 08 20:00:16.623975 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:16.623957 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c7041e6-8d1a-42c3-8a7a-ca027d924a4b" path="/var/lib/kubelet/pods/7c7041e6-8d1a-42c3-8a7a-ca027d924a4b/volumes" Jul 08 20:00:19.881613 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:19.881579 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:00:28.907823 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:28.907780 2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-deba89/test-model-deba89-kserve-9d886787f-kwptp" podUID="e0ed9d11-b667-44ab-a844-58b9e3105a9f" containerName="main" containerID="cri-o://10a74edcb3a795d74c0f8c9a4bcfebc77d0f74ae7a17d0ba66c5a729e09b2a14" gracePeriod=45 Jul 08 20:00:29.160854 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:29.160792 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-deba89/test-model-deba89-kserve-9d886787f-kwptp" Jul 08 20:00:29.236656 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:29.236628 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f4kc2\" (UniqueName: \"kubernetes.io/projected/e0ed9d11-b667-44ab-a844-58b9e3105a9f-kube-api-access-f4kc2\") pod \"e0ed9d11-b667-44ab-a844-58b9e3105a9f\" (UID: \"e0ed9d11-b667-44ab-a844-58b9e3105a9f\") " Jul 08 20:00:29.236656 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:29.236661 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0ed9d11-b667-44ab-a844-58b9e3105a9f-tmp-dir\") pod \"e0ed9d11-b667-44ab-a844-58b9e3105a9f\" (UID: \"e0ed9d11-b667-44ab-a844-58b9e3105a9f\") " Jul 08 20:00:29.236879 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:29.236687 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0ed9d11-b667-44ab-a844-58b9e3105a9f-model-cache\") pod \"e0ed9d11-b667-44ab-a844-58b9e3105a9f\" (UID: \"e0ed9d11-b667-44ab-a844-58b9e3105a9f\") " Jul 08 20:00:29.236879 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:29.236718 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0ed9d11-b667-44ab-a844-58b9e3105a9f-home\") pod \"e0ed9d11-b667-44ab-a844-58b9e3105a9f\" (UID: \"e0ed9d11-b667-44ab-a844-58b9e3105a9f\") " Jul 08 20:00:29.236879 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:29.236767 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0ed9d11-b667-44ab-a844-58b9e3105a9f-dshm\") pod \"e0ed9d11-b667-44ab-a844-58b9e3105a9f\" (UID: \"e0ed9d11-b667-44ab-a844-58b9e3105a9f\") " Jul 08 20:00:29.236879 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:29.236795 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0ed9d11-b667-44ab-a844-58b9e3105a9f-tls-certs\") pod \"e0ed9d11-b667-44ab-a844-58b9e3105a9f\" (UID: \"e0ed9d11-b667-44ab-a844-58b9e3105a9f\") " Jul 08 20:00:29.237094 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:29.236944 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0ed9d11-b667-44ab-a844-58b9e3105a9f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e0ed9d11-b667-44ab-a844-58b9e3105a9f" (UID: "e0ed9d11-b667-44ab-a844-58b9e3105a9f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 20:00:29.237094 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:29.237021 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0ed9d11-b667-44ab-a844-58b9e3105a9f-model-cache" (OuterVolumeSpecName: "model-cache") pod "e0ed9d11-b667-44ab-a844-58b9e3105a9f" (UID: "e0ed9d11-b667-44ab-a844-58b9e3105a9f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 20:00:29.237094 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:29.237034 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0ed9d11-b667-44ab-a844-58b9e3105a9f-home" (OuterVolumeSpecName: "home") pod "e0ed9d11-b667-44ab-a844-58b9e3105a9f" (UID: "e0ed9d11-b667-44ab-a844-58b9e3105a9f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 20:00:29.237237 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:29.237180 2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0ed9d11-b667-44ab-a844-58b9e3105a9f-tmp-dir\") on node \"ip-10-0-142-195.ec2.internal\" DevicePath \"\"" Jul 08 20:00:29.237237 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:29.237202 2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0ed9d11-b667-44ab-a844-58b9e3105a9f-model-cache\") on node \"ip-10-0-142-195.ec2.internal\" DevicePath \"\"" Jul 08 20:00:29.237237 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:29.237213 2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0ed9d11-b667-44ab-a844-58b9e3105a9f-home\") on node \"ip-10-0-142-195.ec2.internal\" DevicePath \"\"" Jul 08 20:00:29.238926 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:29.238895 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0ed9d11-b667-44ab-a844-58b9e3105a9f-dshm" (OuterVolumeSpecName: "dshm") pod "e0ed9d11-b667-44ab-a844-58b9e3105a9f" (UID: "e0ed9d11-b667-44ab-a844-58b9e3105a9f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 20:00:29.239226 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:29.239202 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0ed9d11-b667-44ab-a844-58b9e3105a9f-kube-api-access-f4kc2" (OuterVolumeSpecName: "kube-api-access-f4kc2") pod "e0ed9d11-b667-44ab-a844-58b9e3105a9f" (UID: "e0ed9d11-b667-44ab-a844-58b9e3105a9f"). InnerVolumeSpecName "kube-api-access-f4kc2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 20:00:29.239226 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:29.239199 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0ed9d11-b667-44ab-a844-58b9e3105a9f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e0ed9d11-b667-44ab-a844-58b9e3105a9f" (UID: "e0ed9d11-b667-44ab-a844-58b9e3105a9f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 20:00:29.338017 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:29.337980 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4kc2\" (UniqueName: \"kubernetes.io/projected/e0ed9d11-b667-44ab-a844-58b9e3105a9f-kube-api-access-f4kc2\") on node \"ip-10-0-142-195.ec2.internal\" DevicePath \"\"" Jul 08 20:00:29.338017 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:29.338013 2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0ed9d11-b667-44ab-a844-58b9e3105a9f-dshm\") on node \"ip-10-0-142-195.ec2.internal\" DevicePath \"\"" Jul 08 20:00:29.338209 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:29.338027 2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0ed9d11-b667-44ab-a844-58b9e3105a9f-tls-certs\") on node \"ip-10-0-142-195.ec2.internal\" DevicePath \"\"" Jul 08 20:00:29.859521 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:29.859486 2566 generic.go:356] "Generic (PLEG): container finished" podID="e0ed9d11-b667-44ab-a844-58b9e3105a9f" containerID="10a74edcb3a795d74c0f8c9a4bcfebc77d0f74ae7a17d0ba66c5a729e09b2a14" exitCode=0 Jul 08 20:00:29.859733 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:29.859529 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-deba89/test-model-deba89-kserve-9d886787f-kwptp" Jul 08 20:00:29.859733 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:29.859545 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-deba89/test-model-deba89-kserve-9d886787f-kwptp" event={"ID":"e0ed9d11-b667-44ab-a844-58b9e3105a9f","Type":"ContainerDied","Data":"10a74edcb3a795d74c0f8c9a4bcfebc77d0f74ae7a17d0ba66c5a729e09b2a14"} Jul 08 20:00:29.859733 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:29.859588 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-deba89/test-model-deba89-kserve-9d886787f-kwptp" event={"ID":"e0ed9d11-b667-44ab-a844-58b9e3105a9f","Type":"ContainerDied","Data":"3496858adcff1d70a636be59c4ecd2ffcaccd881dfa4448874fc1649dc0012fa"} Jul 08 20:00:29.859733 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:29.859612 2566 scope.go:117] "RemoveContainer" containerID="10a74edcb3a795d74c0f8c9a4bcfebc77d0f74ae7a17d0ba66c5a729e09b2a14" Jul 08 20:00:29.868621 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:29.868606 2566 scope.go:117] "RemoveContainer" containerID="10a74edcb3a795d74c0f8c9a4bcfebc77d0f74ae7a17d0ba66c5a729e09b2a14" Jul 08 20:00:29.868908 ip-10-0-142-195 kubenswrapper[2566]: E0708 20:00:29.868889 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10a74edcb3a795d74c0f8c9a4bcfebc77d0f74ae7a17d0ba66c5a729e09b2a14\": container with ID starting with 10a74edcb3a795d74c0f8c9a4bcfebc77d0f74ae7a17d0ba66c5a729e09b2a14 not found: ID does not exist" containerID="10a74edcb3a795d74c0f8c9a4bcfebc77d0f74ae7a17d0ba66c5a729e09b2a14" Jul 08 20:00:29.868969 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:29.868914 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10a74edcb3a795d74c0f8c9a4bcfebc77d0f74ae7a17d0ba66c5a729e09b2a14"} err="failed to get container status \"10a74edcb3a795d74c0f8c9a4bcfebc77d0f74ae7a17d0ba66c5a729e09b2a14\": rpc error: code = NotFound desc = could not find container \"10a74edcb3a795d74c0f8c9a4bcfebc77d0f74ae7a17d0ba66c5a729e09b2a14\": container with ID starting with 10a74edcb3a795d74c0f8c9a4bcfebc77d0f74ae7a17d0ba66c5a729e09b2a14 not found: ID does not exist" Jul 08 20:00:29.882653 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:29.882628 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-deba89/test-model-deba89-kserve-9d886787f-kwptp"] Jul 08 20:00:29.886356 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:29.886337 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-deba89/test-model-deba89-kserve-9d886787f-kwptp"] Jul 08 20:00:29.988072 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:29.988041 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 20:00:30.623350 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:30.623319 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0ed9d11-b667-44ab-a844-58b9e3105a9f" path="/var/lib/kubelet/pods/e0ed9d11-b667-44ab-a844-58b9e3105a9f/volumes" Jul 08 20:00:33.947339 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:33.947305 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:00:34.063920 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:34.063885 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:00:56.727677 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:56.727636 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:00:56.994167 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:56.994055 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:00:57.244093 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:57.244056 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:00:57.431611 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:57.431582 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:00:57.819294 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:57.819203 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:00:57.920529 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:57.920503 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:00:58.020387 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:58.020354 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:00:58.082177 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:58.082078 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-deba89-9bfb4c795-2jv94"] Jul 08 20:00:58.082359 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:58.082332 2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-b-deba89-9bfb4c795-2jv94" podUID="13afbd86-76cb-4d29-915b-f18bcd3a58df" containerName="maas-api" containerID="cri-o://2004a6e58d3bd318c188d8a4bacbb6ab94f832da6cf66413b52e4aae8e54f08c" gracePeriod=30 Jul 08 20:00:58.111603 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:58.111576 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:00:58.326635 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:58.326610 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-deba89-9bfb4c795-2jv94" Jul 08 20:00:58.395267 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:58.395172 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13afbd86-76cb-4d29-915b-f18bcd3a58df-openshift-service-ca\") pod \"13afbd86-76cb-4d29-915b-f18bcd3a58df\" (UID: \"13afbd86-76cb-4d29-915b-f18bcd3a58df\") " Jul 08 20:00:58.395267 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:58.395249 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wg9fz\" (UniqueName: \"kubernetes.io/projected/13afbd86-76cb-4d29-915b-f18bcd3a58df-kube-api-access-wg9fz\") pod \"13afbd86-76cb-4d29-915b-f18bcd3a58df\" (UID: \"13afbd86-76cb-4d29-915b-f18bcd3a58df\") " Jul 08 20:00:58.395473 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:58.395310 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13afbd86-76cb-4d29-915b-f18bcd3a58df-maas-api-tls\") pod \"13afbd86-76cb-4d29-915b-f18bcd3a58df\" (UID: \"13afbd86-76cb-4d29-915b-f18bcd3a58df\") " Jul 08 20:00:58.395608 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:58.395582 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13afbd86-76cb-4d29-915b-f18bcd3a58df-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "13afbd86-76cb-4d29-915b-f18bcd3a58df" (UID: "13afbd86-76cb-4d29-915b-f18bcd3a58df"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 20:00:58.397429 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:58.397403 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13afbd86-76cb-4d29-915b-f18bcd3a58df-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "13afbd86-76cb-4d29-915b-f18bcd3a58df" (UID: "13afbd86-76cb-4d29-915b-f18bcd3a58df"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 20:00:58.397556 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:58.397513 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13afbd86-76cb-4d29-915b-f18bcd3a58df-kube-api-access-wg9fz" (OuterVolumeSpecName: "kube-api-access-wg9fz") pod "13afbd86-76cb-4d29-915b-f18bcd3a58df" (UID: "13afbd86-76cb-4d29-915b-f18bcd3a58df"). InnerVolumeSpecName "kube-api-access-wg9fz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 20:00:58.496757 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:58.496722 2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13afbd86-76cb-4d29-915b-f18bcd3a58df-openshift-service-ca\") on node \"ip-10-0-142-195.ec2.internal\" DevicePath \"\"" Jul 08 20:00:58.496757 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:58.496750 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wg9fz\" (UniqueName: \"kubernetes.io/projected/13afbd86-76cb-4d29-915b-f18bcd3a58df-kube-api-access-wg9fz\") on node \"ip-10-0-142-195.ec2.internal\" DevicePath \"\"" Jul 08 20:00:58.496757 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:58.496760 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13afbd86-76cb-4d29-915b-f18bcd3a58df-maas-api-tls\") on node \"ip-10-0-142-195.ec2.internal\" DevicePath \"\"" Jul 08 20:00:58.995252 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:58.995217 2566 generic.go:356] "Generic (PLEG): container finished" podID="13afbd86-76cb-4d29-915b-f18bcd3a58df" containerID="2004a6e58d3bd318c188d8a4bacbb6ab94f832da6cf66413b52e4aae8e54f08c" exitCode=0 Jul 08 20:00:58.995779 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:58.995265 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-deba89-9bfb4c795-2jv94" Jul 08 20:00:58.995779 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:58.995302 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-deba89-9bfb4c795-2jv94" event={"ID":"13afbd86-76cb-4d29-915b-f18bcd3a58df","Type":"ContainerDied","Data":"2004a6e58d3bd318c188d8a4bacbb6ab94f832da6cf66413b52e4aae8e54f08c"} Jul 08 20:00:58.995779 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:58.995358 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-deba89-9bfb4c795-2jv94" event={"ID":"13afbd86-76cb-4d29-915b-f18bcd3a58df","Type":"ContainerDied","Data":"51ff97876a45a19d021d3a5ffff16dfc533fa76e62f3711b248c928c7f4d0490"} Jul 08 20:00:58.995779 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:58.995386 2566 scope.go:117] "RemoveContainer" containerID="2004a6e58d3bd318c188d8a4bacbb6ab94f832da6cf66413b52e4aae8e54f08c" Jul 08 20:00:59.004009 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:59.003991 2566 scope.go:117] "RemoveContainer" containerID="2004a6e58d3bd318c188d8a4bacbb6ab94f832da6cf66413b52e4aae8e54f08c" Jul 08 20:00:59.004278 ip-10-0-142-195 kubenswrapper[2566]: E0708 20:00:59.004255 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2004a6e58d3bd318c188d8a4bacbb6ab94f832da6cf66413b52e4aae8e54f08c\": container with ID starting with 2004a6e58d3bd318c188d8a4bacbb6ab94f832da6cf66413b52e4aae8e54f08c not found: ID does not exist" containerID="2004a6e58d3bd318c188d8a4bacbb6ab94f832da6cf66413b52e4aae8e54f08c" Jul 08 20:00:59.004347 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:59.004285 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2004a6e58d3bd318c188d8a4bacbb6ab94f832da6cf66413b52e4aae8e54f08c"} err="failed to get container status \"2004a6e58d3bd318c188d8a4bacbb6ab94f832da6cf66413b52e4aae8e54f08c\": rpc error: code = NotFound desc = could not find container \"2004a6e58d3bd318c188d8a4bacbb6ab94f832da6cf66413b52e4aae8e54f08c\": container with ID starting with 2004a6e58d3bd318c188d8a4bacbb6ab94f832da6cf66413b52e4aae8e54f08c not found: ID does not exist" Jul 08 20:00:59.012618 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:59.012594 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-deba89-9bfb4c795-2jv94"] Jul 08 20:00:59.018878 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:00:59.018842 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-deba89-9bfb4c795-2jv94"] Jul 08 20:01:00.085132 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:00.085095 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 20:01:00.622688 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:00.622653 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13afbd86-76cb-4d29-915b-f18bcd3a58df" path="/var/lib/kubelet/pods/13afbd86-76cb-4d29-915b-f18bcd3a58df/volumes" Jul 08 20:01:04.039251 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:04.039209 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:04.101250 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:04.101198 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:04.348213 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:04.348141 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:04.574684 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:04.574648 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:04.940090 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:04.940055 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:05.270726 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:05.270639 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:05.873434 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:05.873394 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:07.908332 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:07.908289 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:07.988178 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:07.988140 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-a-6abdf4-569b887fd5-tdf29"] Jul 08 20:01:07.988695 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:07.988678 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a70713e-2dc0-45d4-9660-ef52241d3684" containerName="cleanup" Jul 08 20:01:07.988754 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:07.988697 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a70713e-2dc0-45d4-9660-ef52241d3684" containerName="cleanup" Jul 08 20:01:07.988754 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:07.988706 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13afbd86-76cb-4d29-915b-f18bcd3a58df" containerName="maas-api" Jul 08 20:01:07.988754 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:07.988714 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="13afbd86-76cb-4d29-915b-f18bcd3a58df" containerName="maas-api" Jul 08 20:01:07.988754 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:07.988730 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0ed9d11-b667-44ab-a844-58b9e3105a9f" containerName="main" Jul 08 20:01:07.988754 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:07.988736 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0ed9d11-b667-44ab-a844-58b9e3105a9f" containerName="main" Jul 08 20:01:07.988754 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:07.988749 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c7041e6-8d1a-42c3-8a7a-ca027d924a4b" containerName="maas-api" Jul 08 20:01:07.988754 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:07.988755 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c7041e6-8d1a-42c3-8a7a-ca027d924a4b" containerName="maas-api" Jul 08 20:01:07.988992 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:07.988813 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a70713e-2dc0-45d4-9660-ef52241d3684" containerName="cleanup" Jul 08 20:01:07.988992 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:07.988822 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0ed9d11-b667-44ab-a844-58b9e3105a9f" containerName="main" Jul 08 20:01:07.988992 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:07.988828 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="13afbd86-76cb-4d29-915b-f18bcd3a58df" containerName="maas-api" Jul 08 20:01:07.988992 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:07.988834 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c7041e6-8d1a-42c3-8a7a-ca027d924a4b" containerName="maas-api" Jul 08 20:01:07.991160 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:07.991138 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-6abdf4-569b887fd5-tdf29" Jul 08 20:01:07.991929 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:07.991898 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:07.993925 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:07.993902 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-a-6abdf4-serving-cert\"" Jul 08 20:01:07.994144 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:07.993904 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-7kmg8\"" Jul 08 20:01:08.001548 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:08.001524 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-6abdf4-569b887fd5-tdf29"] Jul 08 20:01:08.080274 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:08.080232 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/465d01fa-55f7-4db6-b997-b22a45429da4-openshift-service-ca\") pod \"maas-api-e2e-api-a-6abdf4-569b887fd5-tdf29\" (UID: \"465d01fa-55f7-4db6-b997-b22a45429da4\") " pod="opendatahub/maas-api-e2e-api-a-6abdf4-569b887fd5-tdf29" Jul 08 20:01:08.080459 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:08.080283 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/465d01fa-55f7-4db6-b997-b22a45429da4-maas-api-tls\") pod \"maas-api-e2e-api-a-6abdf4-569b887fd5-tdf29\" (UID: \"465d01fa-55f7-4db6-b997-b22a45429da4\") " pod="opendatahub/maas-api-e2e-api-a-6abdf4-569b887fd5-tdf29" Jul 08 20:01:08.080459 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:08.080317 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8z48\" (UniqueName: \"kubernetes.io/projected/465d01fa-55f7-4db6-b997-b22a45429da4-kube-api-access-k8z48\") pod \"maas-api-e2e-api-a-6abdf4-569b887fd5-tdf29\" (UID: \"465d01fa-55f7-4db6-b997-b22a45429da4\") " pod="opendatahub/maas-api-e2e-api-a-6abdf4-569b887fd5-tdf29" Jul 08 20:01:08.181337 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:08.181246 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/465d01fa-55f7-4db6-b997-b22a45429da4-openshift-service-ca\") pod \"maas-api-e2e-api-a-6abdf4-569b887fd5-tdf29\" (UID: \"465d01fa-55f7-4db6-b997-b22a45429da4\") " pod="opendatahub/maas-api-e2e-api-a-6abdf4-569b887fd5-tdf29" Jul 08 20:01:08.181337 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:08.181296 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/465d01fa-55f7-4db6-b997-b22a45429da4-maas-api-tls\") pod \"maas-api-e2e-api-a-6abdf4-569b887fd5-tdf29\" (UID: \"465d01fa-55f7-4db6-b997-b22a45429da4\") " pod="opendatahub/maas-api-e2e-api-a-6abdf4-569b887fd5-tdf29" Jul 08 20:01:08.181337 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:08.181333 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8z48\" (UniqueName: \"kubernetes.io/projected/465d01fa-55f7-4db6-b997-b22a45429da4-kube-api-access-k8z48\") pod \"maas-api-e2e-api-a-6abdf4-569b887fd5-tdf29\" (UID: \"465d01fa-55f7-4db6-b997-b22a45429da4\") " pod="opendatahub/maas-api-e2e-api-a-6abdf4-569b887fd5-tdf29" Jul 08 20:01:08.181576 ip-10-0-142-195 kubenswrapper[2566]: E0708 20:01:08.181454 2566 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-api-a-6abdf4-serving-cert: secret "maas-api-e2e-api-a-6abdf4-serving-cert" not found Jul 08 20:01:08.181576 ip-10-0-142-195 kubenswrapper[2566]: E0708 20:01:08.181522 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/465d01fa-55f7-4db6-b997-b22a45429da4-maas-api-tls podName:465d01fa-55f7-4db6-b997-b22a45429da4 nodeName:}" failed. No retries permitted until 2026-07-08 20:01:08.68150496 +0000 UTC m=+2234.567694988 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/465d01fa-55f7-4db6-b997-b22a45429da4-maas-api-tls") pod "maas-api-e2e-api-a-6abdf4-569b887fd5-tdf29" (UID: "465d01fa-55f7-4db6-b997-b22a45429da4") : secret "maas-api-e2e-api-a-6abdf4-serving-cert" not found Jul 08 20:01:08.181907 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:08.181890 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/465d01fa-55f7-4db6-b997-b22a45429da4-openshift-service-ca\") pod \"maas-api-e2e-api-a-6abdf4-569b887fd5-tdf29\" (UID: \"465d01fa-55f7-4db6-b997-b22a45429da4\") " pod="opendatahub/maas-api-e2e-api-a-6abdf4-569b887fd5-tdf29" Jul 08 20:01:08.191354 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:08.191324 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8z48\" (UniqueName: \"kubernetes.io/projected/465d01fa-55f7-4db6-b997-b22a45429da4-kube-api-access-k8z48\") pod \"maas-api-e2e-api-a-6abdf4-569b887fd5-tdf29\" (UID: \"465d01fa-55f7-4db6-b997-b22a45429da4\") " pod="opendatahub/maas-api-e2e-api-a-6abdf4-569b887fd5-tdf29" Jul 08 20:01:08.680568 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:08.680537 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:08.686717 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:08.686690 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/465d01fa-55f7-4db6-b997-b22a45429da4-maas-api-tls\") pod \"maas-api-e2e-api-a-6abdf4-569b887fd5-tdf29\" (UID: \"465d01fa-55f7-4db6-b997-b22a45429da4\") " pod="opendatahub/maas-api-e2e-api-a-6abdf4-569b887fd5-tdf29" Jul 08 20:01:08.689086 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:08.689065 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/465d01fa-55f7-4db6-b997-b22a45429da4-maas-api-tls\") pod \"maas-api-e2e-api-a-6abdf4-569b887fd5-tdf29\" (UID: \"465d01fa-55f7-4db6-b997-b22a45429da4\") " pod="opendatahub/maas-api-e2e-api-a-6abdf4-569b887fd5-tdf29" Jul 08 20:01:08.778844 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:08.778807 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:08.904570 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:08.904533 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-6abdf4-569b887fd5-tdf29" Jul 08 20:01:09.035541 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:09.035509 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-6abdf4-569b887fd5-tdf29"] Jul 08 20:01:09.040079 ip-10-0-142-195 kubenswrapper[2566]: W0708 20:01:09.040055 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod465d01fa_55f7_4db6_b997_b22a45429da4.slice/crio-d0600b3a48372ee4ac2915f67c7282ea8d6aceb0dcd876842ea0cc92055d9c85 WatchSource:0}: Error finding container d0600b3a48372ee4ac2915f67c7282ea8d6aceb0dcd876842ea0cc92055d9c85: Status 404 returned error can't find the container with id d0600b3a48372ee4ac2915f67c7282ea8d6aceb0dcd876842ea0cc92055d9c85 Jul 08 20:01:10.045751 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:10.045718 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-6abdf4-569b887fd5-tdf29" event={"ID":"465d01fa-55f7-4db6-b997-b22a45429da4","Type":"ContainerStarted","Data":"d0600b3a48372ee4ac2915f67c7282ea8d6aceb0dcd876842ea0cc92055d9c85"} Jul 08 20:01:11.052121 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:11.052085 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-6abdf4-569b887fd5-tdf29" event={"ID":"465d01fa-55f7-4db6-b997-b22a45429da4","Type":"ContainerStarted","Data":"f02b2917f2bf6488beea20fa912130179ffa57b5908b35aef9da86d6a7bf11ab"} Jul 08 20:01:11.052612 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:11.052192 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-a-6abdf4-569b887fd5-tdf29" Jul 08 20:01:11.071786 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:11.071739 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-a-6abdf4-569b887fd5-tdf29" podStartSLOduration=2.9131995870000003 podStartE2EDuration="4.071722309s" totalImagesPullingTime="1.158522722s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 20:01:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:01:11.070003066 +0000 UTC m=+2236.956193138" watchObservedRunningTime="2026-07-08 20:01:11.071722309 +0000 UTC m=+2236.957912356" Jul 08 20:01:12.407322 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:12.407290 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:12.887737 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:12.887645 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:13.213146 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:13.213099 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:13.317997 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:13.317963 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:13.792165 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:13.792137 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 20:01:16.046903 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:16.046844 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:16.112097 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:16.112063 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-b-b7fdb0-5956b5cbfd-jwzlp"] Jul 08 20:01:16.115369 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:16.115346 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-b7fdb0-5956b5cbfd-jwzlp" Jul 08 20:01:16.118114 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:16.118062 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-b-b7fdb0-serving-cert\"" Jul 08 20:01:16.127481 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:16.127459 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-b7fdb0-5956b5cbfd-jwzlp"] Jul 08 20:01:16.139133 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:16.139108 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:16.207187 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:16.207147 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:16.258157 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:16.258105 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46hvm\" (UniqueName: \"kubernetes.io/projected/f72c84fa-8449-4204-b91b-d7a19e9fadce-kube-api-access-46hvm\") pod \"maas-api-e2e-api-b-b7fdb0-5956b5cbfd-jwzlp\" (UID: \"f72c84fa-8449-4204-b91b-d7a19e9fadce\") " pod="opendatahub/maas-api-e2e-api-b-b7fdb0-5956b5cbfd-jwzlp" Jul 08 20:01:16.258157 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:16.258162 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f72c84fa-8449-4204-b91b-d7a19e9fadce-openshift-service-ca\") pod \"maas-api-e2e-api-b-b7fdb0-5956b5cbfd-jwzlp\" (UID: \"f72c84fa-8449-4204-b91b-d7a19e9fadce\") " pod="opendatahub/maas-api-e2e-api-b-b7fdb0-5956b5cbfd-jwzlp" Jul 08 20:01:16.258506 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:16.258228 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f72c84fa-8449-4204-b91b-d7a19e9fadce-maas-api-tls\") pod \"maas-api-e2e-api-b-b7fdb0-5956b5cbfd-jwzlp\" (UID: \"f72c84fa-8449-4204-b91b-d7a19e9fadce\") " pod="opendatahub/maas-api-e2e-api-b-b7fdb0-5956b5cbfd-jwzlp" Jul 08 20:01:16.359614 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:16.359531 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f72c84fa-8449-4204-b91b-d7a19e9fadce-maas-api-tls\") pod \"maas-api-e2e-api-b-b7fdb0-5956b5cbfd-jwzlp\" (UID: \"f72c84fa-8449-4204-b91b-d7a19e9fadce\") " pod="opendatahub/maas-api-e2e-api-b-b7fdb0-5956b5cbfd-jwzlp" Jul 08 20:01:16.359783 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:16.359650 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46hvm\" (UniqueName: \"kubernetes.io/projected/f72c84fa-8449-4204-b91b-d7a19e9fadce-kube-api-access-46hvm\") pod \"maas-api-e2e-api-b-b7fdb0-5956b5cbfd-jwzlp\" (UID: \"f72c84fa-8449-4204-b91b-d7a19e9fadce\") " pod="opendatahub/maas-api-e2e-api-b-b7fdb0-5956b5cbfd-jwzlp" Jul 08 20:01:16.359783 ip-10-0-142-195 kubenswrapper[2566]: E0708 20:01:16.359678 2566 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-api-b-b7fdb0-serving-cert: secret "maas-api-e2e-api-b-b7fdb0-serving-cert" not found Jul 08 20:01:16.359783 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:16.359705 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f72c84fa-8449-4204-b91b-d7a19e9fadce-openshift-service-ca\") pod \"maas-api-e2e-api-b-b7fdb0-5956b5cbfd-jwzlp\" (UID: \"f72c84fa-8449-4204-b91b-d7a19e9fadce\") " pod="opendatahub/maas-api-e2e-api-b-b7fdb0-5956b5cbfd-jwzlp" Jul 08 20:01:16.359783 ip-10-0-142-195 kubenswrapper[2566]: E0708 20:01:16.359745 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f72c84fa-8449-4204-b91b-d7a19e9fadce-maas-api-tls podName:f72c84fa-8449-4204-b91b-d7a19e9fadce nodeName:}" failed. No retries permitted until 2026-07-08 20:01:16.85972924 +0000 UTC m=+2242.745919268 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f72c84fa-8449-4204-b91b-d7a19e9fadce-maas-api-tls") pod "maas-api-e2e-api-b-b7fdb0-5956b5cbfd-jwzlp" (UID: "f72c84fa-8449-4204-b91b-d7a19e9fadce") : secret "maas-api-e2e-api-b-b7fdb0-serving-cert" not found Jul 08 20:01:16.360353 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:16.360330 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f72c84fa-8449-4204-b91b-d7a19e9fadce-openshift-service-ca\") pod \"maas-api-e2e-api-b-b7fdb0-5956b5cbfd-jwzlp\" (UID: \"f72c84fa-8449-4204-b91b-d7a19e9fadce\") " pod="opendatahub/maas-api-e2e-api-b-b7fdb0-5956b5cbfd-jwzlp" Jul 08 20:01:16.370875 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:16.370839 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46hvm\" (UniqueName: \"kubernetes.io/projected/f72c84fa-8449-4204-b91b-d7a19e9fadce-kube-api-access-46hvm\") pod \"maas-api-e2e-api-b-b7fdb0-5956b5cbfd-jwzlp\" (UID: \"f72c84fa-8449-4204-b91b-d7a19e9fadce\") " pod="opendatahub/maas-api-e2e-api-b-b7fdb0-5956b5cbfd-jwzlp" Jul 08 20:01:16.533804 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:16.533772 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:16.642921 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:16.642799 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:16.742083 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:16.742051 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:16.864449 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:16.864413 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f72c84fa-8449-4204-b91b-d7a19e9fadce-maas-api-tls\") pod \"maas-api-e2e-api-b-b7fdb0-5956b5cbfd-jwzlp\" (UID: \"f72c84fa-8449-4204-b91b-d7a19e9fadce\") " pod="opendatahub/maas-api-e2e-api-b-b7fdb0-5956b5cbfd-jwzlp" Jul 08 20:01:16.866750 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:16.866726 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f72c84fa-8449-4204-b91b-d7a19e9fadce-maas-api-tls\") pod \"maas-api-e2e-api-b-b7fdb0-5956b5cbfd-jwzlp\" (UID: \"f72c84fa-8449-4204-b91b-d7a19e9fadce\") " pod="opendatahub/maas-api-e2e-api-b-b7fdb0-5956b5cbfd-jwzlp" Jul 08 20:01:17.026388 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:17.026356 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-b7fdb0-5956b5cbfd-jwzlp" Jul 08 20:01:17.063577 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:17.063553 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-a-6abdf4-569b887fd5-tdf29" Jul 08 20:01:17.164495 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:17.164454 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-b7fdb0-5956b5cbfd-jwzlp"] Jul 08 20:01:17.168540 ip-10-0-142-195 kubenswrapper[2566]: W0708 20:01:17.168517 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf72c84fa_8449_4204_b91b_d7a19e9fadce.slice/crio-a210f24d41ceb82086984540943f56d5149ea1f62b3a9e86deb14a9ecd6b336d WatchSource:0}: Error finding container a210f24d41ceb82086984540943f56d5149ea1f62b3a9e86deb14a9ecd6b336d: Status 404 returned error can't find the container with id a210f24d41ceb82086984540943f56d5149ea1f62b3a9e86deb14a9ecd6b336d Jul 08 20:01:18.091323 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:18.091287 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-b7fdb0-5956b5cbfd-jwzlp" event={"ID":"f72c84fa-8449-4204-b91b-d7a19e9fadce","Type":"ContainerStarted","Data":"4aa9148fa8119ede1f37933672ba2bd1956a84eccd302c0854a452a2b10369ab"} Jul 08 20:01:18.091752 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:18.091337 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-b7fdb0-5956b5cbfd-jwzlp" event={"ID":"f72c84fa-8449-4204-b91b-d7a19e9fadce","Type":"ContainerStarted","Data":"a210f24d41ceb82086984540943f56d5149ea1f62b3a9e86deb14a9ecd6b336d"} Jul 08 20:01:18.091752 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:18.091387 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-b-b7fdb0-5956b5cbfd-jwzlp" Jul 08 20:01:18.110739 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:18.110649 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-b-b7fdb0-5956b5cbfd-jwzlp" podStartSLOduration=1.7533153000000001 podStartE2EDuration="2.110627564s" totalImagesPullingTime="357.312264ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 20:01:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:01:18.110033827 +0000 UTC m=+2243.996223872" watchObservedRunningTime="2026-07-08 20:01:18.110627564 +0000 UTC m=+2243.996817611" Jul 08 20:01:20.484576 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:20.484535 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:24.102079 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:24.102049 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-b-b7fdb0-5956b5cbfd-jwzlp" Jul 08 20:01:28.393805 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:28.393758 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 20:01:28.628754 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:28.628723 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:29.019841 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:29.019805 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:29.120714 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:29.120683 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:29.218028 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:29.218004 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:29.316824 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:29.316741 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-6abdf4-569b887fd5-tdf29"] Jul 08 20:01:29.317116 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:29.317087 2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-a-6abdf4-569b887fd5-tdf29" podUID="465d01fa-55f7-4db6-b997-b22a45429da4" containerName="maas-api" containerID="cri-o://f02b2917f2bf6488beea20fa912130179ffa57b5908b35aef9da86d6a7bf11ab" gracePeriod=30 Jul 08 20:01:29.341324 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:29.341292 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:29.582898 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:29.582876 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-6abdf4-569b887fd5-tdf29" Jul 08 20:01:29.672409 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:29.672380 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/465d01fa-55f7-4db6-b997-b22a45429da4-openshift-service-ca\") pod \"465d01fa-55f7-4db6-b997-b22a45429da4\" (UID: \"465d01fa-55f7-4db6-b997-b22a45429da4\") " Jul 08 20:01:29.672569 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:29.672466 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/465d01fa-55f7-4db6-b997-b22a45429da4-maas-api-tls\") pod \"465d01fa-55f7-4db6-b997-b22a45429da4\" (UID: \"465d01fa-55f7-4db6-b997-b22a45429da4\") " Jul 08 20:01:29.672569 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:29.672492 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k8z48\" (UniqueName: \"kubernetes.io/projected/465d01fa-55f7-4db6-b997-b22a45429da4-kube-api-access-k8z48\") pod \"465d01fa-55f7-4db6-b997-b22a45429da4\" (UID: \"465d01fa-55f7-4db6-b997-b22a45429da4\") " Jul 08 20:01:29.672728 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:29.672705 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/465d01fa-55f7-4db6-b997-b22a45429da4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "465d01fa-55f7-4db6-b997-b22a45429da4" (UID: "465d01fa-55f7-4db6-b997-b22a45429da4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 20:01:29.674423 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:29.674391 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/465d01fa-55f7-4db6-b997-b22a45429da4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "465d01fa-55f7-4db6-b997-b22a45429da4" (UID: "465d01fa-55f7-4db6-b997-b22a45429da4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 20:01:29.674567 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:29.674548 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/465d01fa-55f7-4db6-b997-b22a45429da4-kube-api-access-k8z48" (OuterVolumeSpecName: "kube-api-access-k8z48") pod "465d01fa-55f7-4db6-b997-b22a45429da4" (UID: "465d01fa-55f7-4db6-b997-b22a45429da4"). InnerVolumeSpecName "kube-api-access-k8z48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 20:01:29.773213 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:29.773188 2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/465d01fa-55f7-4db6-b997-b22a45429da4-openshift-service-ca\") on node \"ip-10-0-142-195.ec2.internal\" DevicePath \"\"" Jul 08 20:01:29.773213 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:29.773211 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/465d01fa-55f7-4db6-b997-b22a45429da4-maas-api-tls\") on node \"ip-10-0-142-195.ec2.internal\" DevicePath \"\"" Jul 08 20:01:29.773345 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:29.773220 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8z48\" (UniqueName: \"kubernetes.io/projected/465d01fa-55f7-4db6-b997-b22a45429da4-kube-api-access-k8z48\") on node \"ip-10-0-142-195.ec2.internal\" DevicePath \"\"" Jul 08 20:01:30.155946 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:30.155912 2566 generic.go:356] "Generic (PLEG): container finished" podID="465d01fa-55f7-4db6-b997-b22a45429da4" containerID="f02b2917f2bf6488beea20fa912130179ffa57b5908b35aef9da86d6a7bf11ab" exitCode=0 Jul 08 20:01:30.156142 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:30.155976 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-6abdf4-569b887fd5-tdf29" Jul 08 20:01:30.156142 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:30.155996 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-6abdf4-569b887fd5-tdf29" event={"ID":"465d01fa-55f7-4db6-b997-b22a45429da4","Type":"ContainerDied","Data":"f02b2917f2bf6488beea20fa912130179ffa57b5908b35aef9da86d6a7bf11ab"} Jul 08 20:01:30.156142 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:30.156065 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-6abdf4-569b887fd5-tdf29" event={"ID":"465d01fa-55f7-4db6-b997-b22a45429da4","Type":"ContainerDied","Data":"d0600b3a48372ee4ac2915f67c7282ea8d6aceb0dcd876842ea0cc92055d9c85"} Jul 08 20:01:30.156142 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:30.156094 2566 scope.go:117] "RemoveContainer" containerID="f02b2917f2bf6488beea20fa912130179ffa57b5908b35aef9da86d6a7bf11ab" Jul 08 20:01:30.165479 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:30.165458 2566 scope.go:117] "RemoveContainer" containerID="f02b2917f2bf6488beea20fa912130179ffa57b5908b35aef9da86d6a7bf11ab" Jul 08 20:01:30.165784 ip-10-0-142-195 kubenswrapper[2566]: E0708 20:01:30.165767 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f02b2917f2bf6488beea20fa912130179ffa57b5908b35aef9da86d6a7bf11ab\": container with ID starting with f02b2917f2bf6488beea20fa912130179ffa57b5908b35aef9da86d6a7bf11ab not found: ID does not exist" containerID="f02b2917f2bf6488beea20fa912130179ffa57b5908b35aef9da86d6a7bf11ab" Jul 08 20:01:30.165846 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:30.165796 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f02b2917f2bf6488beea20fa912130179ffa57b5908b35aef9da86d6a7bf11ab"} err="failed to get container status \"f02b2917f2bf6488beea20fa912130179ffa57b5908b35aef9da86d6a7bf11ab\": rpc error: code = NotFound desc = could not find container \"f02b2917f2bf6488beea20fa912130179ffa57b5908b35aef9da86d6a7bf11ab\": container with ID starting with f02b2917f2bf6488beea20fa912130179ffa57b5908b35aef9da86d6a7bf11ab not found: ID does not exist" Jul 08 20:01:30.178298 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:30.178275 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-6abdf4-569b887fd5-tdf29"] Jul 08 20:01:30.182374 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:30.182351 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-a-6abdf4-569b887fd5-tdf29"] Jul 08 20:01:30.623379 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:30.623304 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="465d01fa-55f7-4db6-b997-b22a45429da4" path="/var/lib/kubelet/pods/465d01fa-55f7-4db6-b997-b22a45429da4/volumes" Jul 08 20:01:35.045683 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:35.045650 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:35.145720 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:35.145688 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:35.323962 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:35.323855 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:35.547584 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:35.547553 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:35.714341 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:35.714308 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:36.187152 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:36.187115 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:36.380245 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:36.380213 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:36.480828 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:36.480730 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:36.610102 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:36.610070 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-b7fdb0-5956b5cbfd-jwzlp"] Jul 08 20:01:36.610372 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:36.610346 2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-b-b7fdb0-5956b5cbfd-jwzlp" podUID="f72c84fa-8449-4204-b91b-d7a19e9fadce" containerName="maas-api" containerID="cri-o://4aa9148fa8119ede1f37933672ba2bd1956a84eccd302c0854a452a2b10369ab" gracePeriod=30 Jul 08 20:01:36.631034 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:36.631006 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:36.853320 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:36.853298 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-b7fdb0-5956b5cbfd-jwzlp" Jul 08 20:01:36.938158 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:36.938128 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f72c84fa-8449-4204-b91b-d7a19e9fadce-maas-api-tls\") pod \"f72c84fa-8449-4204-b91b-d7a19e9fadce\" (UID: \"f72c84fa-8449-4204-b91b-d7a19e9fadce\") " Jul 08 20:01:36.938302 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:36.938178 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-46hvm\" (UniqueName: \"kubernetes.io/projected/f72c84fa-8449-4204-b91b-d7a19e9fadce-kube-api-access-46hvm\") pod \"f72c84fa-8449-4204-b91b-d7a19e9fadce\" (UID: \"f72c84fa-8449-4204-b91b-d7a19e9fadce\") " Jul 08 20:01:36.938302 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:36.938230 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f72c84fa-8449-4204-b91b-d7a19e9fadce-openshift-service-ca\") pod \"f72c84fa-8449-4204-b91b-d7a19e9fadce\" (UID: \"f72c84fa-8449-4204-b91b-d7a19e9fadce\") " Jul 08 20:01:36.938642 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:36.938617 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f72c84fa-8449-4204-b91b-d7a19e9fadce-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f72c84fa-8449-4204-b91b-d7a19e9fadce" (UID: "f72c84fa-8449-4204-b91b-d7a19e9fadce"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 20:01:36.940277 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:36.940256 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f72c84fa-8449-4204-b91b-d7a19e9fadce-kube-api-access-46hvm" (OuterVolumeSpecName: "kube-api-access-46hvm") pod "f72c84fa-8449-4204-b91b-d7a19e9fadce" (UID: "f72c84fa-8449-4204-b91b-d7a19e9fadce"). InnerVolumeSpecName "kube-api-access-46hvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 20:01:36.940349 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:36.940277 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f72c84fa-8449-4204-b91b-d7a19e9fadce-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f72c84fa-8449-4204-b91b-d7a19e9fadce" (UID: "f72c84fa-8449-4204-b91b-d7a19e9fadce"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 20:01:37.039465 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:37.039404 2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f72c84fa-8449-4204-b91b-d7a19e9fadce-openshift-service-ca\") on node \"ip-10-0-142-195.ec2.internal\" DevicePath \"\"" Jul 08 20:01:37.039465 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:37.039427 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f72c84fa-8449-4204-b91b-d7a19e9fadce-maas-api-tls\") on node \"ip-10-0-142-195.ec2.internal\" DevicePath \"\"" Jul 08 20:01:37.039465 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:37.039438 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46hvm\" (UniqueName: \"kubernetes.io/projected/f72c84fa-8449-4204-b91b-d7a19e9fadce-kube-api-access-46hvm\") on node \"ip-10-0-142-195.ec2.internal\" DevicePath \"\"" Jul 08 20:01:37.188758 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:37.188722 2566 generic.go:356] "Generic (PLEG): container finished" podID="f72c84fa-8449-4204-b91b-d7a19e9fadce" containerID="4aa9148fa8119ede1f37933672ba2bd1956a84eccd302c0854a452a2b10369ab" exitCode=0 Jul 08 20:01:37.189224 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:37.188768 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-b7fdb0-5956b5cbfd-jwzlp" Jul 08 20:01:37.189224 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:37.188808 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-b7fdb0-5956b5cbfd-jwzlp" event={"ID":"f72c84fa-8449-4204-b91b-d7a19e9fadce","Type":"ContainerDied","Data":"4aa9148fa8119ede1f37933672ba2bd1956a84eccd302c0854a452a2b10369ab"} Jul 08 20:01:37.189224 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:37.188857 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-b7fdb0-5956b5cbfd-jwzlp" event={"ID":"f72c84fa-8449-4204-b91b-d7a19e9fadce","Type":"ContainerDied","Data":"a210f24d41ceb82086984540943f56d5149ea1f62b3a9e86deb14a9ecd6b336d"} Jul 08 20:01:37.189224 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:37.188899 2566 scope.go:117] "RemoveContainer" containerID="4aa9148fa8119ede1f37933672ba2bd1956a84eccd302c0854a452a2b10369ab" Jul 08 20:01:37.197797 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:37.197779 2566 scope.go:117] "RemoveContainer" containerID="4aa9148fa8119ede1f37933672ba2bd1956a84eccd302c0854a452a2b10369ab" Jul 08 20:01:37.198114 ip-10-0-142-195 kubenswrapper[2566]: E0708 20:01:37.198097 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4aa9148fa8119ede1f37933672ba2bd1956a84eccd302c0854a452a2b10369ab\": container with ID starting with 4aa9148fa8119ede1f37933672ba2bd1956a84eccd302c0854a452a2b10369ab not found: ID does not exist" containerID="4aa9148fa8119ede1f37933672ba2bd1956a84eccd302c0854a452a2b10369ab" Jul 08 20:01:37.198159 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:37.198122 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4aa9148fa8119ede1f37933672ba2bd1956a84eccd302c0854a452a2b10369ab"} err="failed to get container status \"4aa9148fa8119ede1f37933672ba2bd1956a84eccd302c0854a452a2b10369ab\": rpc error: code = NotFound desc = could not find container \"4aa9148fa8119ede1f37933672ba2bd1956a84eccd302c0854a452a2b10369ab\": container with ID starting with 4aa9148fa8119ede1f37933672ba2bd1956a84eccd302c0854a452a2b10369ab not found: ID does not exist" Jul 08 20:01:37.209650 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:37.209631 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-b7fdb0-5956b5cbfd-jwzlp"] Jul 08 20:01:37.213561 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:37.213544 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-b-b7fdb0-5956b5cbfd-jwzlp"] Jul 08 20:01:38.622984 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:38.622951 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f72c84fa-8449-4204-b91b-d7a19e9fadce" path="/var/lib/kubelet/pods/f72c84fa-8449-4204-b91b-d7a19e9fadce/volumes" Jul 08 20:01:39.886321 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:39.886285 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 20:01:42.342659 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:42.342628 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:42.402022 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:42.401988 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:42.637615 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:42.637533 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:42.873874 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:42.873826 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:43.214089 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:43.214059 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:43.844523 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:43.844493 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:43.903084 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:43.903051 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:46.170433 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:46.170398 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:46.237310 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:46.237268 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-a-960620-587467c688-qqnm9"] Jul 08 20:01:46.237655 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:46.237641 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f72c84fa-8449-4204-b91b-d7a19e9fadce" containerName="maas-api" Jul 08 20:01:46.237700 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:46.237658 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f72c84fa-8449-4204-b91b-d7a19e9fadce" containerName="maas-api" Jul 08 20:01:46.237700 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:46.237676 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="465d01fa-55f7-4db6-b997-b22a45429da4" containerName="maas-api" Jul 08 20:01:46.237700 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:46.237681 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="465d01fa-55f7-4db6-b997-b22a45429da4" containerName="maas-api" Jul 08 20:01:46.237798 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:46.237749 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="465d01fa-55f7-4db6-b997-b22a45429da4" containerName="maas-api" Jul 08 20:01:46.237798 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:46.237757 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f72c84fa-8449-4204-b91b-d7a19e9fadce" containerName="maas-api" Jul 08 20:01:46.240803 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:46.240783 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-960620-587467c688-qqnm9" Jul 08 20:01:46.243304 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:46.243278 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-a-960620-serving-cert\"" Jul 08 20:01:46.243304 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:46.243296 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-7kmg8\"" Jul 08 20:01:46.258741 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:46.258703 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-960620-587467c688-qqnm9"] Jul 08 20:01:46.267041 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:46.267004 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:46.320020 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:46.319980 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnt6c\" (UniqueName: \"kubernetes.io/projected/c651106f-7cba-4f10-ad27-590996640cdb-kube-api-access-fnt6c\") pod \"maas-api-e2e-shared-a-960620-587467c688-qqnm9\" (UID: \"c651106f-7cba-4f10-ad27-590996640cdb\") " pod="opendatahub/maas-api-e2e-shared-a-960620-587467c688-qqnm9" Jul 08 20:01:46.320208 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:46.320153 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c651106f-7cba-4f10-ad27-590996640cdb-maas-api-tls\") pod \"maas-api-e2e-shared-a-960620-587467c688-qqnm9\" (UID: \"c651106f-7cba-4f10-ad27-590996640cdb\") " pod="opendatahub/maas-api-e2e-shared-a-960620-587467c688-qqnm9" Jul 08 20:01:46.320208 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:46.320200 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c651106f-7cba-4f10-ad27-590996640cdb-openshift-service-ca\") pod \"maas-api-e2e-shared-a-960620-587467c688-qqnm9\" (UID: \"c651106f-7cba-4f10-ad27-590996640cdb\") " pod="opendatahub/maas-api-e2e-shared-a-960620-587467c688-qqnm9" Jul 08 20:01:46.334300 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:46.334264 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:46.420918 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:46.420800 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c651106f-7cba-4f10-ad27-590996640cdb-maas-api-tls\") pod \"maas-api-e2e-shared-a-960620-587467c688-qqnm9\" (UID: \"c651106f-7cba-4f10-ad27-590996640cdb\") " pod="opendatahub/maas-api-e2e-shared-a-960620-587467c688-qqnm9" Jul 08 20:01:46.420918 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:46.420891 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c651106f-7cba-4f10-ad27-590996640cdb-openshift-service-ca\") pod \"maas-api-e2e-shared-a-960620-587467c688-qqnm9\" (UID: \"c651106f-7cba-4f10-ad27-590996640cdb\") " pod="opendatahub/maas-api-e2e-shared-a-960620-587467c688-qqnm9" Jul 08 20:01:46.421114 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:46.420940 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnt6c\" (UniqueName: \"kubernetes.io/projected/c651106f-7cba-4f10-ad27-590996640cdb-kube-api-access-fnt6c\") pod \"maas-api-e2e-shared-a-960620-587467c688-qqnm9\" (UID: \"c651106f-7cba-4f10-ad27-590996640cdb\") " pod="opendatahub/maas-api-e2e-shared-a-960620-587467c688-qqnm9" Jul 08 20:01:46.421611 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:46.421593 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c651106f-7cba-4f10-ad27-590996640cdb-openshift-service-ca\") pod \"maas-api-e2e-shared-a-960620-587467c688-qqnm9\" (UID: \"c651106f-7cba-4f10-ad27-590996640cdb\") " pod="opendatahub/maas-api-e2e-shared-a-960620-587467c688-qqnm9" Jul 08 20:01:46.423413 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:46.423386 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c651106f-7cba-4f10-ad27-590996640cdb-maas-api-tls\") pod \"maas-api-e2e-shared-a-960620-587467c688-qqnm9\" (UID: \"c651106f-7cba-4f10-ad27-590996640cdb\") " pod="opendatahub/maas-api-e2e-shared-a-960620-587467c688-qqnm9" Jul 08 20:01:46.432823 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:46.432796 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnt6c\" (UniqueName: \"kubernetes.io/projected/c651106f-7cba-4f10-ad27-590996640cdb-kube-api-access-fnt6c\") pod \"maas-api-e2e-shared-a-960620-587467c688-qqnm9\" (UID: \"c651106f-7cba-4f10-ad27-590996640cdb\") " pod="opendatahub/maas-api-e2e-shared-a-960620-587467c688-qqnm9" Jul 08 20:01:46.473374 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:46.473338 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:46.551097 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:46.551069 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-960620-587467c688-qqnm9" Jul 08 20:01:46.693594 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:46.693572 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-960620-587467c688-qqnm9"] Jul 08 20:01:46.695853 ip-10-0-142-195 kubenswrapper[2566]: W0708 20:01:46.695822 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc651106f_7cba_4f10_ad27_590996640cdb.slice/crio-67d248280b81cbe575bd47939bbb1f9176419dff567692b36053da863d6204a1 WatchSource:0}: Error finding container 67d248280b81cbe575bd47939bbb1f9176419dff567692b36053da863d6204a1: Status 404 returned error can't find the container with id 67d248280b81cbe575bd47939bbb1f9176419dff567692b36053da863d6204a1 Jul 08 20:01:46.738474 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:46.738447 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:46.830121 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:46.830089 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:47.235651 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:47.235624 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-960620-587467c688-qqnm9" event={"ID":"c651106f-7cba-4f10-ad27-590996640cdb","Type":"ContainerStarted","Data":"67d248280b81cbe575bd47939bbb1f9176419dff567692b36053da863d6204a1"} Jul 08 20:01:48.242086 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:48.242049 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-960620-587467c688-qqnm9" event={"ID":"c651106f-7cba-4f10-ad27-590996640cdb","Type":"ContainerStarted","Data":"e739fde5422a786c5f2aa769469a92d0ae91f00eb256b7c90f77643cabf67104"} Jul 08 20:01:48.242461 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:48.242111 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-a-960620-587467c688-qqnm9" Jul 08 20:01:48.262900 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:48.262816 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-a-960620-587467c688-qqnm9" podStartSLOduration=1.767954983 podStartE2EDuration="2.262796124s" totalImagesPullingTime="494.841141ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 20:01:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:01:48.26213648 +0000 UTC m=+2274.148326529" watchObservedRunningTime="2026-07-08 20:01:48.262796124 +0000 UTC m=+2274.148986171" Jul 08 20:01:50.716987 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:50.716955 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:51.036795 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:51.036691 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:51.298310 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:51.298218 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 20:01:51.352410 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:51.352374 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:51.607904 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:51.607800 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:54.244160 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:54.244125 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:54.251879 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:54.251840 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-a-960620-587467c688-qqnm9" Jul 08 20:01:54.323053 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:54.323021 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:54.412789 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:54.412753 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-b-ceba08-c745d7b95-vlx88"] Jul 08 20:01:54.415948 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:54.415929 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-ceba08-c745d7b95-vlx88" Jul 08 20:01:54.418561 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:54.418535 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-b-ceba08-serving-cert\"" Jul 08 20:01:54.431455 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:54.431434 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-ceba08-c745d7b95-vlx88"] Jul 08 20:01:54.488117 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:54.488088 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a84496f5-4fa6-42b5-ae34-4903466fc45f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-ceba08-c745d7b95-vlx88\" (UID: \"a84496f5-4fa6-42b5-ae34-4903466fc45f\") " pod="opendatahub/maas-api-e2e-shared-b-ceba08-c745d7b95-vlx88" Jul 08 20:01:54.488266 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:54.488239 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a84496f5-4fa6-42b5-ae34-4903466fc45f-maas-api-tls\") pod \"maas-api-e2e-shared-b-ceba08-c745d7b95-vlx88\" (UID: \"a84496f5-4fa6-42b5-ae34-4903466fc45f\") " pod="opendatahub/maas-api-e2e-shared-b-ceba08-c745d7b95-vlx88" Jul 08 20:01:54.488327 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:54.488297 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r94jk\" (UniqueName: \"kubernetes.io/projected/a84496f5-4fa6-42b5-ae34-4903466fc45f-kube-api-access-r94jk\") pod \"maas-api-e2e-shared-b-ceba08-c745d7b95-vlx88\" (UID: \"a84496f5-4fa6-42b5-ae34-4903466fc45f\") " pod="opendatahub/maas-api-e2e-shared-b-ceba08-c745d7b95-vlx88" Jul 08 20:01:54.507978 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:54.507878 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:54.589390 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:54.589354 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a84496f5-4fa6-42b5-ae34-4903466fc45f-maas-api-tls\") pod \"maas-api-e2e-shared-b-ceba08-c745d7b95-vlx88\" (UID: \"a84496f5-4fa6-42b5-ae34-4903466fc45f\") " pod="opendatahub/maas-api-e2e-shared-b-ceba08-c745d7b95-vlx88" Jul 08 20:01:54.589523 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:54.589401 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r94jk\" (UniqueName: \"kubernetes.io/projected/a84496f5-4fa6-42b5-ae34-4903466fc45f-kube-api-access-r94jk\") pod \"maas-api-e2e-shared-b-ceba08-c745d7b95-vlx88\" (UID: \"a84496f5-4fa6-42b5-ae34-4903466fc45f\") " pod="opendatahub/maas-api-e2e-shared-b-ceba08-c745d7b95-vlx88" Jul 08 20:01:54.589523 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:54.589435 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a84496f5-4fa6-42b5-ae34-4903466fc45f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-ceba08-c745d7b95-vlx88\" (UID: \"a84496f5-4fa6-42b5-ae34-4903466fc45f\") " pod="opendatahub/maas-api-e2e-shared-b-ceba08-c745d7b95-vlx88" Jul 08 20:01:54.590091 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:54.590073 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a84496f5-4fa6-42b5-ae34-4903466fc45f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-ceba08-c745d7b95-vlx88\" (UID: \"a84496f5-4fa6-42b5-ae34-4903466fc45f\") " pod="opendatahub/maas-api-e2e-shared-b-ceba08-c745d7b95-vlx88" Jul 08 20:01:54.592389 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:54.592373 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-b-ceba08-serving-cert\"" Jul 08 20:01:54.598381 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:54.598362 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r94jk\" (UniqueName: \"kubernetes.io/projected/a84496f5-4fa6-42b5-ae34-4903466fc45f-kube-api-access-r94jk\") pod \"maas-api-e2e-shared-b-ceba08-c745d7b95-vlx88\" (UID: \"a84496f5-4fa6-42b5-ae34-4903466fc45f\") " pod="opendatahub/maas-api-e2e-shared-b-ceba08-c745d7b95-vlx88" Jul 08 20:01:54.600159 ip-10-0-142-195 kubenswrapper[2566]: E0708 20:01:54.600142 2566 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-shared-b-ceba08-serving-cert: secret "maas-api-e2e-shared-b-ceba08-serving-cert" not found Jul 08 20:01:54.600230 ip-10-0-142-195 kubenswrapper[2566]: E0708 20:01:54.600213 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a84496f5-4fa6-42b5-ae34-4903466fc45f-maas-api-tls podName:a84496f5-4fa6-42b5-ae34-4903466fc45f nodeName:}" failed. No retries permitted until 2026-07-08 20:01:55.100191949 +0000 UTC m=+2280.986382001 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a84496f5-4fa6-42b5-ae34-4903466fc45f-maas-api-tls") pod "maas-api-e2e-shared-b-ceba08-c745d7b95-vlx88" (UID: "a84496f5-4fa6-42b5-ae34-4903466fc45f") : secret "maas-api-e2e-shared-b-ceba08-serving-cert" not found Jul 08 20:01:54.739349 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:54.739315 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:55.041833 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:55.041754 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:55.147475 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:55.147440 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:55.194190 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:55.194166 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a84496f5-4fa6-42b5-ae34-4903466fc45f-maas-api-tls\") pod \"maas-api-e2e-shared-b-ceba08-c745d7b95-vlx88\" (UID: \"a84496f5-4fa6-42b5-ae34-4903466fc45f\") " pod="opendatahub/maas-api-e2e-shared-b-ceba08-c745d7b95-vlx88" Jul 08 20:01:55.196584 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:55.196561 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a84496f5-4fa6-42b5-ae34-4903466fc45f-maas-api-tls\") pod \"maas-api-e2e-shared-b-ceba08-c745d7b95-vlx88\" (UID: \"a84496f5-4fa6-42b5-ae34-4903466fc45f\") " pod="opendatahub/maas-api-e2e-shared-b-ceba08-c745d7b95-vlx88" Jul 08 20:01:55.326163 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:55.326094 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-ceba08-c745d7b95-vlx88" Jul 08 20:01:55.472408 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:55.472387 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-ceba08-c745d7b95-vlx88"] Jul 08 20:01:55.475548 ip-10-0-142-195 kubenswrapper[2566]: W0708 20:01:55.475522 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda84496f5_4fa6_42b5_ae34_4903466fc45f.slice/crio-fd6f739a7fedc22ddb7fa198e9624933353121cc2e00d6717bfdd7fb3e7b8c75 WatchSource:0}: Error finding container fd6f739a7fedc22ddb7fa198e9624933353121cc2e00d6717bfdd7fb3e7b8c75: Status 404 returned error can't find the container with id fd6f739a7fedc22ddb7fa198e9624933353121cc2e00d6717bfdd7fb3e7b8c75 Jul 08 20:01:56.286146 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:56.286068 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-ceba08-c745d7b95-vlx88" event={"ID":"a84496f5-4fa6-42b5-ae34-4903466fc45f","Type":"ContainerStarted","Data":"39bb55d5692f1f7035b8270081051f8274efdfcef137ba47906a2bb777fef627"} Jul 08 20:01:56.286146 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:56.286120 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-ceba08-c745d7b95-vlx88" event={"ID":"a84496f5-4fa6-42b5-ae34-4903466fc45f","Type":"ContainerStarted","Data":"fd6f739a7fedc22ddb7fa198e9624933353121cc2e00d6717bfdd7fb3e7b8c75"} Jul 08 20:01:56.286146 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:56.286146 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-b-ceba08-c745d7b95-vlx88" Jul 08 20:01:56.304635 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:56.304582 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-b-ceba08-c745d7b95-vlx88" podStartSLOduration=1.8451613199999999 podStartE2EDuration="2.304565039s" totalImagesPullingTime="459.403719ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 20:01:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:01:56.303614237 +0000 UTC m=+2282.189804278" watchObservedRunningTime="2026-07-08 20:01:56.304565039 +0000 UTC m=+2282.190755086" Jul 08 20:01:58.717603 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:01:58.717560 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:02:02.295980 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:02:02.295954 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-b-ceba08-c745d7b95-vlx88" Jul 08 20:02:06.693650 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:02:06.693620 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 20:02:07.476201 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:02:07.476173 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:02:07.576806 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:02:07.576778 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:02:10.625742 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:02:10.625713 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-960620/auth-test-model-960620-kserve-7c598cc679-75shn"] Jul 08 20:02:10.628392 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:02:10.628373 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-960620/auth-test-model-960620-kserve-7c598cc679-75shn" Jul 08 20:02:10.631164 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:02:10.631142 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-960620\"/\"openshift-service-ca.crt\"" Jul 08 20:02:10.631407 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:02:10.631394 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-960620\"/\"auth-test-model-960620-kserve-self-signed-certs\"" Jul 08 20:02:10.632305 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:02:10.632283 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-960620\"/\"kube-root-ca.crt\"" Jul 08 20:02:10.632425 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:02:10.632386 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-960620\"/\"default-dockercfg-v27lv\"" Jul 08 20:02:10.643403 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:02:10.643379 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-960620/auth-test-model-960620-kserve-7c598cc679-75shn"] Jul 08 20:02:10.652079 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:02:10.652055 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:02:10.729733 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:02:10.729703 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3234b8da-8044-4b73-af0d-898cbc971373-model-cache\") pod \"auth-test-model-960620-kserve-7c598cc679-75shn\" (UID: \"3234b8da-8044-4b73-af0d-898cbc971373\") " pod="ai-tenant-e2e-shared-a-960620/auth-test-model-960620-kserve-7c598cc679-75shn" Jul 08 20:02:10.729896 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:02:10.729777 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rjjg\" (UniqueName: \"kubernetes.io/projected/3234b8da-8044-4b73-af0d-898cbc971373-kube-api-access-7rjjg\") pod \"auth-test-model-960620-kserve-7c598cc679-75shn\" (UID: \"3234b8da-8044-4b73-af0d-898cbc971373\") " pod="ai-tenant-e2e-shared-a-960620/auth-test-model-960620-kserve-7c598cc679-75shn" Jul 08 20:02:10.729896 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:02:10.729838 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3234b8da-8044-4b73-af0d-898cbc971373-home\") pod \"auth-test-model-960620-kserve-7c598cc679-75shn\" (UID: \"3234b8da-8044-4b73-af0d-898cbc971373\") " pod="ai-tenant-e2e-shared-a-960620/auth-test-model-960620-kserve-7c598cc679-75shn" Jul 08 20:02:10.730056 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:02:10.730038 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3234b8da-8044-4b73-af0d-898cbc971373-tls-certs\") pod \"auth-test-model-960620-kserve-7c598cc679-75shn\" (UID: \"3234b8da-8044-4b73-af0d-898cbc971373\") " pod="ai-tenant-e2e-shared-a-960620/auth-test-model-960620-kserve-7c598cc679-75shn" Jul 08 20:02:10.730126 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:02:10.730094 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3234b8da-8044-4b73-af0d-898cbc971373-tmp-dir\") pod \"auth-test-model-960620-kserve-7c598cc679-75shn\" (UID: \"3234b8da-8044-4b73-af0d-898cbc971373\") " pod="ai-tenant-e2e-shared-a-960620/auth-test-model-960620-kserve-7c598cc679-75shn" Jul 08 20:02:10.730223 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:02:10.730204 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3234b8da-8044-4b73-af0d-898cbc971373-dshm\") pod \"auth-test-model-960620-kserve-7c598cc679-75shn\" (UID: \"3234b8da-8044-4b73-af0d-898cbc971373\") " pod="ai-tenant-e2e-shared-a-960620/auth-test-model-960620-kserve-7c598cc679-75shn" Jul 08 20:02:10.830804 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:02:10.830777 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3234b8da-8044-4b73-af0d-898cbc971373-home\") pod \"auth-test-model-960620-kserve-7c598cc679-75shn\" (UID: \"3234b8da-8044-4b73-af0d-898cbc971373\") " pod="ai-tenant-e2e-shared-a-960620/auth-test-model-960620-kserve-7c598cc679-75shn" Jul 08 20:02:10.831053 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:02:10.830819 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3234b8da-8044-4b73-af0d-898cbc971373-tls-certs\") pod \"auth-test-model-960620-kserve-7c598cc679-75shn\" (UID: \"3234b8da-8044-4b73-af0d-898cbc971373\") " pod="ai-tenant-e2e-shared-a-960620/auth-test-model-960620-kserve-7c598cc679-75shn" Jul 08 20:02:10.831053 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:02:10.830843 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3234b8da-8044-4b73-af0d-898cbc971373-tmp-dir\") pod \"auth-test-model-960620-kserve-7c598cc679-75shn\" (UID: \"3234b8da-8044-4b73-af0d-898cbc971373\") " pod="ai-tenant-e2e-shared-a-960620/auth-test-model-960620-kserve-7c598cc679-75shn" Jul 08 20:02:10.831053 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:02:10.830896 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3234b8da-8044-4b73-af0d-898cbc971373-dshm\") pod \"auth-test-model-960620-kserve-7c598cc679-75shn\" (UID: \"3234b8da-8044-4b73-af0d-898cbc971373\") " pod="ai-tenant-e2e-shared-a-960620/auth-test-model-960620-kserve-7c598cc679-75shn" Jul 08 20:02:10.831053 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:02:10.830925 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3234b8da-8044-4b73-af0d-898cbc971373-model-cache\") pod \"auth-test-model-960620-kserve-7c598cc679-75shn\" (UID: \"3234b8da-8044-4b73-af0d-898cbc971373\") " pod="ai-tenant-e2e-shared-a-960620/auth-test-model-960620-kserve-7c598cc679-75shn" Jul 08 20:02:10.831053 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:02:10.830962 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rjjg\" (UniqueName: \"kubernetes.io/projected/3234b8da-8044-4b73-af0d-898cbc971373-kube-api-access-7rjjg\") pod \"auth-test-model-960620-kserve-7c598cc679-75shn\" (UID: \"3234b8da-8044-4b73-af0d-898cbc971373\") " pod="ai-tenant-e2e-shared-a-960620/auth-test-model-960620-kserve-7c598cc679-75shn" Jul 08 20:02:10.831391 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:02:10.831217 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3234b8da-8044-4b73-af0d-898cbc971373-home\") pod \"auth-test-model-960620-kserve-7c598cc679-75shn\" (UID: \"3234b8da-8044-4b73-af0d-898cbc971373\") " pod="ai-tenant-e2e-shared-a-960620/auth-test-model-960620-kserve-7c598cc679-75shn" Jul 08 20:02:10.831391 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:02:10.831252 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3234b8da-8044-4b73-af0d-898cbc971373-tmp-dir\") pod \"auth-test-model-960620-kserve-7c598cc679-75shn\" (UID: \"3234b8da-8044-4b73-af0d-898cbc971373\") " pod="ai-tenant-e2e-shared-a-960620/auth-test-model-960620-kserve-7c598cc679-75shn" Jul 08 20:02:10.831391 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:02:10.831305 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3234b8da-8044-4b73-af0d-898cbc971373-model-cache\") pod \"auth-test-model-960620-kserve-7c598cc679-75shn\" (UID: \"3234b8da-8044-4b73-af0d-898cbc971373\") " pod="ai-tenant-e2e-shared-a-960620/auth-test-model-960620-kserve-7c598cc679-75shn" Jul 08 20:02:10.833058 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:02:10.833035 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3234b8da-8044-4b73-af0d-898cbc971373-dshm\") pod \"auth-test-model-960620-kserve-7c598cc679-75shn\" (UID: \"3234b8da-8044-4b73-af0d-898cbc971373\") " pod="ai-tenant-e2e-shared-a-960620/auth-test-model-960620-kserve-7c598cc679-75shn" Jul 08 20:02:10.833214 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:02:10.833198 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3234b8da-8044-4b73-af0d-898cbc971373-tls-certs\") pod \"auth-test-model-960620-kserve-7c598cc679-75shn\" (UID: \"3234b8da-8044-4b73-af0d-898cbc971373\") " pod="ai-tenant-e2e-shared-a-960620/auth-test-model-960620-kserve-7c598cc679-75shn" Jul 08 20:02:10.839789 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:02:10.839771 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rjjg\" (UniqueName: \"kubernetes.io/projected/3234b8da-8044-4b73-af0d-898cbc971373-kube-api-access-7rjjg\") pod \"auth-test-model-960620-kserve-7c598cc679-75shn\" (UID: \"3234b8da-8044-4b73-af0d-898cbc971373\") " pod="ai-tenant-e2e-shared-a-960620/auth-test-model-960620-kserve-7c598cc679-75shn" Jul 08 20:02:10.938230 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:02:10.938199 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-960620/auth-test-model-960620-kserve-7c598cc679-75shn" Jul 08 20:02:11.067505 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:02:11.067479 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-960620/auth-test-model-960620-kserve-7c598cc679-75shn"] Jul 08 20:02:11.069228 ip-10-0-142-195 kubenswrapper[2566]: W0708 20:02:11.069191 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3234b8da_8044_4b73_af0d_898cbc971373.slice/crio-bcd33eaf4f3dc70cf2a1a589f2ab3d4b18377f820e72e96b040eb6ec0ba84715 WatchSource:0}: Error finding container bcd33eaf4f3dc70cf2a1a589f2ab3d4b18377f820e72e96b040eb6ec0ba84715: Status 404 returned error can't find the container with id bcd33eaf4f3dc70cf2a1a589f2ab3d4b18377f820e72e96b040eb6ec0ba84715 Jul 08 20:02:11.353962 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:02:11.353849 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-960620/auth-test-model-960620-kserve-7c598cc679-75shn" event={"ID":"3234b8da-8044-4b73-af0d-898cbc971373","Type":"ContainerStarted","Data":"99a7f240df3a69808d9f6f27e1f935403cbea4e30e640d7c07ec3ed8f83d0708"} Jul 08 20:02:11.353962 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:02:11.353909 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-960620/auth-test-model-960620-kserve-7c598cc679-75shn" event={"ID":"3234b8da-8044-4b73-af0d-898cbc971373","Type":"ContainerStarted","Data":"bcd33eaf4f3dc70cf2a1a589f2ab3d4b18377f820e72e96b040eb6ec0ba84715"} Jul 08 20:02:11.374093 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:02:11.374043 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-960620/auth-test-model-960620-kserve-7c598cc679-75shn" podStartSLOduration=1.374026716 podStartE2EDuration="1.374026716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 20:02:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:02:11.370620933 +0000 UTC m=+2297.256810992" watchObservedRunningTime="2026-07-08 20:02:11.374026716 +0000 UTC m=+2297.260216797" Jul 08 20:02:15.912745 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:02:15.912714 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:02:16.000314 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:02:16.000282 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 20:02:16.003993 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:02:16.003963 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:02:19.409462 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:02:19.409416 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:02:20.938332 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:02:20.938295 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-960620/auth-test-model-960620-kserve-7c598cc679-75shn" Jul 08 20:02:20.938753 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:02:20.938348 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-960620/auth-test-model-960620-kserve-7c598cc679-75shn" Jul 08 20:02:20.950856 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:02:20.950830 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-960620/auth-test-model-960620-kserve-7c598cc679-75shn" Jul 08 20:02:21.411568 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:02:21.411538 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-960620/auth-test-model-960620-kserve-7c598cc679-75shn" Jul 08 20:02:50.018816 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:02:50.018728 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:02:53.787708 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:02:53.787670 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 20:03:03.680630 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:03:03.680597 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:03:06.487609 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:03:06.487572 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 20:03:23.107151 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:03:23.107107 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 20:03:36.294997 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:03:36.294963 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 20:03:53.587574 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:03:53.587542 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 20:03:57.117492 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:03:57.117452 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:04:11.404496 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:04:11.404462 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 20:04:26.879581 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:04:26.879490 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 20:04:43.785383 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:04:43.785352 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 20:04:45.626276 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:04:45.626238 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:04:49.451141 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:04:49.451106 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-960620/auth-test-model-960620-kserve-7c598cc679-75shn"] Jul 08 20:04:49.471084 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:04:49.471061 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:04:50.512229 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:04:50.512195 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:04:52.179828 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:04:52.179791 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:04:52.280814 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:04:52.280785 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:05:01.290184 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:05:01.290153 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 20:05:04.470357 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:05:04.470300 2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-960620/auth-test-model-960620-kserve-7c598cc679-75shn" podUID="3234b8da-8044-4b73-af0d-898cbc971373" containerName="main" containerID="cri-o://99a7f240df3a69808d9f6f27e1f935403cbea4e30e640d7c07ec3ed8f83d0708" gracePeriod=45 Jul 08 20:05:04.708633 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:05:04.708608 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-960620/auth-test-model-960620-kserve-7c598cc679-75shn" Jul 08 20:05:04.781749 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:05:04.781671 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7rjjg\" (UniqueName: \"kubernetes.io/projected/3234b8da-8044-4b73-af0d-898cbc971373-kube-api-access-7rjjg\") pod \"3234b8da-8044-4b73-af0d-898cbc971373\" (UID: \"3234b8da-8044-4b73-af0d-898cbc971373\") " Jul 08 20:05:04.781749 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:05:04.781712 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3234b8da-8044-4b73-af0d-898cbc971373-tls-certs\") pod \"3234b8da-8044-4b73-af0d-898cbc971373\" (UID: \"3234b8da-8044-4b73-af0d-898cbc971373\") " Jul 08 20:05:04.781980 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:05:04.781759 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3234b8da-8044-4b73-af0d-898cbc971373-model-cache\") pod \"3234b8da-8044-4b73-af0d-898cbc971373\" (UID: \"3234b8da-8044-4b73-af0d-898cbc971373\") " Jul 08 20:05:04.781980 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:05:04.781776 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3234b8da-8044-4b73-af0d-898cbc971373-home\") pod \"3234b8da-8044-4b73-af0d-898cbc971373\" (UID: \"3234b8da-8044-4b73-af0d-898cbc971373\") " Jul 08 20:05:04.781980 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:05:04.781814 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3234b8da-8044-4b73-af0d-898cbc971373-tmp-dir\") pod \"3234b8da-8044-4b73-af0d-898cbc971373\" (UID: \"3234b8da-8044-4b73-af0d-898cbc971373\") " Jul 08 20:05:04.781980 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:05:04.781843 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3234b8da-8044-4b73-af0d-898cbc971373-dshm\") pod \"3234b8da-8044-4b73-af0d-898cbc971373\" (UID: \"3234b8da-8044-4b73-af0d-898cbc971373\") " Jul 08 20:05:04.782186 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:05:04.782063 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3234b8da-8044-4b73-af0d-898cbc971373-model-cache" (OuterVolumeSpecName: "model-cache") pod "3234b8da-8044-4b73-af0d-898cbc971373" (UID: "3234b8da-8044-4b73-af0d-898cbc971373"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 20:05:04.782186 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:05:04.782082 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3234b8da-8044-4b73-af0d-898cbc971373-home" (OuterVolumeSpecName: "home") pod "3234b8da-8044-4b73-af0d-898cbc971373" (UID: "3234b8da-8044-4b73-af0d-898cbc971373"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 20:05:04.782186 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:05:04.782115 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3234b8da-8044-4b73-af0d-898cbc971373-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3234b8da-8044-4b73-af0d-898cbc971373" (UID: "3234b8da-8044-4b73-af0d-898cbc971373"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 20:05:04.782186 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:05:04.782171 2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3234b8da-8044-4b73-af0d-898cbc971373-model-cache\") on node \"ip-10-0-142-195.ec2.internal\" DevicePath \"\"" Jul 08 20:05:04.782186 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:05:04.782189 2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3234b8da-8044-4b73-af0d-898cbc971373-home\") on node \"ip-10-0-142-195.ec2.internal\" DevicePath \"\"" Jul 08 20:05:04.784007 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:05:04.783983 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3234b8da-8044-4b73-af0d-898cbc971373-dshm" (OuterVolumeSpecName: "dshm") pod "3234b8da-8044-4b73-af0d-898cbc971373" (UID: "3234b8da-8044-4b73-af0d-898cbc971373"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 20:05:04.784121 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:05:04.784006 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3234b8da-8044-4b73-af0d-898cbc971373-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3234b8da-8044-4b73-af0d-898cbc971373" (UID: "3234b8da-8044-4b73-af0d-898cbc971373"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 20:05:04.784121 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:05:04.784008 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3234b8da-8044-4b73-af0d-898cbc971373-kube-api-access-7rjjg" (OuterVolumeSpecName: "kube-api-access-7rjjg") pod "3234b8da-8044-4b73-af0d-898cbc971373" (UID: "3234b8da-8044-4b73-af0d-898cbc971373"). InnerVolumeSpecName "kube-api-access-7rjjg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 20:05:04.883622 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:05:04.883588 2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3234b8da-8044-4b73-af0d-898cbc971373-tmp-dir\") on node \"ip-10-0-142-195.ec2.internal\" DevicePath \"\"" Jul 08 20:05:04.883622 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:05:04.883622 2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3234b8da-8044-4b73-af0d-898cbc971373-dshm\") on node \"ip-10-0-142-195.ec2.internal\" DevicePath \"\"" Jul 08 20:05:04.883622 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:05:04.883632 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rjjg\" (UniqueName: \"kubernetes.io/projected/3234b8da-8044-4b73-af0d-898cbc971373-kube-api-access-7rjjg\") on node \"ip-10-0-142-195.ec2.internal\" DevicePath \"\"" Jul 08 20:05:04.883838 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:05:04.883642 2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3234b8da-8044-4b73-af0d-898cbc971373-tls-certs\") on node \"ip-10-0-142-195.ec2.internal\" DevicePath \"\"" Jul 08 20:05:05.155321 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:05:05.155289 2566 generic.go:356] "Generic (PLEG): container finished" podID="3234b8da-8044-4b73-af0d-898cbc971373" containerID="99a7f240df3a69808d9f6f27e1f935403cbea4e30e640d7c07ec3ed8f83d0708" exitCode=0 Jul 08 20:05:05.155517 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:05:05.155369 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-960620/auth-test-model-960620-kserve-7c598cc679-75shn" event={"ID":"3234b8da-8044-4b73-af0d-898cbc971373","Type":"ContainerDied","Data":"99a7f240df3a69808d9f6f27e1f935403cbea4e30e640d7c07ec3ed8f83d0708"} Jul 08 20:05:05.155517 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:05:05.155454 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-960620/auth-test-model-960620-kserve-7c598cc679-75shn" event={"ID":"3234b8da-8044-4b73-af0d-898cbc971373","Type":"ContainerDied","Data":"bcd33eaf4f3dc70cf2a1a589f2ab3d4b18377f820e72e96b040eb6ec0ba84715"} Jul 08 20:05:05.155517 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:05:05.155476 2566 scope.go:117] "RemoveContainer" containerID="99a7f240df3a69808d9f6f27e1f935403cbea4e30e640d7c07ec3ed8f83d0708" Jul 08 20:05:05.155700 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:05:05.155393 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-960620/auth-test-model-960620-kserve-7c598cc679-75shn" Jul 08 20:05:05.173442 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:05:05.173422 2566 scope.go:117] "RemoveContainer" containerID="99a7f240df3a69808d9f6f27e1f935403cbea4e30e640d7c07ec3ed8f83d0708" Jul 08 20:05:05.173704 ip-10-0-142-195 kubenswrapper[2566]: E0708 20:05:05.173685 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99a7f240df3a69808d9f6f27e1f935403cbea4e30e640d7c07ec3ed8f83d0708\": container with ID starting with 99a7f240df3a69808d9f6f27e1f935403cbea4e30e640d7c07ec3ed8f83d0708 not found: ID does not exist" containerID="99a7f240df3a69808d9f6f27e1f935403cbea4e30e640d7c07ec3ed8f83d0708" Jul 08 20:05:05.173768 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:05:05.173716 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99a7f240df3a69808d9f6f27e1f935403cbea4e30e640d7c07ec3ed8f83d0708"} err="failed to get container status \"99a7f240df3a69808d9f6f27e1f935403cbea4e30e640d7c07ec3ed8f83d0708\": rpc error: code = NotFound desc = could not find container \"99a7f240df3a69808d9f6f27e1f935403cbea4e30e640d7c07ec3ed8f83d0708\": container with ID starting with 99a7f240df3a69808d9f6f27e1f935403cbea4e30e640d7c07ec3ed8f83d0708 not found: ID does not exist" Jul 08 20:05:05.182133 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:05:05.182110 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-960620/auth-test-model-960620-kserve-7c598cc679-75shn"] Jul 08 20:05:05.187722 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:05:05.187703 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-960620/auth-test-model-960620-kserve-7c598cc679-75shn"] Jul 08 20:05:06.623712 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:05:06.623680 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3234b8da-8044-4b73-af0d-898cbc971373" path="/var/lib/kubelet/pods/3234b8da-8044-4b73-af0d-898cbc971373/volumes" Jul 08 20:05:13.989462 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:05:13.989424 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-960620/sub-test-model-960620-kserve-56c9c7fb9c-djt49"] Jul 08 20:05:13.989887 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:05:13.989771 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3234b8da-8044-4b73-af0d-898cbc971373" containerName="main" Jul 08 20:05:13.989887 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:05:13.989783 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3234b8da-8044-4b73-af0d-898cbc971373" containerName="main" Jul 08 20:05:13.989887 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:05:13.989885 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3234b8da-8044-4b73-af0d-898cbc971373" containerName="main" Jul 08 20:05:13.994390 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:05:13.994368 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-960620/sub-test-model-960620-kserve-56c9c7fb9c-djt49" Jul 08 20:05:13.998007 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:05:13.997944 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-960620\"/\"sub-test-model-960620-kserve-self-signed-certs\"" Jul 08 20:05:13.998266 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:05:13.998107 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-960620\"/\"kube-root-ca.crt\"" Jul 08 20:05:13.998266 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:05:13.998129 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-960620\"/\"default-dockercfg-v27lv\"" Jul 08 20:05:13.998266 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:05:13.998129 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-960620\"/\"openshift-service-ca.crt\"" Jul 08 20:05:14.003891 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:05:14.003854 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-960620/sub-test-model-960620-kserve-56c9c7fb9c-djt49"] Jul 08 20:05:14.018626 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:05:14.018597 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:05:14.067935 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:05:14.067902 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76780d61-2bf1-48b2-b493-175b27c6e454-home\") pod \"sub-test-model-960620-kserve-56c9c7fb9c-djt49\" (UID: \"76780d61-2bf1-48b2-b493-175b27c6e454\") " pod="ai-tenant-e2e-shared-a-960620/sub-test-model-960620-kserve-56c9c7fb9c-djt49" Jul 08 20:05:14.067935 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:05:14.067936 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76780d61-2bf1-48b2-b493-175b27c6e454-dshm\") pod \"sub-test-model-960620-kserve-56c9c7fb9c-djt49\" (UID: \"76780d61-2bf1-48b2-b493-175b27c6e454\") " pod="ai-tenant-e2e-shared-a-960620/sub-test-model-960620-kserve-56c9c7fb9c-djt49" Jul 08 20:05:14.068131 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:05:14.067985 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76780d61-2bf1-48b2-b493-175b27c6e454-model-cache\") pod \"sub-test-model-960620-kserve-56c9c7fb9c-djt49\" (UID: \"76780d61-2bf1-48b2-b493-175b27c6e454\") " pod="ai-tenant-e2e-shared-a-960620/sub-test-model-960620-kserve-56c9c7fb9c-djt49" Jul 08 20:05:14.068131 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:05:14.068005 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76780d61-2bf1-48b2-b493-175b27c6e454-tls-certs\") pod \"sub-test-model-960620-kserve-56c9c7fb9c-djt49\" (UID: \"76780d61-2bf1-48b2-b493-175b27c6e454\") " pod="ai-tenant-e2e-shared-a-960620/sub-test-model-960620-kserve-56c9c7fb9c-djt49" Jul 08 20:05:14.068131 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:05:14.068099 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jswm8\" (UniqueName: \"kubernetes.io/projected/76780d61-2bf1-48b2-b493-175b27c6e454-kube-api-access-jswm8\") pod \"sub-test-model-960620-kserve-56c9c7fb9c-djt49\" (UID: \"76780d61-2bf1-48b2-b493-175b27c6e454\") " pod="ai-tenant-e2e-shared-a-960620/sub-test-model-960620-kserve-56c9c7fb9c-djt49" Jul 08 20:05:14.068226 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:05:14.068153 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76780d61-2bf1-48b2-b493-175b27c6e454-tmp-dir\") pod \"sub-test-model-960620-kserve-56c9c7fb9c-djt49\" (UID: \"76780d61-2bf1-48b2-b493-175b27c6e454\") " pod="ai-tenant-e2e-shared-a-960620/sub-test-model-960620-kserve-56c9c7fb9c-djt49" Jul 08 20:05:14.168745 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:05:14.168712 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76780d61-2bf1-48b2-b493-175b27c6e454-home\") pod \"sub-test-model-960620-kserve-56c9c7fb9c-djt49\" (UID: \"76780d61-2bf1-48b2-b493-175b27c6e454\") " pod="ai-tenant-e2e-shared-a-960620/sub-test-model-960620-kserve-56c9c7fb9c-djt49" Jul 08 20:05:14.168745 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:05:14.168748 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76780d61-2bf1-48b2-b493-175b27c6e454-dshm\") pod \"sub-test-model-960620-kserve-56c9c7fb9c-djt49\" (UID: \"76780d61-2bf1-48b2-b493-175b27c6e454\") " pod="ai-tenant-e2e-shared-a-960620/sub-test-model-960620-kserve-56c9c7fb9c-djt49" Jul 08 20:05:14.169015 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:05:14.168770 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76780d61-2bf1-48b2-b493-175b27c6e454-model-cache\") pod \"sub-test-model-960620-kserve-56c9c7fb9c-djt49\" (UID: \"76780d61-2bf1-48b2-b493-175b27c6e454\") " pod="ai-tenant-e2e-shared-a-960620/sub-test-model-960620-kserve-56c9c7fb9c-djt49" Jul 08 20:05:14.169015 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:05:14.168796 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76780d61-2bf1-48b2-b493-175b27c6e454-tls-certs\") pod \"sub-test-model-960620-kserve-56c9c7fb9c-djt49\" (UID: \"76780d61-2bf1-48b2-b493-175b27c6e454\") " pod="ai-tenant-e2e-shared-a-960620/sub-test-model-960620-kserve-56c9c7fb9c-djt49" Jul 08 20:05:14.169015 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:05:14.168852 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jswm8\" (UniqueName: \"kubernetes.io/projected/76780d61-2bf1-48b2-b493-175b27c6e454-kube-api-access-jswm8\") pod \"sub-test-model-960620-kserve-56c9c7fb9c-djt49\" (UID: \"76780d61-2bf1-48b2-b493-175b27c6e454\") " pod="ai-tenant-e2e-shared-a-960620/sub-test-model-960620-kserve-56c9c7fb9c-djt49" Jul 08 20:05:14.169015 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:05:14.168923 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76780d61-2bf1-48b2-b493-175b27c6e454-tmp-dir\") pod \"sub-test-model-960620-kserve-56c9c7fb9c-djt49\" (UID: \"76780d61-2bf1-48b2-b493-175b27c6e454\") " pod="ai-tenant-e2e-shared-a-960620/sub-test-model-960620-kserve-56c9c7fb9c-djt49" Jul 08 20:05:14.169359 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:05:14.169334 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76780d61-2bf1-48b2-b493-175b27c6e454-home\") pod \"sub-test-model-960620-kserve-56c9c7fb9c-djt49\" (UID: \"76780d61-2bf1-48b2-b493-175b27c6e454\") " pod="ai-tenant-e2e-shared-a-960620/sub-test-model-960620-kserve-56c9c7fb9c-djt49" Jul 08 20:05:14.169463 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:05:14.169437 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76780d61-2bf1-48b2-b493-175b27c6e454-model-cache\") pod \"sub-test-model-960620-kserve-56c9c7fb9c-djt49\" (UID: \"76780d61-2bf1-48b2-b493-175b27c6e454\") " pod="ai-tenant-e2e-shared-a-960620/sub-test-model-960620-kserve-56c9c7fb9c-djt49" Jul 08 20:05:14.169503 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:05:14.169490 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76780d61-2bf1-48b2-b493-175b27c6e454-tmp-dir\") pod \"sub-test-model-960620-kserve-56c9c7fb9c-djt49\" (UID: \"76780d61-2bf1-48b2-b493-175b27c6e454\") " pod="ai-tenant-e2e-shared-a-960620/sub-test-model-960620-kserve-56c9c7fb9c-djt49" Jul 08 20:05:14.171299 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:05:14.171275 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76780d61-2bf1-48b2-b493-175b27c6e454-dshm\") pod \"sub-test-model-960620-kserve-56c9c7fb9c-djt49\" (UID: \"76780d61-2bf1-48b2-b493-175b27c6e454\") " pod="ai-tenant-e2e-shared-a-960620/sub-test-model-960620-kserve-56c9c7fb9c-djt49" Jul 08 20:05:14.171471 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:05:14.171451 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76780d61-2bf1-48b2-b493-175b27c6e454-tls-certs\") pod \"sub-test-model-960620-kserve-56c9c7fb9c-djt49\" (UID: \"76780d61-2bf1-48b2-b493-175b27c6e454\") " pod="ai-tenant-e2e-shared-a-960620/sub-test-model-960620-kserve-56c9c7fb9c-djt49" Jul 08 20:05:14.176513 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:05:14.176487 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jswm8\" (UniqueName: \"kubernetes.io/projected/76780d61-2bf1-48b2-b493-175b27c6e454-kube-api-access-jswm8\") pod \"sub-test-model-960620-kserve-56c9c7fb9c-djt49\" (UID: \"76780d61-2bf1-48b2-b493-175b27c6e454\") " pod="ai-tenant-e2e-shared-a-960620/sub-test-model-960620-kserve-56c9c7fb9c-djt49" Jul 08 20:05:14.305919 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:05:14.305794 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-960620/sub-test-model-960620-kserve-56c9c7fb9c-djt49" Jul 08 20:05:14.435556 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:05:14.435532 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-960620/sub-test-model-960620-kserve-56c9c7fb9c-djt49"] Jul 08 20:05:14.437780 ip-10-0-142-195 kubenswrapper[2566]: W0708 20:05:14.437753 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod76780d61_2bf1_48b2_b493_175b27c6e454.slice/crio-2f8dfa9763724a0026e448edcd201fb848030245921166d6c9c53ab98fbc71bf WatchSource:0}: Error finding container 2f8dfa9763724a0026e448edcd201fb848030245921166d6c9c53ab98fbc71bf: Status 404 returned error can't find the container with id 2f8dfa9763724a0026e448edcd201fb848030245921166d6c9c53ab98fbc71bf Jul 08 20:05:15.204062 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:05:15.204026 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-960620/sub-test-model-960620-kserve-56c9c7fb9c-djt49" event={"ID":"76780d61-2bf1-48b2-b493-175b27c6e454","Type":"ContainerStarted","Data":"c1c2b0889474741436d420eab00c455f1d29192621485ff10f635bb2c1073a14"} Jul 08 20:05:15.204486 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:05:15.204070 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-960620/sub-test-model-960620-kserve-56c9c7fb9c-djt49" event={"ID":"76780d61-2bf1-48b2-b493-175b27c6e454","Type":"ContainerStarted","Data":"2f8dfa9763724a0026e448edcd201fb848030245921166d6c9c53ab98fbc71bf"} Jul 08 20:05:15.220668 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:05:15.220618 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-960620/sub-test-model-960620-kserve-56c9c7fb9c-djt49" podStartSLOduration=2.220601484 podStartE2EDuration="2.220601484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 20:05:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:05:15.219788726 +0000 UTC m=+2481.105978772" watchObservedRunningTime="2026-07-08 20:05:15.220601484 +0000 UTC m=+2481.106791531" Jul 08 20:05:17.488294 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:05:17.488260 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 20:05:24.306420 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:05:24.306388 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-960620/sub-test-model-960620-kserve-56c9c7fb9c-djt49" Jul 08 20:05:24.306936 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:05:24.306445 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-960620/sub-test-model-960620-kserve-56c9c7fb9c-djt49" Jul 08 20:05:24.318542 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:05:24.318513 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-960620/sub-test-model-960620-kserve-56c9c7fb9c-djt49" Jul 08 20:05:25.268561 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:05:25.268465 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-960620/sub-test-model-960620-kserve-56c9c7fb9c-djt49" Jul 08 20:05:34.778319 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:05:34.778284 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:05:34.877175 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:05:34.877129 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:05:38.803552 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:05:38.803525 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:05:43.082573 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:05:43.082524 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 20:06:05.986207 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:06:05.986169 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 20:06:21.490411 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:06:21.490373 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 20:06:28.070725 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:06:28.070687 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-960620/sub-test-model-960620-kserve-56c9c7fb9c-djt49"] Jul 08 20:06:28.082221 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:06:28.082191 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:06:29.013722 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:06:29.013684 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:06:38.821846 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:06:38.821808 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:06:41.588311 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:06:41.588273 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 20:06:43.092703 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:06:43.092655 2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-960620/sub-test-model-960620-kserve-56c9c7fb9c-djt49" podUID="76780d61-2bf1-48b2-b493-175b27c6e454" containerName="main" containerID="cri-o://c1c2b0889474741436d420eab00c455f1d29192621485ff10f635bb2c1073a14" gracePeriod=45 Jul 08 20:06:43.348256 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:06:43.348197 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-960620/sub-test-model-960620-kserve-56c9c7fb9c-djt49" Jul 08 20:06:43.490801 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:06:43.490771 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76780d61-2bf1-48b2-b493-175b27c6e454-tmp-dir\") pod \"76780d61-2bf1-48b2-b493-175b27c6e454\" (UID: \"76780d61-2bf1-48b2-b493-175b27c6e454\") " Jul 08 20:06:43.490988 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:06:43.490825 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76780d61-2bf1-48b2-b493-175b27c6e454-dshm\") pod \"76780d61-2bf1-48b2-b493-175b27c6e454\" (UID: \"76780d61-2bf1-48b2-b493-175b27c6e454\") " Jul 08 20:06:43.490988 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:06:43.490855 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76780d61-2bf1-48b2-b493-175b27c6e454-tls-certs\") pod \"76780d61-2bf1-48b2-b493-175b27c6e454\" (UID: \"76780d61-2bf1-48b2-b493-175b27c6e454\") " Jul 08 20:06:43.490988 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:06:43.490913 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76780d61-2bf1-48b2-b493-175b27c6e454-home\") pod \"76780d61-2bf1-48b2-b493-175b27c6e454\" (UID: \"76780d61-2bf1-48b2-b493-175b27c6e454\") " Jul 08 20:06:43.491153 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:06:43.491007 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jswm8\" (UniqueName: \"kubernetes.io/projected/76780d61-2bf1-48b2-b493-175b27c6e454-kube-api-access-jswm8\") pod \"76780d61-2bf1-48b2-b493-175b27c6e454\" (UID: \"76780d61-2bf1-48b2-b493-175b27c6e454\") " Jul 08 20:06:43.491153 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:06:43.491048 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76780d61-2bf1-48b2-b493-175b27c6e454-model-cache\") pod \"76780d61-2bf1-48b2-b493-175b27c6e454\" (UID: \"76780d61-2bf1-48b2-b493-175b27c6e454\") " Jul 08 20:06:43.491153 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:06:43.491064 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76780d61-2bf1-48b2-b493-175b27c6e454-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "76780d61-2bf1-48b2-b493-175b27c6e454" (UID: "76780d61-2bf1-48b2-b493-175b27c6e454"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 20:06:43.491286 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:06:43.491223 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76780d61-2bf1-48b2-b493-175b27c6e454-home" (OuterVolumeSpecName: "home") pod "76780d61-2bf1-48b2-b493-175b27c6e454" (UID: "76780d61-2bf1-48b2-b493-175b27c6e454"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 20:06:43.491387 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:06:43.491364 2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76780d61-2bf1-48b2-b493-175b27c6e454-tmp-dir\") on node \"ip-10-0-142-195.ec2.internal\" DevicePath \"\"" Jul 08 20:06:43.491458 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:06:43.491391 2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76780d61-2bf1-48b2-b493-175b27c6e454-home\") on node \"ip-10-0-142-195.ec2.internal\" DevicePath \"\"" Jul 08 20:06:43.491458 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:06:43.491351 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76780d61-2bf1-48b2-b493-175b27c6e454-model-cache" (OuterVolumeSpecName: "model-cache") pod "76780d61-2bf1-48b2-b493-175b27c6e454" (UID: "76780d61-2bf1-48b2-b493-175b27c6e454"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 20:06:43.492981 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:06:43.492955 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76780d61-2bf1-48b2-b493-175b27c6e454-dshm" (OuterVolumeSpecName: "dshm") pod "76780d61-2bf1-48b2-b493-175b27c6e454" (UID: "76780d61-2bf1-48b2-b493-175b27c6e454"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 20:06:43.493079 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:06:43.492976 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76780d61-2bf1-48b2-b493-175b27c6e454-kube-api-access-jswm8" (OuterVolumeSpecName: "kube-api-access-jswm8") pod "76780d61-2bf1-48b2-b493-175b27c6e454" (UID: "76780d61-2bf1-48b2-b493-175b27c6e454"). InnerVolumeSpecName "kube-api-access-jswm8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 20:06:43.493079 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:06:43.493036 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76780d61-2bf1-48b2-b493-175b27c6e454-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "76780d61-2bf1-48b2-b493-175b27c6e454" (UID: "76780d61-2bf1-48b2-b493-175b27c6e454"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 20:06:43.592428 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:06:43.592398 2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76780d61-2bf1-48b2-b493-175b27c6e454-model-cache\") on node \"ip-10-0-142-195.ec2.internal\" DevicePath \"\"" Jul 08 20:06:43.592428 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:06:43.592424 2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76780d61-2bf1-48b2-b493-175b27c6e454-dshm\") on node \"ip-10-0-142-195.ec2.internal\" DevicePath \"\"" Jul 08 20:06:43.592428 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:06:43.592433 2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76780d61-2bf1-48b2-b493-175b27c6e454-tls-certs\") on node \"ip-10-0-142-195.ec2.internal\" DevicePath \"\"" Jul 08 20:06:43.592428 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:06:43.592441 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jswm8\" (UniqueName: \"kubernetes.io/projected/76780d61-2bf1-48b2-b493-175b27c6e454-kube-api-access-jswm8\") on node \"ip-10-0-142-195.ec2.internal\" DevicePath \"\"" Jul 08 20:06:43.618921 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:06:43.618827 2566 generic.go:356] "Generic (PLEG): container finished" podID="76780d61-2bf1-48b2-b493-175b27c6e454" containerID="c1c2b0889474741436d420eab00c455f1d29192621485ff10f635bb2c1073a14" exitCode=0 Jul 08 20:06:43.618921 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:06:43.618887 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-960620/sub-test-model-960620-kserve-56c9c7fb9c-djt49" Jul 08 20:06:43.619103 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:06:43.618923 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-960620/sub-test-model-960620-kserve-56c9c7fb9c-djt49" event={"ID":"76780d61-2bf1-48b2-b493-175b27c6e454","Type":"ContainerDied","Data":"c1c2b0889474741436d420eab00c455f1d29192621485ff10f635bb2c1073a14"} Jul 08 20:06:43.619103 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:06:43.618962 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-960620/sub-test-model-960620-kserve-56c9c7fb9c-djt49" event={"ID":"76780d61-2bf1-48b2-b493-175b27c6e454","Type":"ContainerDied","Data":"2f8dfa9763724a0026e448edcd201fb848030245921166d6c9c53ab98fbc71bf"} Jul 08 20:06:43.619103 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:06:43.618981 2566 scope.go:117] "RemoveContainer" containerID="c1c2b0889474741436d420eab00c455f1d29192621485ff10f635bb2c1073a14" Jul 08 20:06:43.628034 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:06:43.628016 2566 scope.go:117] "RemoveContainer" containerID="c1c2b0889474741436d420eab00c455f1d29192621485ff10f635bb2c1073a14" Jul 08 20:06:43.628279 ip-10-0-142-195 kubenswrapper[2566]: E0708 20:06:43.628259 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1c2b0889474741436d420eab00c455f1d29192621485ff10f635bb2c1073a14\": container with ID starting with c1c2b0889474741436d420eab00c455f1d29192621485ff10f635bb2c1073a14 not found: ID does not exist" containerID="c1c2b0889474741436d420eab00c455f1d29192621485ff10f635bb2c1073a14" Jul 08 20:06:43.628354 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:06:43.628287 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1c2b0889474741436d420eab00c455f1d29192621485ff10f635bb2c1073a14"} err="failed to get container status \"c1c2b0889474741436d420eab00c455f1d29192621485ff10f635bb2c1073a14\": rpc error: code = NotFound desc = could not find container \"c1c2b0889474741436d420eab00c455f1d29192621485ff10f635bb2c1073a14\": container with ID starting with c1c2b0889474741436d420eab00c455f1d29192621485ff10f635bb2c1073a14 not found: ID does not exist" Jul 08 20:06:43.640281 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:06:43.640260 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-960620/sub-test-model-960620-kserve-56c9c7fb9c-djt49"] Jul 08 20:06:43.646203 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:06:43.646183 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-960620/sub-test-model-960620-kserve-56c9c7fb9c-djt49"] Jul 08 20:06:44.622854 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:06:44.622817 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76780d61-2bf1-48b2-b493-175b27c6e454" path="/var/lib/kubelet/pods/76780d61-2bf1-48b2-b493-175b27c6e454/volumes" Jul 08 20:06:54.800002 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:06:54.799974 2566 scope.go:117] "RemoveContainer" containerID="b036c48d0f65af812dd710329be586dd0e413292f9621a85d429a042270074b5" Jul 08 20:07:00.409259 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:07:00.409221 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:07:05.332877 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:07:05.332836 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 20:07:08.266356 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:07:08.266323 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:07:08.364730 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:07:08.364698 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:07:16.632770 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:07:16.632685 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:07:16.733287 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:07:16.733256 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:07:22.781841 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:07:22.781805 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 20:07:38.303821 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:07:38.303787 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 20:07:45.442376 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:07:45.442344 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:07:46.416386 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:07:46.416358 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:07:47.514671 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:07:47.514641 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:07:47.996239 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:07:47.996209 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:07:48.083704 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:07:48.083674 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:07:48.189214 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:07:48.189179 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-960620-587467c688-qqnm9"] Jul 08 20:07:48.189470 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:07:48.189430 2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-a-960620-587467c688-qqnm9" podUID="c651106f-7cba-4f10-ad27-590996640cdb" containerName="maas-api" containerID="cri-o://e739fde5422a786c5f2aa769469a92d0ae91f00eb256b7c90f77643cabf67104" gracePeriod=30 Jul 08 20:07:48.219423 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:07:48.219395 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:07:48.433609 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:07:48.433587 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-960620-587467c688-qqnm9" Jul 08 20:07:48.544409 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:07:48.544375 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c651106f-7cba-4f10-ad27-590996640cdb-maas-api-tls\") pod \"c651106f-7cba-4f10-ad27-590996640cdb\" (UID: \"c651106f-7cba-4f10-ad27-590996640cdb\") " Jul 08 20:07:48.544853 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:07:48.544456 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c651106f-7cba-4f10-ad27-590996640cdb-openshift-service-ca\") pod \"c651106f-7cba-4f10-ad27-590996640cdb\" (UID: \"c651106f-7cba-4f10-ad27-590996640cdb\") " Jul 08 20:07:48.544853 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:07:48.544495 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fnt6c\" (UniqueName: \"kubernetes.io/projected/c651106f-7cba-4f10-ad27-590996640cdb-kube-api-access-fnt6c\") pod \"c651106f-7cba-4f10-ad27-590996640cdb\" (UID: \"c651106f-7cba-4f10-ad27-590996640cdb\") " Jul 08 20:07:48.544853 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:07:48.544786 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c651106f-7cba-4f10-ad27-590996640cdb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c651106f-7cba-4f10-ad27-590996640cdb" (UID: "c651106f-7cba-4f10-ad27-590996640cdb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 20:07:48.546422 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:07:48.546400 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c651106f-7cba-4f10-ad27-590996640cdb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c651106f-7cba-4f10-ad27-590996640cdb" (UID: "c651106f-7cba-4f10-ad27-590996640cdb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 20:07:48.546616 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:07:48.546595 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c651106f-7cba-4f10-ad27-590996640cdb-kube-api-access-fnt6c" (OuterVolumeSpecName: "kube-api-access-fnt6c") pod "c651106f-7cba-4f10-ad27-590996640cdb" (UID: "c651106f-7cba-4f10-ad27-590996640cdb"). InnerVolumeSpecName "kube-api-access-fnt6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 20:07:48.645700 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:07:48.645673 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c651106f-7cba-4f10-ad27-590996640cdb-maas-api-tls\") on node \"ip-10-0-142-195.ec2.internal\" DevicePath \"\"" Jul 08 20:07:48.645700 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:07:48.645695 2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c651106f-7cba-4f10-ad27-590996640cdb-openshift-service-ca\") on node \"ip-10-0-142-195.ec2.internal\" DevicePath \"\"" Jul 08 20:07:48.645852 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:07:48.645705 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnt6c\" (UniqueName: \"kubernetes.io/projected/c651106f-7cba-4f10-ad27-590996640cdb-kube-api-access-fnt6c\") on node \"ip-10-0-142-195.ec2.internal\" DevicePath \"\"" Jul 08 20:07:48.912262 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:07:48.912228 2566 generic.go:356] "Generic (PLEG): container finished" podID="c651106f-7cba-4f10-ad27-590996640cdb" containerID="e739fde5422a786c5f2aa769469a92d0ae91f00eb256b7c90f77643cabf67104" exitCode=0 Jul 08 20:07:48.912429 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:07:48.912265 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-960620-587467c688-qqnm9" event={"ID":"c651106f-7cba-4f10-ad27-590996640cdb","Type":"ContainerDied","Data":"e739fde5422a786c5f2aa769469a92d0ae91f00eb256b7c90f77643cabf67104"} Jul 08 20:07:48.912429 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:07:48.912316 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-960620-587467c688-qqnm9" event={"ID":"c651106f-7cba-4f10-ad27-590996640cdb","Type":"ContainerDied","Data":"67d248280b81cbe575bd47939bbb1f9176419dff567692b36053da863d6204a1"} Jul 08 20:07:48.912429 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:07:48.912338 2566 scope.go:117] "RemoveContainer" containerID="e739fde5422a786c5f2aa769469a92d0ae91f00eb256b7c90f77643cabf67104" Jul 08 20:07:48.912429 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:07:48.912275 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-960620-587467c688-qqnm9" Jul 08 20:07:48.920748 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:07:48.920732 2566 scope.go:117] "RemoveContainer" containerID="e739fde5422a786c5f2aa769469a92d0ae91f00eb256b7c90f77643cabf67104" Jul 08 20:07:48.921010 ip-10-0-142-195 kubenswrapper[2566]: E0708 20:07:48.920993 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e739fde5422a786c5f2aa769469a92d0ae91f00eb256b7c90f77643cabf67104\": container with ID starting with e739fde5422a786c5f2aa769469a92d0ae91f00eb256b7c90f77643cabf67104 not found: ID does not exist" containerID="e739fde5422a786c5f2aa769469a92d0ae91f00eb256b7c90f77643cabf67104" Jul 08 20:07:48.921078 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:07:48.921015 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e739fde5422a786c5f2aa769469a92d0ae91f00eb256b7c90f77643cabf67104"} err="failed to get container status \"e739fde5422a786c5f2aa769469a92d0ae91f00eb256b7c90f77643cabf67104\": rpc error: code = NotFound desc = could not find container \"e739fde5422a786c5f2aa769469a92d0ae91f00eb256b7c90f77643cabf67104\": container with ID starting with e739fde5422a786c5f2aa769469a92d0ae91f00eb256b7c90f77643cabf67104 not found: ID does not exist" Jul 08 20:07:48.939709 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:07:48.939681 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-960620-587467c688-qqnm9"] Jul 08 20:07:48.942257 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:07:48.942238 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-960620-587467c688-qqnm9"] Jul 08 20:07:50.622697 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:07:50.622640 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c651106f-7cba-4f10-ad27-590996640cdb" path="/var/lib/kubelet/pods/c651106f-7cba-4f10-ad27-590996640cdb/volumes" Jul 08 20:07:51.688726 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:07:51.688691 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 20:07:53.018724 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:07:53.018687 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:08:07.407431 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:08:07.407397 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:08:20.170139 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:08:20.170100 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:08:30.148115 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:08:30.148081 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:08:30.423842 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:08:30.423810 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:08:30.676334 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:08:30.676240 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:08:31.013136 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:08:31.013044 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:08:31.410412 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:08:31.410382 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:08:31.523843 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:08:31.523812 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:08:31.628215 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:08:31.628182 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:08:31.734798 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:08:31.734697 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-ceba08-c745d7b95-vlx88"] Jul 08 20:08:31.735000 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:08:31.734975 2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-b-ceba08-c745d7b95-vlx88" podUID="a84496f5-4fa6-42b5-ae34-4903466fc45f" containerName="maas-api" containerID="cri-o://39bb55d5692f1f7035b8270081051f8274efdfcef137ba47906a2bb777fef627" gracePeriod=30 Jul 08 20:08:31.757150 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:08:31.757124 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:08:31.980500 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:08:31.980475 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-ceba08-c745d7b95-vlx88" Jul 08 20:08:32.103337 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:08:32.103249 2566 generic.go:356] "Generic (PLEG): container finished" podID="a84496f5-4fa6-42b5-ae34-4903466fc45f" containerID="39bb55d5692f1f7035b8270081051f8274efdfcef137ba47906a2bb777fef627" exitCode=0 Jul 08 20:08:32.103337 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:08:32.103301 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-ceba08-c745d7b95-vlx88" Jul 08 20:08:32.103580 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:08:32.103340 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-ceba08-c745d7b95-vlx88" event={"ID":"a84496f5-4fa6-42b5-ae34-4903466fc45f","Type":"ContainerDied","Data":"39bb55d5692f1f7035b8270081051f8274efdfcef137ba47906a2bb777fef627"} Jul 08 20:08:32.103580 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:08:32.103398 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-ceba08-c745d7b95-vlx88" event={"ID":"a84496f5-4fa6-42b5-ae34-4903466fc45f","Type":"ContainerDied","Data":"fd6f739a7fedc22ddb7fa198e9624933353121cc2e00d6717bfdd7fb3e7b8c75"} Jul 08 20:08:32.103580 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:08:32.103428 2566 scope.go:117] "RemoveContainer" containerID="39bb55d5692f1f7035b8270081051f8274efdfcef137ba47906a2bb777fef627" Jul 08 20:08:32.113887 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:08:32.113835 2566 scope.go:117] "RemoveContainer" containerID="39bb55d5692f1f7035b8270081051f8274efdfcef137ba47906a2bb777fef627" Jul 08 20:08:32.114166 ip-10-0-142-195 kubenswrapper[2566]: E0708 20:08:32.114148 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39bb55d5692f1f7035b8270081051f8274efdfcef137ba47906a2bb777fef627\": container with ID starting with 39bb55d5692f1f7035b8270081051f8274efdfcef137ba47906a2bb777fef627 not found: ID does not exist" containerID="39bb55d5692f1f7035b8270081051f8274efdfcef137ba47906a2bb777fef627" Jul 08 20:08:32.114230 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:08:32.114177 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39bb55d5692f1f7035b8270081051f8274efdfcef137ba47906a2bb777fef627"} err="failed to get container status \"39bb55d5692f1f7035b8270081051f8274efdfcef137ba47906a2bb777fef627\": rpc error: code = NotFound desc = could not find container \"39bb55d5692f1f7035b8270081051f8274efdfcef137ba47906a2bb777fef627\": container with ID starting with 39bb55d5692f1f7035b8270081051f8274efdfcef137ba47906a2bb777fef627 not found: ID does not exist" Jul 08 20:08:32.141743 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:08:32.141709 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r94jk\" (UniqueName: \"kubernetes.io/projected/a84496f5-4fa6-42b5-ae34-4903466fc45f-kube-api-access-r94jk\") pod \"a84496f5-4fa6-42b5-ae34-4903466fc45f\" (UID: \"a84496f5-4fa6-42b5-ae34-4903466fc45f\") " Jul 08 20:08:32.141899 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:08:32.141770 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a84496f5-4fa6-42b5-ae34-4903466fc45f-openshift-service-ca\") pod \"a84496f5-4fa6-42b5-ae34-4903466fc45f\" (UID: \"a84496f5-4fa6-42b5-ae34-4903466fc45f\") " Jul 08 20:08:32.141899 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:08:32.141793 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a84496f5-4fa6-42b5-ae34-4903466fc45f-maas-api-tls\") pod \"a84496f5-4fa6-42b5-ae34-4903466fc45f\" (UID: \"a84496f5-4fa6-42b5-ae34-4903466fc45f\") " Jul 08 20:08:32.142206 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:08:32.142180 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a84496f5-4fa6-42b5-ae34-4903466fc45f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a84496f5-4fa6-42b5-ae34-4903466fc45f" (UID: "a84496f5-4fa6-42b5-ae34-4903466fc45f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 20:08:32.144032 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:08:32.144005 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a84496f5-4fa6-42b5-ae34-4903466fc45f-kube-api-access-r94jk" (OuterVolumeSpecName: "kube-api-access-r94jk") pod "a84496f5-4fa6-42b5-ae34-4903466fc45f" (UID: "a84496f5-4fa6-42b5-ae34-4903466fc45f"). InnerVolumeSpecName "kube-api-access-r94jk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 20:08:32.144105 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:08:32.144026 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a84496f5-4fa6-42b5-ae34-4903466fc45f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a84496f5-4fa6-42b5-ae34-4903466fc45f" (UID: "a84496f5-4fa6-42b5-ae34-4903466fc45f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 20:08:32.242485 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:08:32.242448 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r94jk\" (UniqueName: \"kubernetes.io/projected/a84496f5-4fa6-42b5-ae34-4903466fc45f-kube-api-access-r94jk\") on node \"ip-10-0-142-195.ec2.internal\" DevicePath \"\"" Jul 08 20:08:32.242485 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:08:32.242478 2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a84496f5-4fa6-42b5-ae34-4903466fc45f-openshift-service-ca\") on node \"ip-10-0-142-195.ec2.internal\" DevicePath \"\"" Jul 08 20:08:32.242485 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:08:32.242488 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a84496f5-4fa6-42b5-ae34-4903466fc45f-maas-api-tls\") on node \"ip-10-0-142-195.ec2.internal\" DevicePath \"\"" Jul 08 20:08:32.424662 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:08:32.424629 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-ceba08-c745d7b95-vlx88"] Jul 08 20:08:32.427839 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:08:32.427812 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-ceba08-c745d7b95-vlx88"] Jul 08 20:08:32.623715 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:08:32.623684 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a84496f5-4fa6-42b5-ae34-4903466fc45f" path="/var/lib/kubelet/pods/a84496f5-4fa6-42b5-ae34-4903466fc45f/volumes" Jul 08 20:08:34.284065 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:08:34.284027 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9vx6x"] Jul 08 20:08:37.364836 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:08:37.364799 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:08:37.424375 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:08:37.424343 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:08:37.615711 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:08:37.615618 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:08:37.868042 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:08:37.867947 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:08:43.849321 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:08:43.849288 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-5kkcw_8080820c-03c1-4be8-8b2c-f8bbf00d5182/server/0.log" Jul 08 20:08:44.368734 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:08:44.368702 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-77ddc6fd87-m2dgt_926adae9-5aea-4f33-8467-166b43bb6a0d/copy-manifests/0.log" Jul 08 20:08:44.375501 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:08:44.375470 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-77ddc6fd87-m2dgt_926adae9-5aea-4f33-8467-166b43bb6a0d/manager/0.log" Jul 08 20:08:46.565464 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:08:46.565385 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-9vx6x_4a97688b-b2d6-4c40-ad27-ada055b7f320/limitador/0.log" Jul 08 20:08:47.185360 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:08:47.185329 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-j57db_13b2effd-726c-4425-aff2-45d8df9443ef/discovery/0.log" Jul 08 20:08:47.945854 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:08:47.945816 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5846dc5c45-fgr6x_cee815ab-2b57-4584-8b6c-a8748c849be9/router/0.log" Jul 08 20:08:48.788481 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:08:48.788448 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-zs5q6_3854bf68-0ce0-401b-9aa7-8bc92c2ccf89/main/0.log" Jul 08 20:08:53.543610 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:08:53.543575 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:08:53.622235 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:08:53.622196 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" limit=1024 Jul 08 20:08:56.919151 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:08:56.919117 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-qdfqb_24b6882e-3f1e-4c28-b128-2b16f3478940/global-pull-secret-syncer/0.log" Jul 08 20:08:56.973783 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:08:56.973742 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-5kt6t_42578ec4-91d1-4d4d-b9a1-12a559fee154/konnectivity-agent/0.log" Jul 08 20:08:57.089777 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:08:57.089750 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-195.ec2.internal_47fb7cb7c700e5809cbc3bdae3a4659d/haproxy/0.log" Jul 08 20:09:02.586723 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:02.586679 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-9vx6x_4a97688b-b2d6-4c40-ad27-ada055b7f320/limitador/0.log" Jul 08 20:09:04.223468 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:04.223434 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a0321d1f-21e9-4d20-af13-ed8a1fa71638/alertmanager/0.log" Jul 08 20:09:04.247910 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:04.247887 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a0321d1f-21e9-4d20-af13-ed8a1fa71638/config-reloader/0.log" Jul 08 20:09:04.271102 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:04.271077 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a0321d1f-21e9-4d20-af13-ed8a1fa71638/kube-rbac-proxy-web/0.log" Jul 08 20:09:04.293160 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:04.293140 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a0321d1f-21e9-4d20-af13-ed8a1fa71638/kube-rbac-proxy/0.log" Jul 08 20:09:04.316472 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:04.316447 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a0321d1f-21e9-4d20-af13-ed8a1fa71638/kube-rbac-proxy-metric/0.log" Jul 08 20:09:04.336669 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:04.336648 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a0321d1f-21e9-4d20-af13-ed8a1fa71638/prom-label-proxy/0.log" Jul 08 20:09:04.360880 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:04.360843 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a0321d1f-21e9-4d20-af13-ed8a1fa71638/init-config-reloader/0.log" Jul 08 20:09:04.417771 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:04.417746 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-64786d9cbb-v4w6v_6e934fe3-7407-4bd1-88cc-21be3143047c/cluster-monitoring-operator/0.log" Jul 08 20:09:04.443984 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:04.443963 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-nz2hj_322a5d1a-2482-4930-a546-31774f7f0c61/kube-state-metrics/0.log" Jul 08 20:09:04.460762 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:04.460743 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-nz2hj_322a5d1a-2482-4930-a546-31774f7f0c61/kube-rbac-proxy-main/0.log" Jul 08 20:09:04.483743 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:04.483679 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-nz2hj_322a5d1a-2482-4930-a546-31774f7f0c61/kube-rbac-proxy-self/0.log" Jul 08 20:09:04.510401 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:04.510379 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-98b9b7785-bj4qt_e47f05c9-7873-4be6-a51f-ed0ff59dd3fd/metrics-server/0.log" Jul 08 20:09:04.698519 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:04.698492 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lps28_406246a8-ab16-4792-8582-83f15d00cafc/node-exporter/0.log" Jul 08 20:09:04.719726 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:04.719709 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lps28_406246a8-ab16-4792-8582-83f15d00cafc/kube-rbac-proxy/0.log" Jul 08 20:09:04.754497 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:04.754437 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lps28_406246a8-ab16-4792-8582-83f15d00cafc/init-textfile/0.log" Jul 08 20:09:05.343482 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:05.343453 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5c4467cbb7-wlnbt_a7df7b87-f170-426a-89b0-9cf26efdeb29/thanos-query/0.log" Jul 08 20:09:05.365595 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:05.365571 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5c4467cbb7-wlnbt_a7df7b87-f170-426a-89b0-9cf26efdeb29/kube-rbac-proxy-web/0.log" Jul 08 20:09:05.386942 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:05.386921 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5c4467cbb7-wlnbt_a7df7b87-f170-426a-89b0-9cf26efdeb29/kube-rbac-proxy/0.log" Jul 08 20:09:05.408760 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:05.408741 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5c4467cbb7-wlnbt_a7df7b87-f170-426a-89b0-9cf26efdeb29/prom-label-proxy/0.log" Jul 08 20:09:05.428519 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:05.428497 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5c4467cbb7-wlnbt_a7df7b87-f170-426a-89b0-9cf26efdeb29/kube-rbac-proxy-rules/0.log" Jul 08 20:09:05.452889 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:05.452856 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5c4467cbb7-wlnbt_a7df7b87-f170-426a-89b0-9cf26efdeb29/kube-rbac-proxy-metrics/0.log" Jul 08 20:09:07.506292 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:07.506254 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4cwpq/perf-node-gather-daemonset-xf4hf"] Jul 08 20:09:07.506770 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:07.506754 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c651106f-7cba-4f10-ad27-590996640cdb" containerName="maas-api" Jul 08 20:09:07.506841 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:07.506773 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c651106f-7cba-4f10-ad27-590996640cdb" containerName="maas-api" Jul 08 20:09:07.506841 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:07.506786 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76780d61-2bf1-48b2-b493-175b27c6e454" containerName="main" Jul 08 20:09:07.506841 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:07.506794 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="76780d61-2bf1-48b2-b493-175b27c6e454" containerName="main" Jul 08 20:09:07.506841 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:07.506819 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a84496f5-4fa6-42b5-ae34-4903466fc45f" containerName="maas-api" Jul 08 20:09:07.506841 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:07.506828 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a84496f5-4fa6-42b5-ae34-4903466fc45f" containerName="maas-api" Jul 08 20:09:07.507134 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:07.506939 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="76780d61-2bf1-48b2-b493-175b27c6e454" containerName="main" Jul 08 20:09:07.507134 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:07.506951 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c651106f-7cba-4f10-ad27-590996640cdb" containerName="maas-api" Jul 08 20:09:07.507134 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:07.506962 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a84496f5-4fa6-42b5-ae34-4903466fc45f" containerName="maas-api" Jul 08 20:09:07.509936 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:07.509915 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4cwpq/perf-node-gather-daemonset-xf4hf" Jul 08 20:09:07.512406 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:07.512387 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4cwpq\"/\"openshift-service-ca.crt\"" Jul 08 20:09:07.513345 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:07.513330 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-4cwpq\"/\"default-dockercfg-jzmnh\"" Jul 08 20:09:07.513442 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:07.513376 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4cwpq\"/\"kube-root-ca.crt\"" Jul 08 20:09:07.519969 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:07.519946 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4cwpq/perf-node-gather-daemonset-xf4hf"] Jul 08 20:09:07.663372 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:07.663345 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0da258fe-e6ff-43e9-b37d-5b735fe10fff-lib-modules\") pod \"perf-node-gather-daemonset-xf4hf\" (UID: \"0da258fe-e6ff-43e9-b37d-5b735fe10fff\") " pod="openshift-must-gather-4cwpq/perf-node-gather-daemonset-xf4hf" Jul 08 20:09:07.663550 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:07.663383 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0da258fe-e6ff-43e9-b37d-5b735fe10fff-sys\") pod \"perf-node-gather-daemonset-xf4hf\" (UID: \"0da258fe-e6ff-43e9-b37d-5b735fe10fff\") " pod="openshift-must-gather-4cwpq/perf-node-gather-daemonset-xf4hf" Jul 08 20:09:07.663550 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:07.663409 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0da258fe-e6ff-43e9-b37d-5b735fe10fff-proc\") pod \"perf-node-gather-daemonset-xf4hf\" (UID: \"0da258fe-e6ff-43e9-b37d-5b735fe10fff\") " pod="openshift-must-gather-4cwpq/perf-node-gather-daemonset-xf4hf" Jul 08 20:09:07.663550 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:07.663487 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0da258fe-e6ff-43e9-b37d-5b735fe10fff-podres\") pod \"perf-node-gather-daemonset-xf4hf\" (UID: \"0da258fe-e6ff-43e9-b37d-5b735fe10fff\") " pod="openshift-must-gather-4cwpq/perf-node-gather-daemonset-xf4hf" Jul 08 20:09:07.663671 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:07.663564 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5flwk\" (UniqueName: \"kubernetes.io/projected/0da258fe-e6ff-43e9-b37d-5b735fe10fff-kube-api-access-5flwk\") pod \"perf-node-gather-daemonset-xf4hf\" (UID: \"0da258fe-e6ff-43e9-b37d-5b735fe10fff\") " pod="openshift-must-gather-4cwpq/perf-node-gather-daemonset-xf4hf" Jul 08 20:09:07.764720 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:07.764616 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5flwk\" (UniqueName: \"kubernetes.io/projected/0da258fe-e6ff-43e9-b37d-5b735fe10fff-kube-api-access-5flwk\") pod \"perf-node-gather-daemonset-xf4hf\" (UID: \"0da258fe-e6ff-43e9-b37d-5b735fe10fff\") " pod="openshift-must-gather-4cwpq/perf-node-gather-daemonset-xf4hf" Jul 08 20:09:07.764720 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:07.764708 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0da258fe-e6ff-43e9-b37d-5b735fe10fff-lib-modules\") pod \"perf-node-gather-daemonset-xf4hf\" (UID: \"0da258fe-e6ff-43e9-b37d-5b735fe10fff\") " pod="openshift-must-gather-4cwpq/perf-node-gather-daemonset-xf4hf" Jul 08 20:09:07.764973 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:07.764741 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0da258fe-e6ff-43e9-b37d-5b735fe10fff-sys\") pod \"perf-node-gather-daemonset-xf4hf\" (UID: \"0da258fe-e6ff-43e9-b37d-5b735fe10fff\") " pod="openshift-must-gather-4cwpq/perf-node-gather-daemonset-xf4hf" Jul 08 20:09:07.764973 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:07.764765 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0da258fe-e6ff-43e9-b37d-5b735fe10fff-proc\") pod \"perf-node-gather-daemonset-xf4hf\" (UID: \"0da258fe-e6ff-43e9-b37d-5b735fe10fff\") " pod="openshift-must-gather-4cwpq/perf-node-gather-daemonset-xf4hf" Jul 08 20:09:07.764973 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:07.764810 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0da258fe-e6ff-43e9-b37d-5b735fe10fff-podres\") pod \"perf-node-gather-daemonset-xf4hf\" (UID: \"0da258fe-e6ff-43e9-b37d-5b735fe10fff\") " pod="openshift-must-gather-4cwpq/perf-node-gather-daemonset-xf4hf" Jul 08 20:09:07.764973 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:07.764837 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0da258fe-e6ff-43e9-b37d-5b735fe10fff-sys\") pod \"perf-node-gather-daemonset-xf4hf\" (UID: \"0da258fe-e6ff-43e9-b37d-5b735fe10fff\") " pod="openshift-must-gather-4cwpq/perf-node-gather-daemonset-xf4hf" Jul 08 20:09:07.764973 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:07.764885 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0da258fe-e6ff-43e9-b37d-5b735fe10fff-lib-modules\") pod \"perf-node-gather-daemonset-xf4hf\" (UID: \"0da258fe-e6ff-43e9-b37d-5b735fe10fff\") " pod="openshift-must-gather-4cwpq/perf-node-gather-daemonset-xf4hf" Jul 08 20:09:07.764973 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:07.764920 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0da258fe-e6ff-43e9-b37d-5b735fe10fff-proc\") pod \"perf-node-gather-daemonset-xf4hf\" (UID: \"0da258fe-e6ff-43e9-b37d-5b735fe10fff\") " pod="openshift-must-gather-4cwpq/perf-node-gather-daemonset-xf4hf" Jul 08 20:09:07.764973 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:07.764955 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0da258fe-e6ff-43e9-b37d-5b735fe10fff-podres\") pod \"perf-node-gather-daemonset-xf4hf\" (UID: \"0da258fe-e6ff-43e9-b37d-5b735fe10fff\") " pod="openshift-must-gather-4cwpq/perf-node-gather-daemonset-xf4hf" Jul 08 20:09:07.772236 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:07.772219 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5flwk\" (UniqueName: \"kubernetes.io/projected/0da258fe-e6ff-43e9-b37d-5b735fe10fff-kube-api-access-5flwk\") pod \"perf-node-gather-daemonset-xf4hf\" (UID: \"0da258fe-e6ff-43e9-b37d-5b735fe10fff\") " pod="openshift-must-gather-4cwpq/perf-node-gather-daemonset-xf4hf" Jul 08 20:09:07.820931 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:07.820909 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4cwpq/perf-node-gather-daemonset-xf4hf" Jul 08 20:09:07.949638 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:07.949610 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4cwpq/perf-node-gather-daemonset-xf4hf"] Jul 08 20:09:07.951783 ip-10-0-142-195 kubenswrapper[2566]: W0708 20:09:07.951756 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0da258fe_e6ff_43e9_b37d_5b735fe10fff.slice/crio-084681fb09451e26e13853ec2f1de04833c7e0ccf23ff2bf9d1dbf012f83966e WatchSource:0}: Error finding container 084681fb09451e26e13853ec2f1de04833c7e0ccf23ff2bf9d1dbf012f83966e: Status 404 returned error can't find the container with id 084681fb09451e26e13853ec2f1de04833c7e0ccf23ff2bf9d1dbf012f83966e Jul 08 20:09:08.269989 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:08.269957 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4cwpq/perf-node-gather-daemonset-xf4hf" event={"ID":"0da258fe-e6ff-43e9-b37d-5b735fe10fff","Type":"ContainerStarted","Data":"e399c23ba4702dbef30c632f0ddf703d6f9b9d69db02e93997ad27d83e016896"} Jul 08 20:09:08.270191 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:08.270017 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4cwpq/perf-node-gather-daemonset-xf4hf" event={"ID":"0da258fe-e6ff-43e9-b37d-5b735fe10fff","Type":"ContainerStarted","Data":"084681fb09451e26e13853ec2f1de04833c7e0ccf23ff2bf9d1dbf012f83966e"} Jul 08 20:09:08.270191 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:08.270083 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-4cwpq/perf-node-gather-daemonset-xf4hf" Jul 08 20:09:08.286420 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:08.286376 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-4cwpq/perf-node-gather-daemonset-xf4hf" podStartSLOduration=1.286363339 podStartE2EDuration="1.286363339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 20:09:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:09:08.285437729 +0000 UTC m=+2714.171627780" watchObservedRunningTime="2026-07-08 20:09:08.286363339 +0000 UTC m=+2714.172553379" Jul 08 20:09:09.578309 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:09.578266 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-npv6z_e2bd191d-a59d-4dbe-811f-44e2c4ca16e1/dns/0.log" Jul 08 20:09:09.599593 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:09.599570 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-npv6z_e2bd191d-a59d-4dbe-811f-44e2c4ca16e1/kube-rbac-proxy/0.log" Jul 08 20:09:09.620543 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:09.620517 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dghh8_474e5893-2b36-4b57-b154-15b0f8b4f300/dns-node-resolver/0.log" Jul 08 20:09:10.213568 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:10.213544 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6955c4f578-svncz_f6bf0fd3-2e15-4d5e-91cf-418a6255cf75/registry/0.log" Jul 08 20:09:10.236473 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:10.236442 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6w52p_8b78ed73-731b-49fa-9220-4b1e45787806/node-ca/0.log" Jul 08 20:09:11.432084 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:11.432046 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-j57db_13b2effd-726c-4425-aff2-45d8df9443ef/discovery/0.log" Jul 08 20:09:11.695845 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:11.695763 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5846dc5c45-fgr6x_cee815ab-2b57-4584-8b6c-a8748c849be9/router/0.log" Jul 08 20:09:12.388653 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:12.388622 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5nfnp_d36b2cab-e4d2-4a23-85f9-0bd24ccbb497/serve-healthcheck-canary/0.log" Jul 08 20:09:12.900927 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:12.900889 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-gfkmh_6c0d602f-7a0a-415c-9716-54ffb6c51e67/insights-operator/0.log" Jul 08 20:09:12.901950 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:12.901918 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-gfkmh_6c0d602f-7a0a-415c-9716-54ffb6c51e67/insights-operator/1.log" Jul 08 20:09:13.056052 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:13.056022 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-db7nv_baee8938-f6f6-4ed9-99f7-832c16ebbd4a/kube-rbac-proxy/0.log" Jul 08 20:09:13.077042 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:13.077021 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-db7nv_baee8938-f6f6-4ed9-99f7-832c16ebbd4a/exporter/0.log" Jul 08 20:09:13.098700 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:13.098663 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-db7nv_baee8938-f6f6-4ed9-99f7-832c16ebbd4a/extractor/0.log" Jul 08 20:09:14.283922 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:14.283896 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-4cwpq/perf-node-gather-daemonset-xf4hf" Jul 08 20:09:15.626884 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:15.626832 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-5kkcw_8080820c-03c1-4be8-8b2c-f8bbf00d5182/server/0.log" Jul 08 20:09:15.884640 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:15.884546 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-77ddc6fd87-m2dgt_926adae9-5aea-4f33-8467-166b43bb6a0d/manager/0.log" Jul 08 20:09:15.907966 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:15.907938 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-77ddc6fd87-m2dgt_926adae9-5aea-4f33-8467-166b43bb6a0d/copy-manifests/0.log" Jul 08 20:09:17.272530 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:17.272485 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6bb9464976-444br_8251066c-61ee-45a0-80b5-3ed028425362/manager/0.log" Jul 08 20:09:17.933995 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:17.933965 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-j57db_13b2effd-726c-4425-aff2-45d8df9443ef/discovery/0.log" Jul 08 20:09:18.166001 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:18.165973 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5846dc5c45-fgr6x_cee815ab-2b57-4584-8b6c-a8748c849be9/router/0.log" Jul 08 20:09:19.528430 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:19.528393 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-j57db_13b2effd-726c-4425-aff2-45d8df9443ef/discovery/0.log" Jul 08 20:09:19.732988 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:19.732951 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5846dc5c45-fgr6x_cee815ab-2b57-4584-8b6c-a8748c849be9/router/0.log" Jul 08 20:09:24.280074 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:24.280046 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-srmfq_dbe0d11c-c9c5-4cdb-ad9b-08b142e7335f/kube-multus-additional-cni-plugins/0.log" Jul 08 20:09:24.301738 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:24.301715 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-srmfq_dbe0d11c-c9c5-4cdb-ad9b-08b142e7335f/egress-router-binary-copy/0.log" Jul 08 20:09:24.322063 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:24.322037 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-srmfq_dbe0d11c-c9c5-4cdb-ad9b-08b142e7335f/cni-plugins/0.log" Jul 08 20:09:24.341253 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:24.341227 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-srmfq_dbe0d11c-c9c5-4cdb-ad9b-08b142e7335f/bond-cni-plugin/0.log" Jul 08 20:09:24.362848 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:24.362823 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-srmfq_dbe0d11c-c9c5-4cdb-ad9b-08b142e7335f/routeoverride-cni/0.log" Jul 08 20:09:24.382336 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:24.382313 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-srmfq_dbe0d11c-c9c5-4cdb-ad9b-08b142e7335f/whereabouts-cni-bincopy/0.log" Jul 08 20:09:24.402630 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:24.402610 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-srmfq_dbe0d11c-c9c5-4cdb-ad9b-08b142e7335f/whereabouts-cni/0.log" Jul 08 20:09:24.595099 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:24.595023 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-gz54s_3a7da2be-d237-4bfa-887d-0be7632bfdc0/kube-multus/0.log" Jul 08 20:09:24.740057 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:24.740033 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-rfbjf_55818509-538d-4733-9e10-eaaf841ef071/network-metrics-daemon/0.log" Jul 08 20:09:24.759171 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:24.759148 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-rfbjf_55818509-538d-4733-9e10-eaaf841ef071/kube-rbac-proxy/0.log" Jul 08 20:09:25.624856 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:25.624825 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cdp8n_3af16262-f54e-4683-b76c-e1ccfdff4c3c/ovn-controller/0.log" Jul 08 20:09:25.642176 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:25.642142 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cdp8n_3af16262-f54e-4683-b76c-e1ccfdff4c3c/ovn-acl-logging/0.log" Jul 08 20:09:25.680210 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:25.680183 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cdp8n_3af16262-f54e-4683-b76c-e1ccfdff4c3c/kube-rbac-proxy-node/0.log" Jul 08 20:09:25.705167 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:25.705136 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cdp8n_3af16262-f54e-4683-b76c-e1ccfdff4c3c/kube-rbac-proxy-ovn-metrics/0.log" Jul 08 20:09:25.721514 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:25.721485 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cdp8n_3af16262-f54e-4683-b76c-e1ccfdff4c3c/northd/0.log" Jul 08 20:09:25.744967 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:25.744934 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cdp8n_3af16262-f54e-4683-b76c-e1ccfdff4c3c/nbdb/0.log" Jul 08 20:09:25.769992 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:25.769964 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cdp8n_3af16262-f54e-4683-b76c-e1ccfdff4c3c/sbdb/0.log" Jul 08 20:09:25.850647 ip-10-0-142-195 kubenswrapper[2566]: I0708 20:09:25.850615 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cdp8n_3af16262-f54e-4683-b76c-e1ccfdff4c3c/ovnkube-controller/0.log"