Jul 22 02:58:09.115356 ip-10-0-131-67 systemd[1]: Starting Kubernetes Kubelet... Jul 22 02:58:09.630043 ip-10-0-131-67 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 02:58:09.630043 ip-10-0-131-67 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 22 02:58:09.630043 ip-10-0-131-67 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 02:58:09.630043 ip-10-0-131-67 kubenswrapper[2582]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 22 02:58:09.630043 ip-10-0-131-67 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 02:58:09.630043 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.590405 2582 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 22 02:58:09.630043 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595152 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 02:58:09.630043 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595164 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 02:58:09.630043 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595168 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 02:58:09.630043 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595171 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 02:58:09.630043 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595174 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 02:58:09.630043 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595177 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 02:58:09.630043 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595180 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 02:58:09.630043 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595184 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 02:58:09.631282 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595188 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 02:58:09.631282 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595191 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 02:58:09.631282 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595194 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 02:58:09.631282 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595197 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 02:58:09.631282 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595201 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 02:58:09.631282 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595212 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 02:58:09.631282 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595216 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 02:58:09.631282 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595219 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 22 02:58:09.631282 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595223 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 02:58:09.631282 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595226 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 02:58:09.631282 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595229 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 02:58:09.631282 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595231 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 02:58:09.631282 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595238 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 02:58:09.631282 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595242 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 02:58:09.631282 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595245 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 02:58:09.631282 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595247 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 02:58:09.631282 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595251 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 02:58:09.631282 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595254 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 02:58:09.631282 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595257 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 02:58:09.631282 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595260 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 02:58:09.632271 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595264 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 02:58:09.632271 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595269 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 02:58:09.632271 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595272 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 02:58:09.632271 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595275 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 02:58:09.632271 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595279 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 02:58:09.632271 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595282 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 02:58:09.632271 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595285 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 02:58:09.632271 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595288 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 02:58:09.632271 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595291 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 02:58:09.632271 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595294 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 02:58:09.632271 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595297 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 02:58:09.632271 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595300 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 02:58:09.632271 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595303 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 02:58:09.632271 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595306 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 02:58:09.632271 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595309 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 02:58:09.632271 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595312 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 02:58:09.632271 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595314 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 02:58:09.632271 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595317 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 02:58:09.632271 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595320 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 02:58:09.632843 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595323 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 02:58:09.632843 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595326 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 02:58:09.632843 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595329 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 02:58:09.632843 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595332 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 02:58:09.632843 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595334 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 02:58:09.632843 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595337 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 02:58:09.632843 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595341 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 02:58:09.632843 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595345 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 02:58:09.632843 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595350 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 02:58:09.632843 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595353 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 02:58:09.632843 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595356 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 02:58:09.632843 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595360 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 02:58:09.632843 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595364 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 02:58:09.632843 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595367 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 02:58:09.632843 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595371 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 02:58:09.632843 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595374 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 02:58:09.632843 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595377 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 02:58:09.632843 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595380 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 02:58:09.632843 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595383 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 02:58:09.633488 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595386 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 02:58:09.633488 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595389 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 02:58:09.633488 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595392 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 02:58:09.633488 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595394 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 02:58:09.633488 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595397 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 02:58:09.633488 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595400 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 02:58:09.633488 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595403 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 02:58:09.633488 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595406 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 02:58:09.633488 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595409 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 02:58:09.633488 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595411 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 02:58:09.633488 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595414 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 02:58:09.633488 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595417 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 02:58:09.633488 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595420 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 02:58:09.633488 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595423 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 02:58:09.633488 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595426 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 02:58:09.633488 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595429 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 02:58:09.633488 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595431 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 02:58:09.633488 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595434 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 02:58:09.633488 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595437 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 02:58:09.633488 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595439 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 02:58:09.634141 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595442 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 02:58:09.634141 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595446 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 02:58:09.634141 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595448 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 02:58:09.634141 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595452 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 02:58:09.634141 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595454 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 02:58:09.634141 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595457 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 02:58:09.634141 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595460 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 02:58:09.634141 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595463 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 02:58:09.634141 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595466 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 02:58:09.634141 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595469 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 02:58:09.634141 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595472 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 02:58:09.634141 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595475 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 02:58:09.634141 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.595477 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 02:58:09.634141 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596245 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 02:58:09.634141 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596253 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 02:58:09.634141 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596256 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 02:58:09.634141 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596259 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 02:58:09.634141 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596262 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 02:58:09.634141 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596266 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 02:58:09.634141 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596269 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 02:58:09.634727 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596272 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 02:58:09.634727 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596274 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 02:58:09.634727 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596278 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 02:58:09.634727 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596281 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 02:58:09.634727 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596284 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 02:58:09.634727 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596288 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 02:58:09.634727 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596291 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 02:58:09.634727 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596294 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 02:58:09.634727 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596297 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 02:58:09.634727 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596300 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 02:58:09.634727 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596302 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 02:58:09.634727 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596305 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 02:58:09.634727 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596308 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 02:58:09.634727 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596311 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 22 02:58:09.634727 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596316 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 02:58:09.634727 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596320 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 02:58:09.634727 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596324 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 02:58:09.634727 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596327 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 02:58:09.634727 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596330 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 02:58:09.634727 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596333 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 02:58:09.635600 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596336 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 02:58:09.635600 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596339 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 02:58:09.635600 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596341 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 02:58:09.635600 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596345 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 02:58:09.635600 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596348 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 02:58:09.635600 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596350 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 02:58:09.635600 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596353 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 02:58:09.635600 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596356 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 02:58:09.635600 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596358 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 02:58:09.635600 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596361 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 02:58:09.635600 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596364 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 02:58:09.635600 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596367 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 02:58:09.635600 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596370 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 02:58:09.635600 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596373 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 02:58:09.635600 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596376 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 02:58:09.635600 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596379 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 02:58:09.635600 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596382 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 02:58:09.635600 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596385 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 02:58:09.635600 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596388 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 02:58:09.635600 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596390 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 02:58:09.636189 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596393 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 02:58:09.636189 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596397 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 02:58:09.636189 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596400 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 02:58:09.636189 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596402 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 02:58:09.636189 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596405 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 02:58:09.636189 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596408 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 02:58:09.636189 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596410 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 02:58:09.636189 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596413 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 02:58:09.636189 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596416 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 02:58:09.636189 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596419 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 02:58:09.636189 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596423 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 02:58:09.636189 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596427 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 02:58:09.636189 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596430 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 02:58:09.636189 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596433 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 02:58:09.636189 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596436 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 02:58:09.636189 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596439 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 02:58:09.636189 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596443 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 02:58:09.636189 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596446 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 02:58:09.636189 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596449 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 02:58:09.636734 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596451 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 02:58:09.636734 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596454 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 02:58:09.636734 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596457 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 02:58:09.636734 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596460 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 02:58:09.636734 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596463 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 02:58:09.636734 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596465 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 02:58:09.636734 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596468 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 02:58:09.636734 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596471 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 02:58:09.636734 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596473 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 02:58:09.636734 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596477 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 02:58:09.636734 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596480 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 02:58:09.636734 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596483 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 02:58:09.636734 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596486 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 02:58:09.636734 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596489 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 02:58:09.636734 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596492 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 02:58:09.636734 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596496 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 02:58:09.636734 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596500 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 02:58:09.636734 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596503 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 02:58:09.636734 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596506 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 02:58:09.637286 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596510 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 02:58:09.637286 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596513 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 02:58:09.637286 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596515 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 02:58:09.637286 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596518 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 02:58:09.637286 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596521 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 02:58:09.637286 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596524 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 02:58:09.637286 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596527 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 02:58:09.637286 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596529 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 02:58:09.637286 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596532 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 02:58:09.637286 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596535 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 02:58:09.637286 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596538 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 02:58:09.637286 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596541 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 02:58:09.637286 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596543 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 02:58:09.637286 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596546 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 02:58:09.637286 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596968 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 02:58:09.637286 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596973 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 02:58:09.637286 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596977 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 02:58:09.637286 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596980 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 02:58:09.637286 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596986 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 02:58:09.637286 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596989 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 02:58:09.637864 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596992 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 02:58:09.637864 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.596996 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 02:58:09.637864 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597000 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 02:58:09.637864 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597003 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 02:58:09.637864 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597006 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 02:58:09.637864 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597008 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 02:58:09.637864 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597011 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 02:58:09.637864 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597015 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 02:58:09.637864 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597019 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 02:58:09.637864 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597022 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 02:58:09.637864 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597026 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 02:58:09.637864 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597029 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 02:58:09.637864 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597032 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 02:58:09.637864 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597034 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 02:58:09.637864 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597037 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 02:58:09.637864 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597040 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 02:58:09.637864 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597043 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 02:58:09.637864 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597046 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 02:58:09.637864 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597048 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 02:58:09.637864 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597051 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 02:58:09.638513 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597054 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 02:58:09.638513 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597057 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 02:58:09.638513 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597059 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 02:58:09.638513 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597063 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 02:58:09.638513 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597065 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 02:58:09.638513 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597068 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 02:58:09.638513 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597071 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 02:58:09.638513 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597073 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 02:58:09.638513 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597076 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 02:58:09.638513 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597078 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 02:58:09.638513 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597083 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 02:58:09.638513 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597101 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 02:58:09.638513 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597104 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 02:58:09.638513 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597109 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 02:58:09.638513 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597112 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 02:58:09.638513 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597115 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 02:58:09.638513 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597118 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 02:58:09.638513 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597121 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 02:58:09.638513 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597124 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 02:58:09.638513 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597127 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 02:58:09.639136 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597130 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 02:58:09.639136 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597132 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 02:58:09.639136 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597135 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 02:58:09.639136 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597138 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 02:58:09.639136 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597140 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 02:58:09.639136 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597143 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 02:58:09.639136 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597145 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 02:58:09.639136 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597148 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 02:58:09.639136 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597151 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 02:58:09.639136 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597154 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 02:58:09.639136 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597157 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 02:58:09.639136 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597160 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 02:58:09.639136 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597163 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 02:58:09.639136 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597166 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 02:58:09.639136 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597168 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 02:58:09.639136 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597172 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 02:58:09.639136 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597175 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 02:58:09.639136 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597178 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 02:58:09.639136 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597181 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 02:58:09.639136 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597183 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 02:58:09.639722 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597186 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 02:58:09.639722 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597189 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 02:58:09.639722 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597193 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 22 02:58:09.639722 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597195 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 02:58:09.639722 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597198 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 02:58:09.639722 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597202 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 02:58:09.639722 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597205 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 02:58:09.639722 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597208 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 02:58:09.639722 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597211 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 02:58:09.639722 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597214 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 02:58:09.639722 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597216 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 02:58:09.639722 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597220 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 02:58:09.639722 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597224 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 02:58:09.639722 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597227 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 02:58:09.639722 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597230 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 02:58:09.639722 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597233 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 02:58:09.639722 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597238 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 02:58:09.639722 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597241 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 02:58:09.639722 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597245 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 02:58:09.640335 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597248 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 02:58:09.640335 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597251 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 02:58:09.640335 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597254 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 02:58:09.640335 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597257 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 02:58:09.640335 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597259 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 02:58:09.640335 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597262 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 02:58:09.640335 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597265 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 02:58:09.640335 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597268 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 02:58:09.640335 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597271 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 02:58:09.640335 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597273 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 02:58:09.640335 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597276 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 02:58:09.640335 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597279 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 02:58:09.640335 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597282 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 02:58:09.640335 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.597285 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 02:58:09.640335 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597357 2582 flags.go:64] FLAG: --address="0.0.0.0" Jul 22 02:58:09.640335 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597373 2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 22 02:58:09.640335 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597380 2582 flags.go:64] FLAG: --anonymous-auth="true" Jul 22 02:58:09.640335 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597385 2582 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 22 02:58:09.640335 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597391 2582 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 22 02:58:09.640335 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597395 2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 22 02:58:09.640335 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597399 2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 22 02:58:09.640925 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597407 2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 22 02:58:09.640925 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597411 2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 22 02:58:09.640925 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597414 2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 22 02:58:09.640925 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597419 2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 22 02:58:09.640925 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597422 2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 22 02:58:09.640925 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597426 2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 22 02:58:09.640925 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597429 2582 flags.go:64] FLAG: --cgroup-root="" Jul 22 02:58:09.640925 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597432 2582 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 22 02:58:09.640925 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597435 2582 flags.go:64] FLAG: --client-ca-file="" Jul 22 02:58:09.640925 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597438 2582 flags.go:64] FLAG: --cloud-provider="external" Jul 22 02:58:09.640925 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597441 2582 flags.go:64] FLAG: --cluster-dns="[]" Jul 22 02:58:09.640925 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597447 2582 flags.go:64] FLAG: --cluster-domain="" Jul 22 02:58:09.640925 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597450 2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 22 02:58:09.640925 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597454 2582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 22 02:58:09.640925 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597457 2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 22 02:58:09.640925 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597461 2582 flags.go:64] FLAG: --container-log-max-files="5" Jul 22 02:58:09.640925 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597465 2582 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 22 02:58:09.640925 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597468 2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 22 02:58:09.640925 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597472 2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 22 02:58:09.640925 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597475 2582 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 22 02:58:09.640925 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597479 2582 flags.go:64] FLAG: --contention-profiling="false" Jul 22 02:58:09.640925 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597482 2582 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 22 02:58:09.640925 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597485 2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 22 02:58:09.640925 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597488 2582 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 22 02:58:09.641570 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597491 2582 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 22 02:58:09.641570 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597496 2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 22 02:58:09.641570 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597500 2582 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 22 02:58:09.641570 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597504 2582 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 22 02:58:09.641570 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597507 2582 flags.go:64] FLAG: --enable-load-reader="false" Jul 22 02:58:09.641570 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597512 2582 flags.go:64] FLAG: --enable-server="true" Jul 22 02:58:09.641570 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597515 2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 22 02:58:09.641570 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597521 2582 flags.go:64] FLAG: --event-burst="100" Jul 22 02:58:09.641570 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597524 2582 flags.go:64] FLAG: --event-qps="50" Jul 22 02:58:09.641570 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597527 2582 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 22 02:58:09.641570 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597531 2582 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 22 02:58:09.641570 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597534 2582 flags.go:64] FLAG: --eviction-hard="" Jul 22 02:58:09.641570 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597539 2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 22 02:58:09.641570 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597542 2582 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 22 02:58:09.641570 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597545 2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 22 02:58:09.641570 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597548 2582 flags.go:64] FLAG: --eviction-soft="" Jul 22 02:58:09.641570 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597551 2582 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 22 02:58:09.641570 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597554 2582 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 22 02:58:09.641570 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597557 2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 22 02:58:09.641570 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597561 2582 flags.go:64] FLAG: --experimental-mounter-path="" Jul 22 02:58:09.641570 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597564 2582 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 22 02:58:09.641570 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597567 2582 flags.go:64] FLAG: --fail-swap-on="true" Jul 22 02:58:09.641570 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597570 2582 flags.go:64] FLAG: --feature-gates="" Jul 22 02:58:09.641570 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597574 2582 flags.go:64] FLAG: --file-check-frequency="20s" Jul 22 02:58:09.641570 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597577 2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 22 02:58:09.642404 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597581 2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 22 02:58:09.642404 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597584 2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 22 02:58:09.642404 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597587 2582 flags.go:64] FLAG: --healthz-port="10248" Jul 22 02:58:09.642404 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597590 2582 flags.go:64] FLAG: --help="false" Jul 22 02:58:09.642404 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597593 2582 flags.go:64] FLAG: --hostname-override="ip-10-0-131-67.ec2.internal" Jul 22 02:58:09.642404 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597597 2582 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 22 02:58:09.642404 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597600 2582 flags.go:64] FLAG: --http-check-frequency="20s" Jul 22 02:58:09.642404 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597603 2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 22 02:58:09.642404 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597607 2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 22 02:58:09.642404 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597612 2582 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 22 02:58:09.642404 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597615 2582 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 22 02:58:09.642404 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597618 2582 flags.go:64] FLAG: --image-service-endpoint="" Jul 22 02:58:09.642404 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597622 2582 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 22 02:58:09.642404 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597626 2582 flags.go:64] FLAG: --kube-api-burst="100" Jul 22 02:58:09.642404 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597629 2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 22 02:58:09.642404 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597632 2582 flags.go:64] FLAG: --kube-api-qps="50" Jul 22 02:58:09.642404 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597635 2582 flags.go:64] FLAG: --kube-reserved="" Jul 22 02:58:09.642404 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597639 2582 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 22 02:58:09.642404 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597643 2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 22 02:58:09.642404 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597646 2582 flags.go:64] FLAG: --kubelet-cgroups="" Jul 22 02:58:09.642404 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597649 2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 22 02:58:09.642404 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597652 2582 flags.go:64] FLAG: --lock-file="" Jul 22 02:58:09.642404 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597655 2582 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 22 02:58:09.642404 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597659 2582 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 22 02:58:09.643025 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597662 2582 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 22 02:58:09.643025 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597668 2582 flags.go:64] FLAG: --log-json-split-stream="false" Jul 22 02:58:09.643025 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597671 2582 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 22 02:58:09.643025 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597674 2582 flags.go:64] FLAG: --log-text-split-stream="false" Jul 22 02:58:09.643025 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597677 2582 flags.go:64] FLAG: --logging-format="text" Jul 22 02:58:09.643025 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597680 2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 22 02:58:09.643025 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597684 2582 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 22 02:58:09.643025 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597687 2582 flags.go:64] FLAG: --manifest-url="" Jul 22 02:58:09.643025 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597690 2582 flags.go:64] FLAG: --manifest-url-header="" Jul 22 02:58:09.643025 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597695 2582 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 22 02:58:09.643025 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597698 2582 flags.go:64] FLAG: --max-open-files="1000000" Jul 22 02:58:09.643025 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597703 2582 flags.go:64] FLAG: --max-pods="110" Jul 22 02:58:09.643025 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597706 2582 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 22 02:58:09.643025 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597709 2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 22 02:58:09.643025 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597713 2582 flags.go:64] FLAG: --memory-manager-policy="None" Jul 22 02:58:09.643025 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597716 2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 22 02:58:09.643025 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597719 2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 22 02:58:09.643025 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597724 2582 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 22 02:58:09.643025 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597727 2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 22 02:58:09.643025 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597735 2582 flags.go:64] FLAG: --node-status-max-images="50" Jul 22 02:58:09.643025 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597740 2582 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 22 02:58:09.643025 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597743 2582 flags.go:64] FLAG: --oom-score-adj="-999" Jul 22 02:58:09.643025 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597746 2582 flags.go:64] FLAG: --pod-cidr="" Jul 22 02:58:09.643637 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597749 2582 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 22 02:58:09.643637 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597756 2582 flags.go:64] FLAG: --pod-manifest-path="" Jul 22 02:58:09.643637 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597759 2582 flags.go:64] FLAG: --pod-max-pids="-1" Jul 22 02:58:09.643637 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597762 2582 flags.go:64] FLAG: --pods-per-core="0" Jul 22 02:58:09.643637 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597766 2582 flags.go:64] FLAG: --port="10250" Jul 22 02:58:09.643637 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597769 2582 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 22 02:58:09.643637 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597772 2582 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-03f481aa8f079b952" Jul 22 02:58:09.643637 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597776 2582 flags.go:64] FLAG: --qos-reserved="" Jul 22 02:58:09.643637 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597779 2582 flags.go:64] FLAG: --read-only-port="10255" Jul 22 02:58:09.643637 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597782 2582 flags.go:64] FLAG: --register-node="true" Jul 22 02:58:09.643637 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597786 2582 flags.go:64] FLAG: --register-with-taints="" Jul 22 02:58:09.643637 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597789 2582 flags.go:64] FLAG: --registry-burst="10" Jul 22 02:58:09.643637 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597792 2582 flags.go:64] FLAG: --registry-qps="5" Jul 22 02:58:09.643637 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597795 2582 flags.go:64] FLAG: --reserved-cpus="" Jul 22 02:58:09.643637 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597798 2582 flags.go:64] FLAG: --reserved-memory="" Jul 22 02:58:09.643637 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597803 2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 22 02:58:09.643637 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597806 2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 22 02:58:09.643637 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597809 2582 flags.go:64] FLAG: --rotate-certificates="false" Jul 22 02:58:09.643637 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597812 2582 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 22 02:58:09.643637 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597815 2582 flags.go:64] FLAG: --runonce="false" Jul 22 02:58:09.643637 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597818 2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 22 02:58:09.643637 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597821 2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 22 02:58:09.643637 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597825 2582 flags.go:64] FLAG: --seccomp-default="false" Jul 22 02:58:09.643637 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597828 2582 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 22 02:58:09.643637 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597831 2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 22 02:58:09.644342 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597834 2582 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 22 02:58:09.644342 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597837 2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 22 02:58:09.644342 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597842 2582 flags.go:64] FLAG: --storage-driver-password="root" Jul 22 02:58:09.644342 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597846 2582 flags.go:64] FLAG: --storage-driver-secure="false" Jul 22 02:58:09.644342 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597850 2582 flags.go:64] FLAG: --storage-driver-table="stats" Jul 22 02:58:09.644342 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597853 2582 flags.go:64] FLAG: --storage-driver-user="root" Jul 22 02:58:09.644342 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597856 2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 22 02:58:09.644342 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597859 2582 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 22 02:58:09.644342 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597862 2582 flags.go:64] FLAG: --system-cgroups="" Jul 22 02:58:09.644342 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597866 2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 22 02:58:09.644342 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597871 2582 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 22 02:58:09.644342 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597874 2582 flags.go:64] FLAG: --tls-cert-file="" Jul 22 02:58:09.644342 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597878 2582 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 22 02:58:09.644342 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597883 2582 flags.go:64] FLAG: --tls-min-version="" Jul 22 02:58:09.644342 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597886 2582 flags.go:64] FLAG: --tls-private-key-file="" Jul 22 02:58:09.644342 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597889 2582 flags.go:64] FLAG: --topology-manager-policy="none" Jul 22 02:58:09.644342 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597892 2582 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 22 02:58:09.644342 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597895 2582 flags.go:64] FLAG: --topology-manager-scope="container" Jul 22 02:58:09.644342 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597898 2582 flags.go:64] FLAG: --v="2" Jul 22 02:58:09.644342 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597902 2582 flags.go:64] FLAG: --version="false" Jul 22 02:58:09.644342 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597906 2582 flags.go:64] FLAG: --vmodule="" Jul 22 02:58:09.644342 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597910 2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 22 02:58:09.644342 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.597913 2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 22 02:58:09.644342 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598033 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 02:58:09.644342 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598036 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 02:58:09.644997 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598040 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 02:58:09.644997 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598043 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 02:58:09.644997 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598046 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 02:58:09.644997 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598049 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 02:58:09.644997 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598052 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 02:58:09.644997 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598055 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 02:58:09.644997 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598058 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 02:58:09.644997 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598061 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 02:58:09.644997 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598064 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 02:58:09.644997 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598068 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 02:58:09.644997 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598071 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 02:58:09.644997 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598079 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 02:58:09.644997 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598082 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 02:58:09.644997 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598099 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 02:58:09.644997 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598103 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 02:58:09.644997 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598106 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 02:58:09.644997 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598109 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 02:58:09.644997 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598112 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 02:58:09.644997 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598115 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 02:58:09.645509 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598118 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 02:58:09.645509 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598124 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 02:58:09.645509 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598128 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 02:58:09.645509 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598132 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 02:58:09.645509 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598135 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 02:58:09.645509 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598138 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 02:58:09.645509 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598141 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 02:58:09.645509 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598145 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 02:58:09.645509 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598148 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 02:58:09.645509 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598151 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 02:58:09.645509 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598154 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 02:58:09.645509 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598157 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 02:58:09.645509 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598162 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 02:58:09.645509 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598166 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 02:58:09.645509 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598169 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 02:58:09.645509 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598172 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 02:58:09.645509 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598175 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 02:58:09.645509 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598178 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 22 02:58:09.645509 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598181 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 02:58:09.646006 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598184 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 02:58:09.646006 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598186 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 02:58:09.646006 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598189 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 02:58:09.646006 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598194 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 02:58:09.646006 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598197 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 02:58:09.646006 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598201 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 02:58:09.646006 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598204 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 02:58:09.646006 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598207 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 02:58:09.646006 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598210 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 02:58:09.646006 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598213 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 02:58:09.646006 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598216 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 02:58:09.646006 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598219 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 02:58:09.646006 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598222 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 02:58:09.646006 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598225 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 02:58:09.646006 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598228 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 02:58:09.646006 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598231 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 02:58:09.646006 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598234 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 02:58:09.646006 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598236 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 02:58:09.646006 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598239 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 02:58:09.646546 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598242 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 02:58:09.646546 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598245 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 02:58:09.646546 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598247 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 02:58:09.646546 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598250 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 02:58:09.646546 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598253 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 02:58:09.646546 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598256 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 02:58:09.646546 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598259 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 02:58:09.646546 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598261 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 02:58:09.646546 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598264 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 02:58:09.646546 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598269 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 02:58:09.646546 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598272 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 02:58:09.646546 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598275 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 02:58:09.646546 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598278 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 02:58:09.646546 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598282 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 02:58:09.646546 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598285 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 02:58:09.646546 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598288 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 02:58:09.646546 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598293 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 02:58:09.646546 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598296 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 02:58:09.646546 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598301 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 02:58:09.646546 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598304 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 02:58:09.647071 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598306 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 02:58:09.647071 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598309 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 02:58:09.647071 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598312 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 02:58:09.647071 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598315 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 02:58:09.647071 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598317 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 02:58:09.647071 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598320 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 02:58:09.647071 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598323 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 02:58:09.647071 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598326 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 02:58:09.647071 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598328 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 02:58:09.647071 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598332 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 02:58:09.647071 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598335 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 02:58:09.647071 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598338 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 02:58:09.647071 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598340 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 02:58:09.647071 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598343 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 02:58:09.647071 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598345 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 02:58:09.647071 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598358 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 02:58:09.647071 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598362 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 02:58:09.647071 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598364 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 02:58:09.647071 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598367 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 02:58:09.647071 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.598369 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 02:58:09.647608 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.598379 2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 02:58:09.647608 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.606187 2582 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 22 02:58:09.647608 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.606203 2582 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 22 02:58:09.647608 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.606217 2582 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 22 02:58:09.647608 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.606223 2582 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 22 02:58:09.647608 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606308 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 02:58:09.647608 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606313 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 02:58:09.647608 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606317 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 02:58:09.647608 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606322 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 02:58:09.647608 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606328 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 02:58:09.647608 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606331 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 02:58:09.647608 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606335 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 02:58:09.647608 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606338 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 02:58:09.647608 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606342 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 02:58:09.647981 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606347 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 02:58:09.647981 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606350 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 02:58:09.647981 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606353 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 02:58:09.647981 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606356 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 02:58:09.647981 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606358 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 02:58:09.647981 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606361 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 02:58:09.647981 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606364 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 02:58:09.647981 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606373 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 02:58:09.647981 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606376 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 02:58:09.647981 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606379 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 02:58:09.647981 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606382 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 02:58:09.647981 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606385 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 02:58:09.647981 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606387 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 02:58:09.647981 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606390 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 02:58:09.647981 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606393 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 02:58:09.647981 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606396 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 02:58:09.647981 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606398 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 02:58:09.647981 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606401 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 02:58:09.647981 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606404 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 02:58:09.648480 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606407 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 02:58:09.648480 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606411 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 02:58:09.648480 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606414 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 02:58:09.648480 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606417 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 02:58:09.648480 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606420 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 02:58:09.648480 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606422 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 02:58:09.648480 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606425 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 02:58:09.648480 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606428 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 02:58:09.648480 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606431 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 02:58:09.648480 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606434 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 02:58:09.648480 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606436 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 02:58:09.648480 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606439 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 02:58:09.648480 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606442 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 02:58:09.648480 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606445 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 02:58:09.648480 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606447 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 02:58:09.648480 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606450 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 22 02:58:09.648480 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606454 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 02:58:09.648480 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606458 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 02:58:09.648480 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606465 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 02:58:09.648480 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606474 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 02:58:09.649006 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606478 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 02:58:09.649006 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606487 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 02:58:09.649006 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606490 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 02:58:09.649006 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606493 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 02:58:09.649006 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606496 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 02:58:09.649006 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606499 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 02:58:09.649006 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606502 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 02:58:09.649006 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606504 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 02:58:09.649006 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606507 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 02:58:09.649006 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606510 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 02:58:09.649006 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606513 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 02:58:09.649006 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606515 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 02:58:09.649006 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606518 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 02:58:09.649006 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606522 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 02:58:09.649006 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606525 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 02:58:09.649006 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606527 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 02:58:09.649006 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606531 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 02:58:09.649006 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606533 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 02:58:09.649006 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606536 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 02:58:09.649006 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606539 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 02:58:09.649545 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606542 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 02:58:09.649545 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606545 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 02:58:09.649545 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606547 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 02:58:09.649545 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606550 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 02:58:09.649545 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606552 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 02:58:09.649545 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606555 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 02:58:09.649545 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606558 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 02:58:09.649545 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606560 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 02:58:09.649545 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606563 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 02:58:09.649545 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606566 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 02:58:09.649545 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606568 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 02:58:09.649545 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606571 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 02:58:09.649545 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606574 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 02:58:09.649545 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606577 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 02:58:09.649545 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606587 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 02:58:09.649545 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606590 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 02:58:09.649545 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606592 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 02:58:09.649545 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606595 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 02:58:09.649545 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606597 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 02:58:09.649545 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606600 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 02:58:09.650109 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606603 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 02:58:09.650109 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606606 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 02:58:09.650109 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606608 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 02:58:09.650109 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606611 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 02:58:09.650109 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606613 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 02:58:09.650109 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606619 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 02:58:09.650109 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606622 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 02:58:09.650109 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606624 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 02:58:09.650109 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606627 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 02:58:09.650109 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606630 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 02:58:09.650109 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606633 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 02:58:09.650109 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.606638 2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 02:58:09.650109 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606817 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 02:58:09.650109 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606823 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 02:58:09.650109 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606826 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 02:58:09.650503 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606829 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 02:58:09.650503 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606832 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 02:58:09.650503 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606835 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 02:58:09.650503 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606838 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 02:58:09.650503 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606841 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 02:58:09.650503 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606843 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 02:58:09.650503 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606846 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 02:58:09.650503 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606848 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 02:58:09.650503 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606851 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 02:58:09.650503 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606854 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 02:58:09.650503 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606857 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 02:58:09.650503 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606859 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 02:58:09.650503 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606868 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 02:58:09.650503 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606871 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 02:58:09.650503 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606874 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 02:58:09.650503 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606876 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 02:58:09.650503 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606879 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 02:58:09.650503 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606882 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 02:58:09.650503 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606884 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 02:58:09.650503 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606887 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 02:58:09.650503 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606890 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 02:58:09.651040 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606893 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 02:58:09.651040 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606896 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 02:58:09.651040 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606903 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 02:58:09.651040 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606906 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 02:58:09.651040 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606909 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 02:58:09.651040 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606912 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 02:58:09.651040 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606915 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 02:58:09.651040 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606917 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 02:58:09.651040 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606920 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 02:58:09.651040 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606923 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 02:58:09.651040 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606926 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 02:58:09.651040 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606929 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 02:58:09.651040 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606932 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 02:58:09.651040 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606934 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 02:58:09.651040 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606937 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 02:58:09.651040 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606940 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 02:58:09.651040 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606942 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 22 02:58:09.651040 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606945 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 02:58:09.651040 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606948 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 02:58:09.651560 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606951 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 02:58:09.651560 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606953 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 02:58:09.651560 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606956 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 02:58:09.651560 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606959 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 02:58:09.651560 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606962 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 02:58:09.651560 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606970 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 02:58:09.651560 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606973 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 02:58:09.651560 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606976 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 02:58:09.651560 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606979 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 02:58:09.651560 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606981 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 02:58:09.651560 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606984 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 02:58:09.651560 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606987 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 02:58:09.651560 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606989 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 02:58:09.651560 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606992 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 02:58:09.651560 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.606996 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 02:58:09.651560 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.607000 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 02:58:09.651560 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.607003 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 02:58:09.651560 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.607006 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 02:58:09.651560 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.607008 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 02:58:09.651560 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.607011 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 02:58:09.652108 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.607023 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 02:58:09.652108 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.607026 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 02:58:09.652108 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.607029 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 02:58:09.652108 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.607032 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 02:58:09.652108 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.607035 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 02:58:09.652108 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.607038 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 02:58:09.652108 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.607040 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 02:58:09.652108 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.607043 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 02:58:09.652108 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.607046 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 02:58:09.652108 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.607049 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 02:58:09.652108 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.607051 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 02:58:09.652108 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.607055 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 02:58:09.652108 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.607059 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 02:58:09.652108 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.607063 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 02:58:09.652108 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.607066 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 02:58:09.652108 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.607069 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 02:58:09.652108 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.607072 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 02:58:09.652108 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.607075 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 02:58:09.652108 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.607108 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 02:58:09.652108 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.607113 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 02:58:09.652634 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.607116 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 02:58:09.652634 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.607120 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 02:58:09.652634 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.607124 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 02:58:09.652634 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.607128 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 02:58:09.652634 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.607131 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 02:58:09.652634 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.607134 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 02:58:09.652634 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.607142 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 02:58:09.652634 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.607148 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 02:58:09.652634 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.607153 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 02:58:09.652634 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.607158 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 02:58:09.652634 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.607163 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 02:58:09.652634 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.607168 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 02:58:09.652634 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.607171 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 02:58:09.652634 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.607175 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 02:58:09.652634 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.607180 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 02:58:09.652634 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:09.607183 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 02:58:09.653056 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.607189 2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 02:58:09.653056 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.607320 2582 server.go:962] "Client rotation is on, will bootstrap in background" Jul 22 02:58:09.653056 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.610386 2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 22 02:58:09.653056 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.612240 2582 server.go:1019] "Starting client certificate rotation" Jul 22 02:58:09.653056 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.612355 2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 22 02:58:09.653056 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.612400 2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 22 02:58:09.653056 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.615154 2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 22 02:58:09.653056 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.622508 2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 22 02:58:09.653056 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.632201 2582 log.go:25] "Validated CRI v1 runtime API" Jul 22 02:58:09.653056 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.634440 2582 log.go:25] "Validated CRI v1 image API" Jul 22 02:58:09.653056 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.635324 2582 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 22 02:58:09.653056 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.643198 2582 fs.go:135] Filesystem UUIDs: map[18f75532-c830-4e17-bcea-b5c5b74271a8:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 f2ca7930-249f-47d8-b8d0-a417c0b6064a:/dev/nvme0n1p3] Jul 22 02:58:09.653400 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.643229 2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 22 02:58:09.653435 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.651231 2582 manager.go:216] Machine: {Timestamp:2026-07-22 02:58:09.64928451 +0000 UTC m=+0.380905351 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099295 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28d245eb06dfbb5de8a1decc263aa9 SystemUUID:ec28d245-eb06-dfbb-5de8-a1decc263aa9 BootID:6cda4a2d-9dc8-400e-a08b-49b7f93b4035 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:8f:4c:23:94:fd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:8f:4c:23:94:fd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:92:ac:38:77:2f:11 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 22 02:58:09.653435 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.651386 2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 22 02:58:09.653435 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.651565 2582 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 22 02:58:09.655084 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.655048 2582 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 22 02:58:09.655253 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.655097 2582 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-67.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 22 02:58:09.656034 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.656023 2582 topology_manager.go:138] "Creating topology manager with none policy" Jul 22 02:58:09.656068 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.656037 2582 container_manager_linux.go:309] "Creating device plugin manager" Jul 22 02:58:09.656068 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.656045 2582 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 22 02:58:09.656370 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.656354 2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 22 02:58:09.656637 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.656624 2582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 22 02:58:09.658352 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.658340 2582 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 22 02:58:09.661817 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.661803 2582 state_mem.go:36] "Initialized new in-memory state store" Jul 22 02:58:09.661960 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.661950 2582 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 22 02:58:09.661998 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.661986 2582 kubelet.go:486] "Attempting to sync node with API server" Jul 22 02:58:09.662028 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.661998 2582 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 22 02:58:09.662028 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.662017 2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 22 02:58:09.662081 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.662030 2582 kubelet.go:398] "Adding apiserver pod source" Jul 22 02:58:09.662081 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.662043 2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 22 02:58:09.662507 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.662495 2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 22 02:58:09.662544 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.662517 2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 22 02:58:09.665983 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.665952 2582 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 22 02:58:09.667958 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.667941 2582 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 22 02:58:09.668027 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.667974 2582 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 22 02:58:09.669343 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.669330 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 22 02:58:09.669398 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.669353 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 22 02:58:09.669398 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.669364 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 22 02:58:09.669398 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.669375 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 22 02:58:09.669398 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.669387 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 22 02:58:09.669398 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.669393 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 22 02:58:09.669528 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.669402 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 22 02:58:09.669528 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.669408 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 22 02:58:09.669528 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.669416 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 22 02:58:09.669528 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.669422 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 22 02:58:09.669528 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.669431 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 22 02:58:09.669528 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.669436 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 22 02:58:09.670387 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.670376 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 22 02:58:09.670428 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.670389 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 22 02:58:09.672965 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.672942 2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qhwln" Jul 22 02:58:09.673332 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.673303 2582 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-67.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 22 02:58:09.673474 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:09.673450 2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-67.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 22 02:58:09.673548 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:09.673531 2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 22 02:58:09.674339 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.674325 2582 server.go:1268] "Started kubelet" Jul 22 02:58:09.674425 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.674396 2582 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 22 02:58:09.674527 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.674460 2582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 22 02:58:09.681339 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.681254 2582 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 22 02:58:09.681823 ip-10-0-131-67 systemd[1]: Started Kubernetes Kubelet. Jul 22 02:58:09.683810 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.683156 2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qhwln" Jul 22 02:58:09.683810 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.683204 2582 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 22 02:58:09.685939 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:09.685916 2582 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 22 02:58:09.686013 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.685996 2582 server.go:310] "Adding debug handlers to kubelet server" Jul 22 02:58:09.689381 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:09.688158 2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-67.ec2.internal.18c47d45788d3cb4 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-67.ec2.internal,UID:ip-10-0-131-67.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-67.ec2.internal,},FirstTimestamp:2026-07-22 02:58:09.6742965 +0000 UTC m=+0.405917342,LastTimestamp:2026-07-22 02:58:09.6742965 +0000 UTC m=+0.405917342,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-67.ec2.internal,}" Jul 22 02:58:09.690612 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.690565 2582 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 22 02:58:09.690685 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:09.690667 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-67.ec2.internal\" not found" Jul 22 02:58:09.690742 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.690568 2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 22 02:58:09.691194 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.691178 2582 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 22 02:58:09.691287 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.691182 2582 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 22 02:58:09.691376 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.691358 2582 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 22 02:58:09.691492 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.691478 2582 reconstruct.go:97] "Volume reconstruction finished" Jul 22 02:58:09.691492 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.691489 2582 reconciler.go:29] "Reconciler: start to sync state" Jul 22 02:58:09.693395 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.693376 2582 factory.go:55] Registering systemd factory Jul 22 02:58:09.693515 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.693504 2582 factory.go:223] Registration of the systemd container factory successfully Jul 22 02:58:09.694430 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.694408 2582 factory.go:153] Registering CRI-O factory Jul 22 02:58:09.694430 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.694427 2582 factory.go:223] Registration of the crio container factory successfully Jul 22 02:58:09.694624 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.694480 2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 22 02:58:09.694624 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.694505 2582 factory.go:103] Registering Raw factory Jul 22 02:58:09.694624 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.694519 2582 manager.go:1196] Started watching for new ooms in manager Jul 22 02:58:09.695003 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.694986 2582 manager.go:349] Starting recovery of all containers Jul 22 02:58:09.705392 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.705364 2582 manager.go:354] Recovery completed Jul 22 02:58:09.705906 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.705882 2582 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 02:58:09.709784 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:09.709758 2582 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-131-67.ec2.internal\" not found" node="ip-10-0-131-67.ec2.internal" Jul 22 02:58:09.711455 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.711440 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 02:58:09.713930 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.713912 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-67.ec2.internal" event="NodeHasSufficientMemory" Jul 22 02:58:09.714034 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.713944 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-67.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 02:58:09.714034 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.713965 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-67.ec2.internal" event="NodeHasSufficientPID" Jul 22 02:58:09.714745 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.714730 2582 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 22 02:58:09.714745 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.714741 2582 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 22 02:58:09.714958 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.714749 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 02:58:09.714958 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.714761 2582 state_mem.go:36] "Initialized new in-memory state store" Jul 22 02:58:09.716000 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.715979 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-67.ec2.internal" event="NodeHasSufficientMemory" Jul 22 02:58:09.716108 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.716016 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-67.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 02:58:09.716108 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.716048 2582 policy_none.go:49] "None policy: Start" Jul 22 02:58:09.716108 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.716071 2582 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 22 02:58:09.716108 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.716086 2582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 22 02:58:09.716318 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.716055 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-67.ec2.internal" event="NodeHasSufficientPID" Jul 22 02:58:09.717362 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.717344 2582 policy_none.go:47] "Start" Jul 22 02:58:09.758130 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.757925 2582 manager.go:337] "Starting Device Plugin manager" Jul 22 02:58:09.758325 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:09.758171 2582 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 22 02:58:09.758325 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.758188 2582 server.go:87] "Starting device plugin registration server" Jul 22 02:58:09.758487 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.758474 2582 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 22 02:58:09.758753 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.758491 2582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 22 02:58:09.758753 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.758574 2582 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 22 02:58:09.758753 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.758659 2582 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 22 02:58:09.758753 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.758667 2582 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 22 02:58:09.759690 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:09.759669 2582 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 22 02:58:09.759807 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:09.759742 2582 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-67.ec2.internal\" not found" Jul 22 02:58:09.859324 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.859294 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 02:58:09.860778 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.860757 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-67.ec2.internal" event="NodeHasSufficientMemory" Jul 22 02:58:09.860853 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.860799 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-67.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 02:58:09.860853 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.860812 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-67.ec2.internal" event="NodeHasSufficientPID" Jul 22 02:58:09.860853 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.860841 2582 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-67.ec2.internal" Jul 22 02:58:09.869415 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.869392 2582 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-67.ec2.internal" Jul 22 02:58:09.869518 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:09.869421 2582 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-67.ec2.internal\": node \"ip-10-0-131-67.ec2.internal\" not found" Jul 22 02:58:09.883121 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:09.883087 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-67.ec2.internal\" not found" Jul 22 02:58:09.893353 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.893314 2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 22 02:58:09.894678 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.894657 2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 22 02:58:09.894786 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.894706 2582 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 22 02:58:09.894786 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.894734 2582 kubelet.go:2448] "Starting kubelet main sync loop" Jul 22 02:58:09.894786 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:09.894777 2582 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 22 02:58:09.896926 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.896902 2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 02:58:09.988127 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:09.988033 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-67.ec2.internal\" not found" Jul 22 02:58:09.995792 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.995732 2582 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-67.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-67.ec2.internal"] Jul 22 02:58:09.995934 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.995845 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 02:58:09.998500 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.998483 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-67.ec2.internal" event="NodeHasSufficientMemory" Jul 22 02:58:09.998581 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.998515 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-67.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 02:58:09.998581 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:09.998526 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-67.ec2.internal" event="NodeHasSufficientPID" Jul 22 02:58:10.000112 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:10.000098 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 02:58:10.000215 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:10.000200 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-67.ec2.internal" Jul 22 02:58:10.000260 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:10.000243 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 02:58:10.001223 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:10.001205 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-67.ec2.internal" event="NodeHasSufficientMemory" Jul 22 02:58:10.001297 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:10.001236 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-67.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 02:58:10.001297 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:10.001250 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-67.ec2.internal" event="NodeHasSufficientPID" Jul 22 02:58:10.001297 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:10.001268 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-67.ec2.internal" event="NodeHasSufficientMemory" Jul 22 02:58:10.001297 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:10.001291 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-67.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 02:58:10.001457 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:10.001302 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-67.ec2.internal" event="NodeHasSufficientPID" Jul 22 02:58:10.002515 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:10.002498 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-67.ec2.internal" Jul 22 02:58:10.002571 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:10.002533 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 02:58:10.003281 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:10.003265 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-67.ec2.internal" event="NodeHasSufficientMemory" Jul 22 02:58:10.003335 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:10.003298 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-67.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 02:58:10.003335 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:10.003313 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-67.ec2.internal" event="NodeHasSufficientPID" Jul 22 02:58:10.037162 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:10.037137 2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-67.ec2.internal\" not found" node="ip-10-0-131-67.ec2.internal" Jul 22 02:58:10.042022 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:10.042006 2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-67.ec2.internal\" not found" node="ip-10-0-131-67.ec2.internal" Jul 22 02:58:10.088839 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:10.088800 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-67.ec2.internal\" not found" Jul 22 02:58:10.094645 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:10.094622 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7f651843bca4c08d31d08118e2ebefbe-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-67.ec2.internal\" (UID: \"7f651843bca4c08d31d08118e2ebefbe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-67.ec2.internal" Jul 22 02:58:10.094722 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:10.094653 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f651843bca4c08d31d08118e2ebefbe-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-67.ec2.internal\" (UID: \"7f651843bca4c08d31d08118e2ebefbe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-67.ec2.internal" Jul 22 02:58:10.094722 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:10.094673 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5d8b9a289a92f6d831f68731b2b2419a-config\") pod \"kube-apiserver-proxy-ip-10-0-131-67.ec2.internal\" (UID: \"5d8b9a289a92f6d831f68731b2b2419a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-67.ec2.internal" Jul 22 02:58:10.190647 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:10.190599 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-67.ec2.internal\" not found" Jul 22 02:58:10.195250 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:10.195224 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7f651843bca4c08d31d08118e2ebefbe-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-67.ec2.internal\" (UID: \"7f651843bca4c08d31d08118e2ebefbe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-67.ec2.internal" Jul 22 02:58:10.195310 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:10.195256 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f651843bca4c08d31d08118e2ebefbe-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-67.ec2.internal\" (UID: \"7f651843bca4c08d31d08118e2ebefbe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-67.ec2.internal" Jul 22 02:58:10.195310 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:10.195275 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5d8b9a289a92f6d831f68731b2b2419a-config\") pod \"kube-apiserver-proxy-ip-10-0-131-67.ec2.internal\" (UID: \"5d8b9a289a92f6d831f68731b2b2419a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-67.ec2.internal" Jul 22 02:58:10.195389 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:10.195328 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f651843bca4c08d31d08118e2ebefbe-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-67.ec2.internal\" (UID: \"7f651843bca4c08d31d08118e2ebefbe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-67.ec2.internal" Jul 22 02:58:10.195389 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:10.195338 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7f651843bca4c08d31d08118e2ebefbe-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-67.ec2.internal\" (UID: \"7f651843bca4c08d31d08118e2ebefbe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-67.ec2.internal" Jul 22 02:58:10.195389 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:10.195371 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5d8b9a289a92f6d831f68731b2b2419a-config\") pod \"kube-apiserver-proxy-ip-10-0-131-67.ec2.internal\" (UID: \"5d8b9a289a92f6d831f68731b2b2419a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-67.ec2.internal" Jul 22 02:58:10.296680 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:10.296591 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-67.ec2.internal\" not found" Jul 22 02:58:10.343349 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:10.343307 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-67.ec2.internal" Jul 22 02:58:10.345648 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:10.345629 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-67.ec2.internal" Jul 22 02:58:10.396907 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:10.396864 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-67.ec2.internal\" not found" Jul 22 02:58:10.497824 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:10.497787 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-67.ec2.internal\" not found" Jul 22 02:58:10.542033 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:10.541999 2582 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 02:58:10.598320 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:10.598240 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-67.ec2.internal\" not found" Jul 22 02:58:10.611560 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:10.611528 2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 22 02:58:10.611768 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:10.611741 2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 22 02:58:10.611768 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:10.611735 2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 22 02:58:10.611864 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:10.611749 2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 22 02:58:10.689468 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:10.689429 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-21 02:53:10 +0000 UTC" deadline="2028-03-08 17:58:55.236349599 +0000 UTC" Jul 22 02:58:10.689468 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:10.689459 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14295h0m44.546894053s" Jul 22 02:58:10.698872 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:10.698843 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-67.ec2.internal\" not found" Jul 22 02:58:10.799518 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:10.799479 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-67.ec2.internal\" not found" Jul 22 02:58:10.899574 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:10.899543 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-67.ec2.internal\" not found" Jul 22 02:58:10.963469 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:10.963450 2582 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 02:58:10.975459 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:10.975371 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f651843bca4c08d31d08118e2ebefbe.slice/crio-b2d1ec54e410809c9dbcaa0b72f86676ca77359c9119e75ed3bddaeef8c712bb WatchSource:0}: Error finding container b2d1ec54e410809c9dbcaa0b72f86676ca77359c9119e75ed3bddaeef8c712bb: Status 404 returned error can't find the container with id b2d1ec54e410809c9dbcaa0b72f86676ca77359c9119e75ed3bddaeef8c712bb Jul 22 02:58:10.975863 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:10.975826 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d8b9a289a92f6d831f68731b2b2419a.slice/crio-21a673823e1513bd0d112fdde45265c897e6e0966e4c8d55a63acbd2b7425bfb WatchSource:0}: Error finding container 21a673823e1513bd0d112fdde45265c897e6e0966e4c8d55a63acbd2b7425bfb: Status 404 returned error can't find the container with id 21a673823e1513bd0d112fdde45265c897e6e0966e4c8d55a63acbd2b7425bfb Jul 22 02:58:10.981987 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:10.981969 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 02:58:10.991762 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:10.991743 2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-67.ec2.internal" Jul 22 02:58:11.002814 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.002795 2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 22 02:58:11.004080 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.004067 2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-67.ec2.internal" Jul 22 02:58:11.010887 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.010873 2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 22 02:58:11.636385 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.636353 2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 02:58:11.666701 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.666667 2582 apiserver.go:52] "Watching apiserver" Jul 22 02:58:11.673668 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.673632 2582 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 22 02:58:11.675108 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.675074 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-5gftd","openshift-image-registry/node-ca-vssht","openshift-multus/multus-additional-cni-plugins-6hdd2","openshift-multus/network-metrics-daemon-jqnjj","kube-system/konnectivity-agent-28hhd","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-67.ec2.internal","openshift-multus/multus-vddpp","openshift-network-diagnostics/network-check-target-5xv9c","openshift-network-operator/iptables-alerter-7tf82","openshift-ovn-kubernetes/ovnkube-node-5rdkb","kube-system/global-pull-secret-syncer-lc5mn","kube-system/kube-apiserver-proxy-ip-10-0-131-67.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wcq24","openshift-cluster-node-tuning-operator/tuned-dlq9k"] Jul 22 02:58:11.676673 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.676648 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vddpp" Jul 22 02:58:11.678189 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.678168 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wcq24" Jul 22 02:58:11.679336 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.679311 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 22 02:58:11.679441 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.679312 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 22 02:58:11.679529 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.679459 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 22 02:58:11.679608 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.679568 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-nhb88\"" Jul 22 02:58:11.679802 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.679782 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 22 02:58:11.679868 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.679844 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6hdd2" Jul 22 02:58:11.680866 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.680581 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 22 02:58:11.680866 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.680612 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5kn59\"" Jul 22 02:58:11.681061 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.680923 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jqnjj" Jul 22 02:58:11.681061 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:11.680993 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jqnjj" podUID="75610210-6577-4ea8-b77d-a29a12c9a972" Jul 22 02:58:11.681405 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.681373 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 22 02:58:11.681600 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.681584 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 22 02:58:11.682143 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.682122 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 22 02:58:11.682292 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.682275 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 22 02:58:11.682356 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.682323 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-vzz55\"" Jul 22 02:58:11.684006 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.683985 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-28hhd" Jul 22 02:58:11.685704 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.685682 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dlq9k" Jul 22 02:58:11.686283 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.686016 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-vcmq5\"" Jul 22 02:58:11.686283 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.686231 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 22 02:58:11.686446 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.686323 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 22 02:58:11.687609 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.687587 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7tf82" Jul 22 02:58:11.688049 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.688024 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 22 02:58:11.688491 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.688473 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-nlh5f\"" Jul 22 02:58:11.688979 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.688550 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 22 02:58:11.689598 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.689244 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5gftd" Jul 22 02:58:11.690180 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.690064 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 22 02:58:11.690180 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.690142 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 22 02:58:11.690526 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.690509 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 22 02:58:11.691013 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.690756 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vssht" Jul 22 02:58:11.691322 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.691304 2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 22 02:58:11.691894 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.691852 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 22 02:58:11.693394 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.692796 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ct7fj\"" Jul 22 02:58:11.693394 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.692978 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5rdkb" Jul 22 02:58:11.693394 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.693136 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-mkqr4\"" Jul 22 02:58:11.693650 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.693621 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 22 02:58:11.695451 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.695424 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 22 02:58:11.695811 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.695782 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zkbfs\"" Jul 22 02:58:11.695918 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.695851 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 22 02:58:11.695993 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.695783 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 22 02:58:11.696080 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.695999 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 22 02:58:11.699455 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.698030 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 22 02:58:11.699455 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.698526 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 22 02:58:11.699455 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.698694 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 22 02:58:11.699455 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.698720 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 22 02:58:11.699455 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.698996 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5ch4h\"" Jul 22 02:58:11.699455 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.699007 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lc5mn" Jul 22 02:58:11.699455 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.699095 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 22 02:58:11.699455 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:11.699125 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lc5mn" podUID="5bd7c5db-509d-4f85-82ac-8e357591ae72" Jul 22 02:58:11.702027 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.702008 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5xv9c" Jul 22 02:58:11.702316 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:11.702292 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5xv9c" podUID="0dc67dcb-94d3-439a-8b1b-6c238ccc621d" Jul 22 02:58:11.703557 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.703535 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/50b3c714-4147-42c8-94fa-5c2bfd912518-etc-kubernetes\") pod \"tuned-dlq9k\" (UID: \"50b3c714-4147-42c8-94fa-5c2bfd912518\") " pod="openshift-cluster-node-tuning-operator/tuned-dlq9k" Jul 22 02:58:11.703661 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.703573 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/620683de-e754-438d-9d96-3e0cb06a80c6-host-slash\") pod \"ovnkube-node-5rdkb\" (UID: \"620683de-e754-438d-9d96-3e0cb06a80c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rdkb" Jul 22 02:58:11.703661 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.703602 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/620683de-e754-438d-9d96-3e0cb06a80c6-var-lib-openvswitch\") pod \"ovnkube-node-5rdkb\" (UID: \"620683de-e754-438d-9d96-3e0cb06a80c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rdkb" Jul 22 02:58:11.703661 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.703628 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/620683de-e754-438d-9d96-3e0cb06a80c6-log-socket\") pod \"ovnkube-node-5rdkb\" (UID: \"620683de-e754-438d-9d96-3e0cb06a80c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rdkb" Jul 22 02:58:11.703661 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.703654 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/620683de-e754-438d-9d96-3e0cb06a80c6-ovn-node-metrics-cert\") pod \"ovnkube-node-5rdkb\" (UID: \"620683de-e754-438d-9d96-3e0cb06a80c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rdkb" Jul 22 02:58:11.703869 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.703678 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60fd8d72-791a-4e19-9867-6256650aa2da-host-var-lib-kubelet\") pod \"multus-vddpp\" (UID: \"60fd8d72-791a-4e19-9867-6256650aa2da\") " pod="openshift-multus/multus-vddpp" Jul 22 02:58:11.703869 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.703703 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/50b3c714-4147-42c8-94fa-5c2bfd912518-etc-sysconfig\") pod \"tuned-dlq9k\" (UID: \"50b3c714-4147-42c8-94fa-5c2bfd912518\") " pod="openshift-cluster-node-tuning-operator/tuned-dlq9k" Jul 22 02:58:11.703869 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.703728 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60fd8d72-791a-4e19-9867-6256650aa2da-etc-kubernetes\") pod \"multus-vddpp\" (UID: \"60fd8d72-791a-4e19-9867-6256650aa2da\") " pod="openshift-multus/multus-vddpp" Jul 22 02:58:11.703869 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.703753 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a0d8c32d-7c06-494e-9b06-af164d19eb20-socket-dir\") pod \"aws-ebs-csi-driver-node-wcq24\" (UID: \"a0d8c32d-7c06-494e-9b06-af164d19eb20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wcq24" Jul 22 02:58:11.703869 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.703777 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/50b3c714-4147-42c8-94fa-5c2bfd912518-etc-systemd\") pod \"tuned-dlq9k\" (UID: \"50b3c714-4147-42c8-94fa-5c2bfd912518\") " pod="openshift-cluster-node-tuning-operator/tuned-dlq9k" Jul 22 02:58:11.703869 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.703800 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/60fd8d72-791a-4e19-9867-6256650aa2da-host-var-lib-cni-multus\") pod \"multus-vddpp\" (UID: \"60fd8d72-791a-4e19-9867-6256650aa2da\") " pod="openshift-multus/multus-vddpp" Jul 22 02:58:11.703869 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.703825 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/60fd8d72-791a-4e19-9867-6256650aa2da-host-run-multus-certs\") pod \"multus-vddpp\" (UID: \"60fd8d72-791a-4e19-9867-6256650aa2da\") " pod="openshift-multus/multus-vddpp" Jul 22 02:58:11.703869 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.703849 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a0d8c32d-7c06-494e-9b06-af164d19eb20-device-dir\") pod \"aws-ebs-csi-driver-node-wcq24\" (UID: \"a0d8c32d-7c06-494e-9b06-af164d19eb20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wcq24" Jul 22 02:58:11.704322 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.703889 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/50b3c714-4147-42c8-94fa-5c2bfd912518-host\") pod \"tuned-dlq9k\" (UID: \"50b3c714-4147-42c8-94fa-5c2bfd912518\") " pod="openshift-cluster-node-tuning-operator/tuned-dlq9k" Jul 22 02:58:11.704322 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.703912 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/620683de-e754-438d-9d96-3e0cb06a80c6-run-ovn\") pod \"ovnkube-node-5rdkb\" (UID: \"620683de-e754-438d-9d96-3e0cb06a80c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rdkb" Jul 22 02:58:11.704322 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.703959 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/620683de-e754-438d-9d96-3e0cb06a80c6-host-kubelet\") pod \"ovnkube-node-5rdkb\" (UID: \"620683de-e754-438d-9d96-3e0cb06a80c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rdkb" Jul 22 02:58:11.704322 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.703983 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/620683de-e754-438d-9d96-3e0cb06a80c6-host-run-netns\") pod \"ovnkube-node-5rdkb\" (UID: \"620683de-e754-438d-9d96-3e0cb06a80c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rdkb" Jul 22 02:58:11.704322 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.704064 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/620683de-e754-438d-9d96-3e0cb06a80c6-env-overrides\") pod \"ovnkube-node-5rdkb\" (UID: \"620683de-e754-438d-9d96-3e0cb06a80c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rdkb" Jul 22 02:58:11.704322 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.704128 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a0d8c32d-7c06-494e-9b06-af164d19eb20-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wcq24\" (UID: \"a0d8c32d-7c06-494e-9b06-af164d19eb20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wcq24" Jul 22 02:58:11.704322 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.704158 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6ded3042-9b57-4211-af1c-6660ad8363ef-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6hdd2\" (UID: \"6ded3042-9b57-4211-af1c-6660ad8363ef\") " pod="openshift-multus/multus-additional-cni-plugins-6hdd2" Jul 22 02:58:11.704322 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.704187 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6ded3042-9b57-4211-af1c-6660ad8363ef-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6hdd2\" (UID: \"6ded3042-9b57-4211-af1c-6660ad8363ef\") " pod="openshift-multus/multus-additional-cni-plugins-6hdd2" Jul 22 02:58:11.704322 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.704212 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e33233aa-5e20-483c-8bb0-0cd5a577da98-agent-certs\") pod \"konnectivity-agent-28hhd\" (UID: \"e33233aa-5e20-483c-8bb0-0cd5a577da98\") " pod="kube-system/konnectivity-agent-28hhd" Jul 22 02:58:11.704322 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.704236 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/50b3c714-4147-42c8-94fa-5c2bfd912518-etc-sysctl-conf\") pod \"tuned-dlq9k\" (UID: \"50b3c714-4147-42c8-94fa-5c2bfd912518\") " pod="openshift-cluster-node-tuning-operator/tuned-dlq9k" Jul 22 02:58:11.704322 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.704261 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/50b3c714-4147-42c8-94fa-5c2bfd912518-lib-modules\") pod \"tuned-dlq9k\" (UID: \"50b3c714-4147-42c8-94fa-5c2bfd912518\") " pod="openshift-cluster-node-tuning-operator/tuned-dlq9k" Jul 22 02:58:11.704322 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.704286 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a47f1abd-b70b-402b-9cf6-cdff61276fec-iptables-alerter-script\") pod \"iptables-alerter-7tf82\" (UID: \"a47f1abd-b70b-402b-9cf6-cdff61276fec\") " pod="openshift-network-operator/iptables-alerter-7tf82" Jul 22 02:58:11.704322 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.704312 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50b3c714-4147-42c8-94fa-5c2bfd912518-tmp\") pod \"tuned-dlq9k\" (UID: \"50b3c714-4147-42c8-94fa-5c2bfd912518\") " pod="openshift-cluster-node-tuning-operator/tuned-dlq9k" Jul 22 02:58:11.704950 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.704337 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60fd8d72-791a-4e19-9867-6256650aa2da-system-cni-dir\") pod \"multus-vddpp\" (UID: \"60fd8d72-791a-4e19-9867-6256650aa2da\") " pod="openshift-multus/multus-vddpp" Jul 22 02:58:11.704950 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.704372 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/60fd8d72-791a-4e19-9867-6256650aa2da-multus-socket-dir-parent\") pod \"multus-vddpp\" (UID: \"60fd8d72-791a-4e19-9867-6256650aa2da\") " pod="openshift-multus/multus-vddpp" Jul 22 02:58:11.704950 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.704395 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/60fd8d72-791a-4e19-9867-6256650aa2da-hostroot\") pod \"multus-vddpp\" (UID: \"60fd8d72-791a-4e19-9867-6256650aa2da\") " pod="openshift-multus/multus-vddpp" Jul 22 02:58:11.704950 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.704418 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a0d8c32d-7c06-494e-9b06-af164d19eb20-sys-fs\") pod \"aws-ebs-csi-driver-node-wcq24\" (UID: \"a0d8c32d-7c06-494e-9b06-af164d19eb20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wcq24" Jul 22 02:58:11.704950 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.704452 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rflwk\" (UniqueName: \"kubernetes.io/projected/a0d8c32d-7c06-494e-9b06-af164d19eb20-kube-api-access-rflwk\") pod \"aws-ebs-csi-driver-node-wcq24\" (UID: \"a0d8c32d-7c06-494e-9b06-af164d19eb20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wcq24" Jul 22 02:58:11.704950 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.704495 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e33233aa-5e20-483c-8bb0-0cd5a577da98-konnectivity-ca\") pod \"konnectivity-agent-28hhd\" (UID: \"e33233aa-5e20-483c-8bb0-0cd5a577da98\") " pod="kube-system/konnectivity-agent-28hhd" Jul 22 02:58:11.704950 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.704579 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/50b3c714-4147-42c8-94fa-5c2bfd912518-etc-modprobe-d\") pod \"tuned-dlq9k\" (UID: \"50b3c714-4147-42c8-94fa-5c2bfd912518\") " pod="openshift-cluster-node-tuning-operator/tuned-dlq9k" Jul 22 02:58:11.704950 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.704616 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/60fd8d72-791a-4e19-9867-6256650aa2da-host-run-k8s-cni-cncf-io\") pod \"multus-vddpp\" (UID: \"60fd8d72-791a-4e19-9867-6256650aa2da\") " pod="openshift-multus/multus-vddpp" Jul 22 02:58:11.704950 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.704652 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/60fd8d72-791a-4e19-9867-6256650aa2da-host-var-lib-cni-bin\") pod \"multus-vddpp\" (UID: \"60fd8d72-791a-4e19-9867-6256650aa2da\") " pod="openshift-multus/multus-vddpp" Jul 22 02:58:11.704950 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.704696 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb8qj\" (UniqueName: \"kubernetes.io/projected/6ded3042-9b57-4211-af1c-6660ad8363ef-kube-api-access-hb8qj\") pod \"multus-additional-cni-plugins-6hdd2\" (UID: \"6ded3042-9b57-4211-af1c-6660ad8363ef\") " pod="openshift-multus/multus-additional-cni-plugins-6hdd2" Jul 22 02:58:11.704950 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.704727 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9zqv\" (UniqueName: \"kubernetes.io/projected/8c3db93b-85e3-4f29-a0cc-1b6b3d847a94-kube-api-access-j9zqv\") pod \"node-resolver-5gftd\" (UID: \"8c3db93b-85e3-4f29-a0cc-1b6b3d847a94\") " pod="openshift-dns/node-resolver-5gftd" Jul 22 02:58:11.704950 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.704760 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knw87\" (UniqueName: \"kubernetes.io/projected/5ee42689-5bff-46fe-8047-adbe6cb34273-kube-api-access-knw87\") pod \"node-ca-vssht\" (UID: \"5ee42689-5bff-46fe-8047-adbe6cb34273\") " pod="openshift-image-registry/node-ca-vssht" Jul 22 02:58:11.704950 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.704797 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/620683de-e754-438d-9d96-3e0cb06a80c6-run-systemd\") pod \"ovnkube-node-5rdkb\" (UID: \"620683de-e754-438d-9d96-3e0cb06a80c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rdkb" Jul 22 02:58:11.704950 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.704826 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/620683de-e754-438d-9d96-3e0cb06a80c6-host-cni-bin\") pod \"ovnkube-node-5rdkb\" (UID: \"620683de-e754-438d-9d96-3e0cb06a80c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rdkb" Jul 22 02:58:11.705569 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.704911 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/620683de-e754-438d-9d96-3e0cb06a80c6-ovnkube-script-lib\") pod \"ovnkube-node-5rdkb\" (UID: \"620683de-e754-438d-9d96-3e0cb06a80c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rdkb" Jul 22 02:58:11.705569 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.704965 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkz59\" (UniqueName: \"kubernetes.io/projected/620683de-e754-438d-9d96-3e0cb06a80c6-kube-api-access-qkz59\") pod \"ovnkube-node-5rdkb\" (UID: \"620683de-e754-438d-9d96-3e0cb06a80c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rdkb" Jul 22 02:58:11.705569 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.705000 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a0d8c32d-7c06-494e-9b06-af164d19eb20-registration-dir\") pod \"aws-ebs-csi-driver-node-wcq24\" (UID: \"a0d8c32d-7c06-494e-9b06-af164d19eb20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wcq24" Jul 22 02:58:11.705569 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.705042 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6ded3042-9b57-4211-af1c-6660ad8363ef-system-cni-dir\") pod \"multus-additional-cni-plugins-6hdd2\" (UID: \"6ded3042-9b57-4211-af1c-6660ad8363ef\") " pod="openshift-multus/multus-additional-cni-plugins-6hdd2" Jul 22 02:58:11.705569 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.705066 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/50b3c714-4147-42c8-94fa-5c2bfd912518-run\") pod \"tuned-dlq9k\" (UID: \"50b3c714-4147-42c8-94fa-5c2bfd912518\") " pod="openshift-cluster-node-tuning-operator/tuned-dlq9k" Jul 22 02:58:11.705569 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.705088 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/50b3c714-4147-42c8-94fa-5c2bfd912518-sys\") pod \"tuned-dlq9k\" (UID: \"50b3c714-4147-42c8-94fa-5c2bfd912518\") " pod="openshift-cluster-node-tuning-operator/tuned-dlq9k" Jul 22 02:58:11.705569 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.705128 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8c3db93b-85e3-4f29-a0cc-1b6b3d847a94-hosts-file\") pod \"node-resolver-5gftd\" (UID: \"8c3db93b-85e3-4f29-a0cc-1b6b3d847a94\") " pod="openshift-dns/node-resolver-5gftd" Jul 22 02:58:11.705569 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.705153 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/620683de-e754-438d-9d96-3e0cb06a80c6-etc-openvswitch\") pod \"ovnkube-node-5rdkb\" (UID: \"620683de-e754-438d-9d96-3e0cb06a80c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rdkb" Jul 22 02:58:11.705569 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.705179 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/620683de-e754-438d-9d96-3e0cb06a80c6-run-openvswitch\") pod \"ovnkube-node-5rdkb\" (UID: \"620683de-e754-438d-9d96-3e0cb06a80c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rdkb" Jul 22 02:58:11.705569 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.705213 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60fd8d72-791a-4e19-9867-6256650aa2da-multus-cni-dir\") pod \"multus-vddpp\" (UID: \"60fd8d72-791a-4e19-9867-6256650aa2da\") " pod="openshift-multus/multus-vddpp" Jul 22 02:58:11.705569 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.705228 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8fh5\" (UniqueName: \"kubernetes.io/projected/a47f1abd-b70b-402b-9cf6-cdff61276fec-kube-api-access-c8fh5\") pod \"iptables-alerter-7tf82\" (UID: \"a47f1abd-b70b-402b-9cf6-cdff61276fec\") " pod="openshift-network-operator/iptables-alerter-7tf82" Jul 22 02:58:11.705569 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.705251 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6ded3042-9b57-4211-af1c-6660ad8363ef-cnibin\") pod \"multus-additional-cni-plugins-6hdd2\" (UID: \"6ded3042-9b57-4211-af1c-6660ad8363ef\") " pod="openshift-multus/multus-additional-cni-plugins-6hdd2" Jul 22 02:58:11.705569 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.705285 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75610210-6577-4ea8-b77d-a29a12c9a972-metrics-certs\") pod \"network-metrics-daemon-jqnjj\" (UID: \"75610210-6577-4ea8-b77d-a29a12c9a972\") " pod="openshift-multus/network-metrics-daemon-jqnjj" Jul 22 02:58:11.706215 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.705301 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6ded3042-9b57-4211-af1c-6660ad8363ef-cni-binary-copy\") pod \"multus-additional-cni-plugins-6hdd2\" (UID: \"6ded3042-9b57-4211-af1c-6660ad8363ef\") " pod="openshift-multus/multus-additional-cni-plugins-6hdd2" Jul 22 02:58:11.706215 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.705318 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/50b3c714-4147-42c8-94fa-5c2bfd912518-var-lib-kubelet\") pod \"tuned-dlq9k\" (UID: \"50b3c714-4147-42c8-94fa-5c2bfd912518\") " pod="openshift-cluster-node-tuning-operator/tuned-dlq9k" Jul 22 02:58:11.706215 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.705342 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkdjn\" (UniqueName: \"kubernetes.io/projected/50b3c714-4147-42c8-94fa-5c2bfd912518-kube-api-access-gkdjn\") pod \"tuned-dlq9k\" (UID: \"50b3c714-4147-42c8-94fa-5c2bfd912518\") " pod="openshift-cluster-node-tuning-operator/tuned-dlq9k" Jul 22 02:58:11.706215 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.705382 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/620683de-e754-438d-9d96-3e0cb06a80c6-node-log\") pod \"ovnkube-node-5rdkb\" (UID: \"620683de-e754-438d-9d96-3e0cb06a80c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rdkb" Jul 22 02:58:11.706215 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.705405 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/620683de-e754-438d-9d96-3e0cb06a80c6-ovnkube-config\") pod \"ovnkube-node-5rdkb\" (UID: \"620683de-e754-438d-9d96-3e0cb06a80c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rdkb" Jul 22 02:58:11.706215 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.705430 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a0d8c32d-7c06-494e-9b06-af164d19eb20-etc-selinux\") pod \"aws-ebs-csi-driver-node-wcq24\" (UID: \"a0d8c32d-7c06-494e-9b06-af164d19eb20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wcq24" Jul 22 02:58:11.706215 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.705457 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a47f1abd-b70b-402b-9cf6-cdff61276fec-host-slash\") pod \"iptables-alerter-7tf82\" (UID: \"a47f1abd-b70b-402b-9cf6-cdff61276fec\") " pod="openshift-network-operator/iptables-alerter-7tf82" Jul 22 02:58:11.706215 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.705484 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prr8p\" (UniqueName: \"kubernetes.io/projected/75610210-6577-4ea8-b77d-a29a12c9a972-kube-api-access-prr8p\") pod \"network-metrics-daemon-jqnjj\" (UID: \"75610210-6577-4ea8-b77d-a29a12c9a972\") " pod="openshift-multus/network-metrics-daemon-jqnjj" Jul 22 02:58:11.706215 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.705523 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/620683de-e754-438d-9d96-3e0cb06a80c6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5rdkb\" (UID: \"620683de-e754-438d-9d96-3e0cb06a80c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rdkb" Jul 22 02:58:11.706215 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.705550 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/60fd8d72-791a-4e19-9867-6256650aa2da-cnibin\") pod \"multus-vddpp\" (UID: \"60fd8d72-791a-4e19-9867-6256650aa2da\") " pod="openshift-multus/multus-vddpp" Jul 22 02:58:11.706215 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.705575 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/60fd8d72-791a-4e19-9867-6256650aa2da-multus-daemon-config\") pod \"multus-vddpp\" (UID: \"60fd8d72-791a-4e19-9867-6256650aa2da\") " pod="openshift-multus/multus-vddpp" Jul 22 02:58:11.706215 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.705610 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6ded3042-9b57-4211-af1c-6660ad8363ef-os-release\") pod \"multus-additional-cni-plugins-6hdd2\" (UID: \"6ded3042-9b57-4211-af1c-6660ad8363ef\") " pod="openshift-multus/multus-additional-cni-plugins-6hdd2" Jul 22 02:58:11.706215 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.705634 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/50b3c714-4147-42c8-94fa-5c2bfd912518-etc-tuned\") pod \"tuned-dlq9k\" (UID: \"50b3c714-4147-42c8-94fa-5c2bfd912518\") " pod="openshift-cluster-node-tuning-operator/tuned-dlq9k" Jul 22 02:58:11.706855 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.705658 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c3db93b-85e3-4f29-a0cc-1b6b3d847a94-tmp-dir\") pod \"node-resolver-5gftd\" (UID: \"8c3db93b-85e3-4f29-a0cc-1b6b3d847a94\") " pod="openshift-dns/node-resolver-5gftd" Jul 22 02:58:11.706855 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.705681 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/60fd8d72-791a-4e19-9867-6256650aa2da-os-release\") pod \"multus-vddpp\" (UID: \"60fd8d72-791a-4e19-9867-6256650aa2da\") " pod="openshift-multus/multus-vddpp" Jul 22 02:58:11.706855 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.705707 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6ded3042-9b57-4211-af1c-6660ad8363ef-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6hdd2\" (UID: \"6ded3042-9b57-4211-af1c-6660ad8363ef\") " pod="openshift-multus/multus-additional-cni-plugins-6hdd2" Jul 22 02:58:11.706855 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.705742 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/50b3c714-4147-42c8-94fa-5c2bfd912518-etc-sysctl-d\") pod \"tuned-dlq9k\" (UID: \"50b3c714-4147-42c8-94fa-5c2bfd912518\") " pod="openshift-cluster-node-tuning-operator/tuned-dlq9k" Jul 22 02:58:11.706855 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.705787 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5ee42689-5bff-46fe-8047-adbe6cb34273-host\") pod \"node-ca-vssht\" (UID: \"5ee42689-5bff-46fe-8047-adbe6cb34273\") " pod="openshift-image-registry/node-ca-vssht" Jul 22 02:58:11.706855 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.705831 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/620683de-e754-438d-9d96-3e0cb06a80c6-systemd-units\") pod \"ovnkube-node-5rdkb\" (UID: \"620683de-e754-438d-9d96-3e0cb06a80c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rdkb" Jul 22 02:58:11.706855 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.705871 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/620683de-e754-438d-9d96-3e0cb06a80c6-host-cni-netd\") pod \"ovnkube-node-5rdkb\" (UID: \"620683de-e754-438d-9d96-3e0cb06a80c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rdkb" Jul 22 02:58:11.706855 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.705895 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/60fd8d72-791a-4e19-9867-6256650aa2da-cni-binary-copy\") pod \"multus-vddpp\" (UID: \"60fd8d72-791a-4e19-9867-6256650aa2da\") " pod="openshift-multus/multus-vddpp" Jul 22 02:58:11.706855 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.705930 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/60fd8d72-791a-4e19-9867-6256650aa2da-multus-conf-dir\") pod \"multus-vddpp\" (UID: \"60fd8d72-791a-4e19-9867-6256650aa2da\") " pod="openshift-multus/multus-vddpp" Jul 22 02:58:11.706855 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.705962 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5ee42689-5bff-46fe-8047-adbe6cb34273-serviceca\") pod \"node-ca-vssht\" (UID: \"5ee42689-5bff-46fe-8047-adbe6cb34273\") " pod="openshift-image-registry/node-ca-vssht" Jul 22 02:58:11.706855 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.705998 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/620683de-e754-438d-9d96-3e0cb06a80c6-host-run-ovn-kubernetes\") pod \"ovnkube-node-5rdkb\" (UID: \"620683de-e754-438d-9d96-3e0cb06a80c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rdkb" Jul 22 02:58:11.706855 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.706037 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/60fd8d72-791a-4e19-9867-6256650aa2da-host-run-netns\") pod \"multus-vddpp\" (UID: \"60fd8d72-791a-4e19-9867-6256650aa2da\") " pod="openshift-multus/multus-vddpp" Jul 22 02:58:11.706855 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.706062 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mxfm\" (UniqueName: \"kubernetes.io/projected/60fd8d72-791a-4e19-9867-6256650aa2da-kube-api-access-4mxfm\") pod \"multus-vddpp\" (UID: \"60fd8d72-791a-4e19-9867-6256650aa2da\") " pod="openshift-multus/multus-vddpp" Jul 22 02:58:11.709844 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.709609 2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 22 02:58:11.730339 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.730299 2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-5cptw" Jul 22 02:58:11.738440 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.738411 2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-5cptw" Jul 22 02:58:11.792804 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.792763 2582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 22 02:58:11.806365 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.806247 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a47f1abd-b70b-402b-9cf6-cdff61276fec-host-slash\") pod \"iptables-alerter-7tf82\" (UID: \"a47f1abd-b70b-402b-9cf6-cdff61276fec\") " pod="openshift-network-operator/iptables-alerter-7tf82" Jul 22 02:58:11.806365 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.806294 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5bd7c5db-509d-4f85-82ac-8e357591ae72-original-pull-secret\") pod \"global-pull-secret-syncer-lc5mn\" (UID: \"5bd7c5db-509d-4f85-82ac-8e357591ae72\") " pod="kube-system/global-pull-secret-syncer-lc5mn" Jul 22 02:58:11.806365 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.806322 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh7j6\" (UniqueName: \"kubernetes.io/projected/0dc67dcb-94d3-439a-8b1b-6c238ccc621d-kube-api-access-dh7j6\") pod \"network-check-target-5xv9c\" (UID: \"0dc67dcb-94d3-439a-8b1b-6c238ccc621d\") " pod="openshift-network-diagnostics/network-check-target-5xv9c" Jul 22 02:58:11.806659 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.806377 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prr8p\" (UniqueName: \"kubernetes.io/projected/75610210-6577-4ea8-b77d-a29a12c9a972-kube-api-access-prr8p\") pod \"network-metrics-daemon-jqnjj\" (UID: \"75610210-6577-4ea8-b77d-a29a12c9a972\") " pod="openshift-multus/network-metrics-daemon-jqnjj" Jul 22 02:58:11.806659 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.806417 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a47f1abd-b70b-402b-9cf6-cdff61276fec-host-slash\") pod \"iptables-alerter-7tf82\" (UID: \"a47f1abd-b70b-402b-9cf6-cdff61276fec\") " pod="openshift-network-operator/iptables-alerter-7tf82" Jul 22 02:58:11.806659 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.806475 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/620683de-e754-438d-9d96-3e0cb06a80c6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5rdkb\" (UID: \"620683de-e754-438d-9d96-3e0cb06a80c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rdkb" Jul 22 02:58:11.806659 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.806539 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/620683de-e754-438d-9d96-3e0cb06a80c6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5rdkb\" (UID: \"620683de-e754-438d-9d96-3e0cb06a80c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rdkb" Jul 22 02:58:11.806659 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.806599 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/60fd8d72-791a-4e19-9867-6256650aa2da-cnibin\") pod \"multus-vddpp\" (UID: \"60fd8d72-791a-4e19-9867-6256650aa2da\") " pod="openshift-multus/multus-vddpp" Jul 22 02:58:11.806918 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.806715 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/60fd8d72-791a-4e19-9867-6256650aa2da-multus-daemon-config\") pod \"multus-vddpp\" (UID: \"60fd8d72-791a-4e19-9867-6256650aa2da\") " pod="openshift-multus/multus-vddpp" Jul 22 02:58:11.806918 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.806744 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6ded3042-9b57-4211-af1c-6660ad8363ef-os-release\") pod \"multus-additional-cni-plugins-6hdd2\" (UID: \"6ded3042-9b57-4211-af1c-6660ad8363ef\") " pod="openshift-multus/multus-additional-cni-plugins-6hdd2" Jul 22 02:58:11.806918 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.806801 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/50b3c714-4147-42c8-94fa-5c2bfd912518-etc-tuned\") pod \"tuned-dlq9k\" (UID: \"50b3c714-4147-42c8-94fa-5c2bfd912518\") " pod="openshift-cluster-node-tuning-operator/tuned-dlq9k" Jul 22 02:58:11.806918 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.806900 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/60fd8d72-791a-4e19-9867-6256650aa2da-cnibin\") pod \"multus-vddpp\" (UID: \"60fd8d72-791a-4e19-9867-6256650aa2da\") " pod="openshift-multus/multus-vddpp" Jul 22 02:58:11.807151 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.806928 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c3db93b-85e3-4f29-a0cc-1b6b3d847a94-tmp-dir\") pod \"node-resolver-5gftd\" (UID: \"8c3db93b-85e3-4f29-a0cc-1b6b3d847a94\") " pod="openshift-dns/node-resolver-5gftd" Jul 22 02:58:11.807151 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.806970 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/60fd8d72-791a-4e19-9867-6256650aa2da-os-release\") pod \"multus-vddpp\" (UID: \"60fd8d72-791a-4e19-9867-6256650aa2da\") " pod="openshift-multus/multus-vddpp" Jul 22 02:58:11.807151 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.806995 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6ded3042-9b57-4211-af1c-6660ad8363ef-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6hdd2\" (UID: \"6ded3042-9b57-4211-af1c-6660ad8363ef\") " pod="openshift-multus/multus-additional-cni-plugins-6hdd2" Jul 22 02:58:11.807151 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.807032 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/50b3c714-4147-42c8-94fa-5c2bfd912518-etc-sysctl-d\") pod \"tuned-dlq9k\" (UID: \"50b3c714-4147-42c8-94fa-5c2bfd912518\") " pod="openshift-cluster-node-tuning-operator/tuned-dlq9k" Jul 22 02:58:11.807151 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.807108 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5ee42689-5bff-46fe-8047-adbe6cb34273-host\") pod \"node-ca-vssht\" (UID: \"5ee42689-5bff-46fe-8047-adbe6cb34273\") " pod="openshift-image-registry/node-ca-vssht" Jul 22 02:58:11.807151 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.807133 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/620683de-e754-438d-9d96-3e0cb06a80c6-systemd-units\") pod \"ovnkube-node-5rdkb\" (UID: \"620683de-e754-438d-9d96-3e0cb06a80c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rdkb" Jul 22 02:58:11.807449 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.807146 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6ded3042-9b57-4211-af1c-6660ad8363ef-os-release\") pod \"multus-additional-cni-plugins-6hdd2\" (UID: \"6ded3042-9b57-4211-af1c-6660ad8363ef\") " pod="openshift-multus/multus-additional-cni-plugins-6hdd2" Jul 22 02:58:11.807449 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.807158 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/620683de-e754-438d-9d96-3e0cb06a80c6-host-cni-netd\") pod \"ovnkube-node-5rdkb\" (UID: \"620683de-e754-438d-9d96-3e0cb06a80c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rdkb" Jul 22 02:58:11.807449 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.807167 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/60fd8d72-791a-4e19-9867-6256650aa2da-os-release\") pod \"multus-vddpp\" (UID: \"60fd8d72-791a-4e19-9867-6256650aa2da\") " pod="openshift-multus/multus-vddpp" Jul 22 02:58:11.807449 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.807193 2582 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 22 02:58:11.807449 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.807220 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/620683de-e754-438d-9d96-3e0cb06a80c6-host-cni-netd\") pod \"ovnkube-node-5rdkb\" (UID: \"620683de-e754-438d-9d96-3e0cb06a80c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rdkb" Jul 22 02:58:11.807693 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.807602 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c3db93b-85e3-4f29-a0cc-1b6b3d847a94-tmp-dir\") pod \"node-resolver-5gftd\" (UID: \"8c3db93b-85e3-4f29-a0cc-1b6b3d847a94\") " pod="openshift-dns/node-resolver-5gftd" Jul 22 02:58:11.807693 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.807642 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/50b3c714-4147-42c8-94fa-5c2bfd912518-etc-sysctl-d\") pod \"tuned-dlq9k\" (UID: \"50b3c714-4147-42c8-94fa-5c2bfd912518\") " pod="openshift-cluster-node-tuning-operator/tuned-dlq9k" Jul 22 02:58:11.807693 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.807674 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/620683de-e754-438d-9d96-3e0cb06a80c6-systemd-units\") pod \"ovnkube-node-5rdkb\" (UID: \"620683de-e754-438d-9d96-3e0cb06a80c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rdkb" Jul 22 02:58:11.807857 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.807757 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5ee42689-5bff-46fe-8047-adbe6cb34273-host\") pod \"node-ca-vssht\" (UID: \"5ee42689-5bff-46fe-8047-adbe6cb34273\") " pod="openshift-image-registry/node-ca-vssht" Jul 22 02:58:11.807910 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.807221 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/60fd8d72-791a-4e19-9867-6256650aa2da-cni-binary-copy\") pod \"multus-vddpp\" (UID: \"60fd8d72-791a-4e19-9867-6256650aa2da\") " pod="openshift-multus/multus-vddpp" Jul 22 02:58:11.807965 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.807892 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/60fd8d72-791a-4e19-9867-6256650aa2da-multus-daemon-config\") pod \"multus-vddpp\" (UID: \"60fd8d72-791a-4e19-9867-6256650aa2da\") " pod="openshift-multus/multus-vddpp" Jul 22 02:58:11.807965 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.807920 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/60fd8d72-791a-4e19-9867-6256650aa2da-multus-conf-dir\") pod \"multus-vddpp\" (UID: \"60fd8d72-791a-4e19-9867-6256650aa2da\") " pod="openshift-multus/multus-vddpp" Jul 22 02:58:11.807965 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.807922 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/60fd8d72-791a-4e19-9867-6256650aa2da-cni-binary-copy\") pod \"multus-vddpp\" (UID: \"60fd8d72-791a-4e19-9867-6256650aa2da\") " pod="openshift-multus/multus-vddpp" Jul 22 02:58:11.808155 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.807990 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/60fd8d72-791a-4e19-9867-6256650aa2da-multus-conf-dir\") pod \"multus-vddpp\" (UID: \"60fd8d72-791a-4e19-9867-6256650aa2da\") " pod="openshift-multus/multus-vddpp" Jul 22 02:58:11.808155 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.808026 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5ee42689-5bff-46fe-8047-adbe6cb34273-serviceca\") pod \"node-ca-vssht\" (UID: \"5ee42689-5bff-46fe-8047-adbe6cb34273\") " pod="openshift-image-registry/node-ca-vssht" Jul 22 02:58:11.808155 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.808053 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/620683de-e754-438d-9d96-3e0cb06a80c6-host-run-ovn-kubernetes\") pod \"ovnkube-node-5rdkb\" (UID: \"620683de-e754-438d-9d96-3e0cb06a80c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rdkb" Jul 22 02:58:11.808155 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.808080 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/60fd8d72-791a-4e19-9867-6256650aa2da-host-run-netns\") pod \"multus-vddpp\" (UID: \"60fd8d72-791a-4e19-9867-6256650aa2da\") " pod="openshift-multus/multus-vddpp" Jul 22 02:58:11.808155 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.808119 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mxfm\" (UniqueName: \"kubernetes.io/projected/60fd8d72-791a-4e19-9867-6256650aa2da-kube-api-access-4mxfm\") pod \"multus-vddpp\" (UID: \"60fd8d72-791a-4e19-9867-6256650aa2da\") " pod="openshift-multus/multus-vddpp" Jul 22 02:58:11.808155 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.808143 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/50b3c714-4147-42c8-94fa-5c2bfd912518-etc-kubernetes\") pod \"tuned-dlq9k\" (UID: \"50b3c714-4147-42c8-94fa-5c2bfd912518\") " pod="openshift-cluster-node-tuning-operator/tuned-dlq9k" Jul 22 02:58:11.808473 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.808164 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/620683de-e754-438d-9d96-3e0cb06a80c6-host-slash\") pod \"ovnkube-node-5rdkb\" (UID: \"620683de-e754-438d-9d96-3e0cb06a80c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rdkb" Jul 22 02:58:11.808473 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.808186 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/620683de-e754-438d-9d96-3e0cb06a80c6-var-lib-openvswitch\") pod \"ovnkube-node-5rdkb\" (UID: \"620683de-e754-438d-9d96-3e0cb06a80c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rdkb" Jul 22 02:58:11.808473 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.808187 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/60fd8d72-791a-4e19-9867-6256650aa2da-host-run-netns\") pod \"multus-vddpp\" (UID: \"60fd8d72-791a-4e19-9867-6256650aa2da\") " pod="openshift-multus/multus-vddpp" Jul 22 02:58:11.808473 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.808208 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/620683de-e754-438d-9d96-3e0cb06a80c6-log-socket\") pod \"ovnkube-node-5rdkb\" (UID: \"620683de-e754-438d-9d96-3e0cb06a80c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rdkb" Jul 22 02:58:11.808473 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.808234 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6ded3042-9b57-4211-af1c-6660ad8363ef-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6hdd2\" (UID: \"6ded3042-9b57-4211-af1c-6660ad8363ef\") " pod="openshift-multus/multus-additional-cni-plugins-6hdd2" Jul 22 02:58:11.808473 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.808230 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/620683de-e754-438d-9d96-3e0cb06a80c6-ovn-node-metrics-cert\") pod \"ovnkube-node-5rdkb\" (UID: \"620683de-e754-438d-9d96-3e0cb06a80c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rdkb" Jul 22 02:58:11.808473 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.808240 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/620683de-e754-438d-9d96-3e0cb06a80c6-host-run-ovn-kubernetes\") pod \"ovnkube-node-5rdkb\" (UID: \"620683de-e754-438d-9d96-3e0cb06a80c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rdkb" Jul 22 02:58:11.808473 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.808299 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60fd8d72-791a-4e19-9867-6256650aa2da-host-var-lib-kubelet\") pod \"multus-vddpp\" (UID: \"60fd8d72-791a-4e19-9867-6256650aa2da\") " pod="openshift-multus/multus-vddpp" Jul 22 02:58:11.808473 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.808320 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/620683de-e754-438d-9d96-3e0cb06a80c6-host-slash\") pod \"ovnkube-node-5rdkb\" (UID: \"620683de-e754-438d-9d96-3e0cb06a80c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rdkb" Jul 22 02:58:11.808473 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.808339 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/50b3c714-4147-42c8-94fa-5c2bfd912518-etc-sysconfig\") pod \"tuned-dlq9k\" (UID: \"50b3c714-4147-42c8-94fa-5c2bfd912518\") " pod="openshift-cluster-node-tuning-operator/tuned-dlq9k" Jul 22 02:58:11.808473 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.808367 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/620683de-e754-438d-9d96-3e0cb06a80c6-log-socket\") pod \"ovnkube-node-5rdkb\" (UID: \"620683de-e754-438d-9d96-3e0cb06a80c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rdkb" Jul 22 02:58:11.808473 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.808375 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60fd8d72-791a-4e19-9867-6256650aa2da-etc-kubernetes\") pod \"multus-vddpp\" (UID: \"60fd8d72-791a-4e19-9867-6256650aa2da\") " pod="openshift-multus/multus-vddpp" Jul 22 02:58:11.808473 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.808433 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/50b3c714-4147-42c8-94fa-5c2bfd912518-etc-kubernetes\") pod \"tuned-dlq9k\" (UID: \"50b3c714-4147-42c8-94fa-5c2bfd912518\") " pod="openshift-cluster-node-tuning-operator/tuned-dlq9k" Jul 22 02:58:11.808473 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.808443 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/620683de-e754-438d-9d96-3e0cb06a80c6-var-lib-openvswitch\") pod \"ovnkube-node-5rdkb\" (UID: \"620683de-e754-438d-9d96-3e0cb06a80c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rdkb" Jul 22 02:58:11.808473 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.808472 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60fd8d72-791a-4e19-9867-6256650aa2da-host-var-lib-kubelet\") pod \"multus-vddpp\" (UID: \"60fd8d72-791a-4e19-9867-6256650aa2da\") " pod="openshift-multus/multus-vddpp" Jul 22 02:58:11.809327 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.808480 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60fd8d72-791a-4e19-9867-6256650aa2da-etc-kubernetes\") pod \"multus-vddpp\" (UID: \"60fd8d72-791a-4e19-9867-6256650aa2da\") " pod="openshift-multus/multus-vddpp" Jul 22 02:58:11.809327 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.808527 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a0d8c32d-7c06-494e-9b06-af164d19eb20-socket-dir\") pod \"aws-ebs-csi-driver-node-wcq24\" (UID: \"a0d8c32d-7c06-494e-9b06-af164d19eb20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wcq24" Jul 22 02:58:11.809327 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.808571 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5bd7c5db-509d-4f85-82ac-8e357591ae72-dbus\") pod \"global-pull-secret-syncer-lc5mn\" (UID: \"5bd7c5db-509d-4f85-82ac-8e357591ae72\") " pod="kube-system/global-pull-secret-syncer-lc5mn" Jul 22 02:58:11.809327 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.808569 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/50b3c714-4147-42c8-94fa-5c2bfd912518-etc-sysconfig\") pod \"tuned-dlq9k\" (UID: \"50b3c714-4147-42c8-94fa-5c2bfd912518\") " pod="openshift-cluster-node-tuning-operator/tuned-dlq9k" Jul 22 02:58:11.809327 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.808611 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/50b3c714-4147-42c8-94fa-5c2bfd912518-etc-systemd\") pod \"tuned-dlq9k\" (UID: \"50b3c714-4147-42c8-94fa-5c2bfd912518\") " pod="openshift-cluster-node-tuning-operator/tuned-dlq9k" Jul 22 02:58:11.809327 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.808654 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/60fd8d72-791a-4e19-9867-6256650aa2da-host-var-lib-cni-multus\") pod \"multus-vddpp\" (UID: \"60fd8d72-791a-4e19-9867-6256650aa2da\") " pod="openshift-multus/multus-vddpp" Jul 22 02:58:11.809327 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.808680 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a0d8c32d-7c06-494e-9b06-af164d19eb20-socket-dir\") pod \"aws-ebs-csi-driver-node-wcq24\" (UID: \"a0d8c32d-7c06-494e-9b06-af164d19eb20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wcq24" Jul 22 02:58:11.809327 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.808738 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/50b3c714-4147-42c8-94fa-5c2bfd912518-etc-systemd\") pod \"tuned-dlq9k\" (UID: \"50b3c714-4147-42c8-94fa-5c2bfd912518\") " pod="openshift-cluster-node-tuning-operator/tuned-dlq9k" Jul 22 02:58:11.809327 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.808747 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/60fd8d72-791a-4e19-9867-6256650aa2da-host-var-lib-cni-multus\") pod \"multus-vddpp\" (UID: \"60fd8d72-791a-4e19-9867-6256650aa2da\") " pod="openshift-multus/multus-vddpp" Jul 22 02:58:11.809327 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.808792 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/60fd8d72-791a-4e19-9867-6256650aa2da-host-run-multus-certs\") pod \"multus-vddpp\" (UID: \"60fd8d72-791a-4e19-9867-6256650aa2da\") " pod="openshift-multus/multus-vddpp" Jul 22 02:58:11.809327 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.808819 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a0d8c32d-7c06-494e-9b06-af164d19eb20-device-dir\") pod \"aws-ebs-csi-driver-node-wcq24\" (UID: \"a0d8c32d-7c06-494e-9b06-af164d19eb20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wcq24" Jul 22 02:58:11.809327 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.808833 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/60fd8d72-791a-4e19-9867-6256650aa2da-host-run-multus-certs\") pod \"multus-vddpp\" (UID: \"60fd8d72-791a-4e19-9867-6256650aa2da\") " pod="openshift-multus/multus-vddpp" Jul 22 02:58:11.809327 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.808849 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5bd7c5db-509d-4f85-82ac-8e357591ae72-kubelet-config\") pod \"global-pull-secret-syncer-lc5mn\" (UID: \"5bd7c5db-509d-4f85-82ac-8e357591ae72\") " pod="kube-system/global-pull-secret-syncer-lc5mn" Jul 22 02:58:11.809327 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.808889 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/50b3c714-4147-42c8-94fa-5c2bfd912518-host\") pod \"tuned-dlq9k\" (UID: \"50b3c714-4147-42c8-94fa-5c2bfd912518\") " pod="openshift-cluster-node-tuning-operator/tuned-dlq9k" Jul 22 02:58:11.810026 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.808891 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a0d8c32d-7c06-494e-9b06-af164d19eb20-device-dir\") pod \"aws-ebs-csi-driver-node-wcq24\" (UID: \"a0d8c32d-7c06-494e-9b06-af164d19eb20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wcq24" Jul 22 02:58:11.810026 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.808935 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/620683de-e754-438d-9d96-3e0cb06a80c6-run-ovn\") pod \"ovnkube-node-5rdkb\" (UID: \"620683de-e754-438d-9d96-3e0cb06a80c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rdkb" Jul 22 02:58:11.810026 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.808946 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/50b3c714-4147-42c8-94fa-5c2bfd912518-host\") pod \"tuned-dlq9k\" (UID: \"50b3c714-4147-42c8-94fa-5c2bfd912518\") " pod="openshift-cluster-node-tuning-operator/tuned-dlq9k" Jul 22 02:58:11.810026 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.808977 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/620683de-e754-438d-9d96-3e0cb06a80c6-host-kubelet\") pod \"ovnkube-node-5rdkb\" (UID: \"620683de-e754-438d-9d96-3e0cb06a80c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rdkb" Jul 22 02:58:11.810026 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.809003 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/620683de-e754-438d-9d96-3e0cb06a80c6-host-run-netns\") pod \"ovnkube-node-5rdkb\" (UID: \"620683de-e754-438d-9d96-3e0cb06a80c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rdkb" Jul 22 02:58:11.810026 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.808979 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/620683de-e754-438d-9d96-3e0cb06a80c6-run-ovn\") pod \"ovnkube-node-5rdkb\" (UID: \"620683de-e754-438d-9d96-3e0cb06a80c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rdkb" Jul 22 02:58:11.810026 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.809029 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/620683de-e754-438d-9d96-3e0cb06a80c6-env-overrides\") pod \"ovnkube-node-5rdkb\" (UID: \"620683de-e754-438d-9d96-3e0cb06a80c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rdkb" Jul 22 02:58:11.810026 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.809056 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a0d8c32d-7c06-494e-9b06-af164d19eb20-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wcq24\" (UID: \"a0d8c32d-7c06-494e-9b06-af164d19eb20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wcq24" Jul 22 02:58:11.810026 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.809068 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5ee42689-5bff-46fe-8047-adbe6cb34273-serviceca\") pod \"node-ca-vssht\" (UID: \"5ee42689-5bff-46fe-8047-adbe6cb34273\") " pod="openshift-image-registry/node-ca-vssht" Jul 22 02:58:11.810026 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.809083 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6ded3042-9b57-4211-af1c-6660ad8363ef-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6hdd2\" (UID: \"6ded3042-9b57-4211-af1c-6660ad8363ef\") " pod="openshift-multus/multus-additional-cni-plugins-6hdd2" Jul 22 02:58:11.810026 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.809128 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/620683de-e754-438d-9d96-3e0cb06a80c6-host-run-netns\") pod \"ovnkube-node-5rdkb\" (UID: \"620683de-e754-438d-9d96-3e0cb06a80c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rdkb" Jul 22 02:58:11.810026 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.809071 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/620683de-e754-438d-9d96-3e0cb06a80c6-host-kubelet\") pod \"ovnkube-node-5rdkb\" (UID: \"620683de-e754-438d-9d96-3e0cb06a80c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rdkb" Jul 22 02:58:11.810026 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.809141 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6ded3042-9b57-4211-af1c-6660ad8363ef-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6hdd2\" (UID: \"6ded3042-9b57-4211-af1c-6660ad8363ef\") " pod="openshift-multus/multus-additional-cni-plugins-6hdd2" Jul 22 02:58:11.810026 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.809138 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a0d8c32d-7c06-494e-9b06-af164d19eb20-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wcq24\" (UID: \"a0d8c32d-7c06-494e-9b06-af164d19eb20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wcq24" Jul 22 02:58:11.810791 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.809252 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6ded3042-9b57-4211-af1c-6660ad8363ef-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6hdd2\" (UID: \"6ded3042-9b57-4211-af1c-6660ad8363ef\") " pod="openshift-multus/multus-additional-cni-plugins-6hdd2" Jul 22 02:58:11.810791 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.809265 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e33233aa-5e20-483c-8bb0-0cd5a577da98-agent-certs\") pod \"konnectivity-agent-28hhd\" (UID: \"e33233aa-5e20-483c-8bb0-0cd5a577da98\") " pod="kube-system/konnectivity-agent-28hhd" Jul 22 02:58:11.810791 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.809294 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/50b3c714-4147-42c8-94fa-5c2bfd912518-etc-sysctl-conf\") pod \"tuned-dlq9k\" (UID: \"50b3c714-4147-42c8-94fa-5c2bfd912518\") " pod="openshift-cluster-node-tuning-operator/tuned-dlq9k" Jul 22 02:58:11.810791 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.809319 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/50b3c714-4147-42c8-94fa-5c2bfd912518-lib-modules\") pod \"tuned-dlq9k\" (UID: \"50b3c714-4147-42c8-94fa-5c2bfd912518\") " pod="openshift-cluster-node-tuning-operator/tuned-dlq9k" Jul 22 02:58:11.810791 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.809346 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a47f1abd-b70b-402b-9cf6-cdff61276fec-iptables-alerter-script\") pod \"iptables-alerter-7tf82\" (UID: \"a47f1abd-b70b-402b-9cf6-cdff61276fec\") " pod="openshift-network-operator/iptables-alerter-7tf82" Jul 22 02:58:11.810791 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.809370 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50b3c714-4147-42c8-94fa-5c2bfd912518-tmp\") pod \"tuned-dlq9k\" (UID: \"50b3c714-4147-42c8-94fa-5c2bfd912518\") " pod="openshift-cluster-node-tuning-operator/tuned-dlq9k" Jul 22 02:58:11.810791 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.809395 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60fd8d72-791a-4e19-9867-6256650aa2da-system-cni-dir\") pod \"multus-vddpp\" (UID: \"60fd8d72-791a-4e19-9867-6256650aa2da\") " pod="openshift-multus/multus-vddpp" Jul 22 02:58:11.810791 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.809420 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/60fd8d72-791a-4e19-9867-6256650aa2da-multus-socket-dir-parent\") pod \"multus-vddpp\" (UID: \"60fd8d72-791a-4e19-9867-6256650aa2da\") " pod="openshift-multus/multus-vddpp" Jul 22 02:58:11.810791 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.809450 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/60fd8d72-791a-4e19-9867-6256650aa2da-hostroot\") pod \"multus-vddpp\" (UID: \"60fd8d72-791a-4e19-9867-6256650aa2da\") " pod="openshift-multus/multus-vddpp" Jul 22 02:58:11.810791 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.809474 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a0d8c32d-7c06-494e-9b06-af164d19eb20-sys-fs\") pod \"aws-ebs-csi-driver-node-wcq24\" (UID: \"a0d8c32d-7c06-494e-9b06-af164d19eb20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wcq24" Jul 22 02:58:11.810791 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.809495 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/50b3c714-4147-42c8-94fa-5c2bfd912518-lib-modules\") pod \"tuned-dlq9k\" (UID: \"50b3c714-4147-42c8-94fa-5c2bfd912518\") " pod="openshift-cluster-node-tuning-operator/tuned-dlq9k" Jul 22 02:58:11.810791 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.809502 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rflwk\" (UniqueName: \"kubernetes.io/projected/a0d8c32d-7c06-494e-9b06-af164d19eb20-kube-api-access-rflwk\") pod \"aws-ebs-csi-driver-node-wcq24\" (UID: \"a0d8c32d-7c06-494e-9b06-af164d19eb20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wcq24" Jul 22 02:58:11.810791 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.809533 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e33233aa-5e20-483c-8bb0-0cd5a577da98-konnectivity-ca\") pod \"konnectivity-agent-28hhd\" (UID: \"e33233aa-5e20-483c-8bb0-0cd5a577da98\") " pod="kube-system/konnectivity-agent-28hhd" Jul 22 02:58:11.810791 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.809620 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/620683de-e754-438d-9d96-3e0cb06a80c6-env-overrides\") pod \"ovnkube-node-5rdkb\" (UID: \"620683de-e754-438d-9d96-3e0cb06a80c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rdkb" Jul 22 02:58:11.810791 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.809693 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60fd8d72-791a-4e19-9867-6256650aa2da-system-cni-dir\") pod \"multus-vddpp\" (UID: \"60fd8d72-791a-4e19-9867-6256650aa2da\") " pod="openshift-multus/multus-vddpp" Jul 22 02:58:11.811560 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.809708 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6ded3042-9b57-4211-af1c-6660ad8363ef-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6hdd2\" (UID: \"6ded3042-9b57-4211-af1c-6660ad8363ef\") " pod="openshift-multus/multus-additional-cni-plugins-6hdd2" Jul 22 02:58:11.811560 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.809737 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/50b3c714-4147-42c8-94fa-5c2bfd912518-etc-modprobe-d\") pod \"tuned-dlq9k\" (UID: \"50b3c714-4147-42c8-94fa-5c2bfd912518\") " pod="openshift-cluster-node-tuning-operator/tuned-dlq9k" Jul 22 02:58:11.811560 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.809789 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/60fd8d72-791a-4e19-9867-6256650aa2da-host-run-k8s-cni-cncf-io\") pod \"multus-vddpp\" (UID: \"60fd8d72-791a-4e19-9867-6256650aa2da\") " pod="openshift-multus/multus-vddpp" Jul 22 02:58:11.811560 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.809814 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/60fd8d72-791a-4e19-9867-6256650aa2da-multus-socket-dir-parent\") pod \"multus-vddpp\" (UID: \"60fd8d72-791a-4e19-9867-6256650aa2da\") " pod="openshift-multus/multus-vddpp" Jul 22 02:58:11.811560 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.809817 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/60fd8d72-791a-4e19-9867-6256650aa2da-host-var-lib-cni-bin\") pod \"multus-vddpp\" (UID: \"60fd8d72-791a-4e19-9867-6256650aa2da\") " pod="openshift-multus/multus-vddpp" Jul 22 02:58:11.811560 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.809849 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb8qj\" (UniqueName: \"kubernetes.io/projected/6ded3042-9b57-4211-af1c-6660ad8363ef-kube-api-access-hb8qj\") pod \"multus-additional-cni-plugins-6hdd2\" (UID: \"6ded3042-9b57-4211-af1c-6660ad8363ef\") " pod="openshift-multus/multus-additional-cni-plugins-6hdd2" Jul 22 02:58:11.811560 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.809853 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/60fd8d72-791a-4e19-9867-6256650aa2da-host-var-lib-cni-bin\") pod \"multus-vddpp\" (UID: \"60fd8d72-791a-4e19-9867-6256650aa2da\") " pod="openshift-multus/multus-vddpp" Jul 22 02:58:11.811560 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.809899 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9zqv\" (UniqueName: \"kubernetes.io/projected/8c3db93b-85e3-4f29-a0cc-1b6b3d847a94-kube-api-access-j9zqv\") pod \"node-resolver-5gftd\" (UID: \"8c3db93b-85e3-4f29-a0cc-1b6b3d847a94\") " pod="openshift-dns/node-resolver-5gftd" Jul 22 02:58:11.811560 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.809940 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knw87\" (UniqueName: \"kubernetes.io/projected/5ee42689-5bff-46fe-8047-adbe6cb34273-kube-api-access-knw87\") pod \"node-ca-vssht\" (UID: \"5ee42689-5bff-46fe-8047-adbe6cb34273\") " pod="openshift-image-registry/node-ca-vssht" Jul 22 02:58:11.811560 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.809958 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/50b3c714-4147-42c8-94fa-5c2bfd912518-etc-modprobe-d\") pod \"tuned-dlq9k\" (UID: \"50b3c714-4147-42c8-94fa-5c2bfd912518\") " pod="openshift-cluster-node-tuning-operator/tuned-dlq9k" Jul 22 02:58:11.811560 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.809985 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/60fd8d72-791a-4e19-9867-6256650aa2da-hostroot\") pod \"multus-vddpp\" (UID: \"60fd8d72-791a-4e19-9867-6256650aa2da\") " pod="openshift-multus/multus-vddpp" Jul 22 02:58:11.811560 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.809997 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/620683de-e754-438d-9d96-3e0cb06a80c6-run-systemd\") pod \"ovnkube-node-5rdkb\" (UID: \"620683de-e754-438d-9d96-3e0cb06a80c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rdkb" Jul 22 02:58:11.811560 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.810134 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/50b3c714-4147-42c8-94fa-5c2bfd912518-etc-sysctl-conf\") pod \"tuned-dlq9k\" (UID: \"50b3c714-4147-42c8-94fa-5c2bfd912518\") " pod="openshift-cluster-node-tuning-operator/tuned-dlq9k" Jul 22 02:58:11.811560 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.810200 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a0d8c32d-7c06-494e-9b06-af164d19eb20-sys-fs\") pod \"aws-ebs-csi-driver-node-wcq24\" (UID: \"a0d8c32d-7c06-494e-9b06-af164d19eb20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wcq24" Jul 22 02:58:11.811560 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.810333 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/60fd8d72-791a-4e19-9867-6256650aa2da-host-run-k8s-cni-cncf-io\") pod \"multus-vddpp\" (UID: \"60fd8d72-791a-4e19-9867-6256650aa2da\") " pod="openshift-multus/multus-vddpp" Jul 22 02:58:11.812379 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.810340 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/620683de-e754-438d-9d96-3e0cb06a80c6-host-cni-bin\") pod \"ovnkube-node-5rdkb\" (UID: \"620683de-e754-438d-9d96-3e0cb06a80c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rdkb" Jul 22 02:58:11.812379 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.810376 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/620683de-e754-438d-9d96-3e0cb06a80c6-ovnkube-script-lib\") pod \"ovnkube-node-5rdkb\" (UID: \"620683de-e754-438d-9d96-3e0cb06a80c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rdkb" Jul 22 02:58:11.812379 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.810385 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/620683de-e754-438d-9d96-3e0cb06a80c6-run-systemd\") pod \"ovnkube-node-5rdkb\" (UID: \"620683de-e754-438d-9d96-3e0cb06a80c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rdkb" Jul 22 02:58:11.812379 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.810405 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkz59\" (UniqueName: \"kubernetes.io/projected/620683de-e754-438d-9d96-3e0cb06a80c6-kube-api-access-qkz59\") pod \"ovnkube-node-5rdkb\" (UID: \"620683de-e754-438d-9d96-3e0cb06a80c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rdkb" Jul 22 02:58:11.812379 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.810423 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/620683de-e754-438d-9d96-3e0cb06a80c6-host-cni-bin\") pod \"ovnkube-node-5rdkb\" (UID: \"620683de-e754-438d-9d96-3e0cb06a80c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rdkb" Jul 22 02:58:11.812379 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.810436 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a0d8c32d-7c06-494e-9b06-af164d19eb20-registration-dir\") pod \"aws-ebs-csi-driver-node-wcq24\" (UID: \"a0d8c32d-7c06-494e-9b06-af164d19eb20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wcq24" Jul 22 02:58:11.812379 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.810463 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6ded3042-9b57-4211-af1c-6660ad8363ef-system-cni-dir\") pod \"multus-additional-cni-plugins-6hdd2\" (UID: \"6ded3042-9b57-4211-af1c-6660ad8363ef\") " pod="openshift-multus/multus-additional-cni-plugins-6hdd2" Jul 22 02:58:11.812379 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.810490 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/50b3c714-4147-42c8-94fa-5c2bfd912518-run\") pod \"tuned-dlq9k\" (UID: \"50b3c714-4147-42c8-94fa-5c2bfd912518\") " pod="openshift-cluster-node-tuning-operator/tuned-dlq9k" Jul 22 02:58:11.812379 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.810513 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/50b3c714-4147-42c8-94fa-5c2bfd912518-sys\") pod \"tuned-dlq9k\" (UID: \"50b3c714-4147-42c8-94fa-5c2bfd912518\") " pod="openshift-cluster-node-tuning-operator/tuned-dlq9k" Jul 22 02:58:11.812379 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.810539 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8c3db93b-85e3-4f29-a0cc-1b6b3d847a94-hosts-file\") pod \"node-resolver-5gftd\" (UID: \"8c3db93b-85e3-4f29-a0cc-1b6b3d847a94\") " pod="openshift-dns/node-resolver-5gftd" Jul 22 02:58:11.812379 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.810563 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/620683de-e754-438d-9d96-3e0cb06a80c6-etc-openvswitch\") pod \"ovnkube-node-5rdkb\" (UID: \"620683de-e754-438d-9d96-3e0cb06a80c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rdkb" Jul 22 02:58:11.812379 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.810588 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a47f1abd-b70b-402b-9cf6-cdff61276fec-iptables-alerter-script\") pod \"iptables-alerter-7tf82\" (UID: \"a47f1abd-b70b-402b-9cf6-cdff61276fec\") " pod="openshift-network-operator/iptables-alerter-7tf82" Jul 22 02:58:11.812379 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.810624 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/620683de-e754-438d-9d96-3e0cb06a80c6-run-openvswitch\") pod \"ovnkube-node-5rdkb\" (UID: \"620683de-e754-438d-9d96-3e0cb06a80c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rdkb" Jul 22 02:58:11.812379 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.810633 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/50b3c714-4147-42c8-94fa-5c2bfd912518-run\") pod \"tuned-dlq9k\" (UID: \"50b3c714-4147-42c8-94fa-5c2bfd912518\") " pod="openshift-cluster-node-tuning-operator/tuned-dlq9k" Jul 22 02:58:11.812379 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.810657 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60fd8d72-791a-4e19-9867-6256650aa2da-multus-cni-dir\") pod \"multus-vddpp\" (UID: \"60fd8d72-791a-4e19-9867-6256650aa2da\") " pod="openshift-multus/multus-vddpp" Jul 22 02:58:11.813152 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.810685 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8fh5\" (UniqueName: \"kubernetes.io/projected/a47f1abd-b70b-402b-9cf6-cdff61276fec-kube-api-access-c8fh5\") pod \"iptables-alerter-7tf82\" (UID: \"a47f1abd-b70b-402b-9cf6-cdff61276fec\") " pod="openshift-network-operator/iptables-alerter-7tf82" Jul 22 02:58:11.813152 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.810718 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6ded3042-9b57-4211-af1c-6660ad8363ef-cnibin\") pod \"multus-additional-cni-plugins-6hdd2\" (UID: \"6ded3042-9b57-4211-af1c-6660ad8363ef\") " pod="openshift-multus/multus-additional-cni-plugins-6hdd2" Jul 22 02:58:11.813152 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.810685 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/50b3c714-4147-42c8-94fa-5c2bfd912518-sys\") pod \"tuned-dlq9k\" (UID: \"50b3c714-4147-42c8-94fa-5c2bfd912518\") " pod="openshift-cluster-node-tuning-operator/tuned-dlq9k" Jul 22 02:58:11.813152 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.810744 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75610210-6577-4ea8-b77d-a29a12c9a972-metrics-certs\") pod \"network-metrics-daemon-jqnjj\" (UID: \"75610210-6577-4ea8-b77d-a29a12c9a972\") " pod="openshift-multus/network-metrics-daemon-jqnjj" Jul 22 02:58:11.813152 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.810774 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6ded3042-9b57-4211-af1c-6660ad8363ef-cni-binary-copy\") pod \"multus-additional-cni-plugins-6hdd2\" (UID: \"6ded3042-9b57-4211-af1c-6660ad8363ef\") " pod="openshift-multus/multus-additional-cni-plugins-6hdd2" Jul 22 02:58:11.813152 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.810765 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e33233aa-5e20-483c-8bb0-0cd5a577da98-konnectivity-ca\") pod \"konnectivity-agent-28hhd\" (UID: \"e33233aa-5e20-483c-8bb0-0cd5a577da98\") " pod="kube-system/konnectivity-agent-28hhd" Jul 22 02:58:11.813152 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.810790 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a0d8c32d-7c06-494e-9b06-af164d19eb20-registration-dir\") pod \"aws-ebs-csi-driver-node-wcq24\" (UID: \"a0d8c32d-7c06-494e-9b06-af164d19eb20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wcq24" Jul 22 02:58:11.813152 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.810800 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/50b3c714-4147-42c8-94fa-5c2bfd912518-var-lib-kubelet\") pod \"tuned-dlq9k\" (UID: \"50b3c714-4147-42c8-94fa-5c2bfd912518\") " pod="openshift-cluster-node-tuning-operator/tuned-dlq9k" Jul 22 02:58:11.813152 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.810826 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkdjn\" (UniqueName: \"kubernetes.io/projected/50b3c714-4147-42c8-94fa-5c2bfd912518-kube-api-access-gkdjn\") pod \"tuned-dlq9k\" (UID: \"50b3c714-4147-42c8-94fa-5c2bfd912518\") " pod="openshift-cluster-node-tuning-operator/tuned-dlq9k" Jul 22 02:58:11.813152 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.810872 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6ded3042-9b57-4211-af1c-6660ad8363ef-system-cni-dir\") pod \"multus-additional-cni-plugins-6hdd2\" (UID: \"6ded3042-9b57-4211-af1c-6660ad8363ef\") " pod="openshift-multus/multus-additional-cni-plugins-6hdd2" Jul 22 02:58:11.813152 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.810907 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/620683de-e754-438d-9d96-3e0cb06a80c6-ovnkube-script-lib\") pod \"ovnkube-node-5rdkb\" (UID: \"620683de-e754-438d-9d96-3e0cb06a80c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rdkb" Jul 22 02:58:11.813152 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.810909 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/620683de-e754-438d-9d96-3e0cb06a80c6-node-log\") pod \"ovnkube-node-5rdkb\" (UID: \"620683de-e754-438d-9d96-3e0cb06a80c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rdkb" Jul 22 02:58:11.813152 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.810945 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/620683de-e754-438d-9d96-3e0cb06a80c6-node-log\") pod \"ovnkube-node-5rdkb\" (UID: \"620683de-e754-438d-9d96-3e0cb06a80c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rdkb" Jul 22 02:58:11.813152 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.810965 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/620683de-e754-438d-9d96-3e0cb06a80c6-ovnkube-config\") pod \"ovnkube-node-5rdkb\" (UID: \"620683de-e754-438d-9d96-3e0cb06a80c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rdkb" Jul 22 02:58:11.813867 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.810967 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6ded3042-9b57-4211-af1c-6660ad8363ef-cnibin\") pod \"multus-additional-cni-plugins-6hdd2\" (UID: \"6ded3042-9b57-4211-af1c-6660ad8363ef\") " pod="openshift-multus/multus-additional-cni-plugins-6hdd2" Jul 22 02:58:11.813867 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.810928 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60fd8d72-791a-4e19-9867-6256650aa2da-multus-cni-dir\") pod \"multus-vddpp\" (UID: \"60fd8d72-791a-4e19-9867-6256650aa2da\") " pod="openshift-multus/multus-vddpp" Jul 22 02:58:11.813867 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.811012 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/620683de-e754-438d-9d96-3e0cb06a80c6-etc-openvswitch\") pod \"ovnkube-node-5rdkb\" (UID: \"620683de-e754-438d-9d96-3e0cb06a80c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rdkb" Jul 22 02:58:11.813867 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.811013 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8c3db93b-85e3-4f29-a0cc-1b6b3d847a94-hosts-file\") pod \"node-resolver-5gftd\" (UID: \"8c3db93b-85e3-4f29-a0cc-1b6b3d847a94\") " pod="openshift-dns/node-resolver-5gftd" Jul 22 02:58:11.813867 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.811066 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/620683de-e754-438d-9d96-3e0cb06a80c6-run-openvswitch\") pod \"ovnkube-node-5rdkb\" (UID: \"620683de-e754-438d-9d96-3e0cb06a80c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rdkb" Jul 22 02:58:11.813867 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.811080 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/50b3c714-4147-42c8-94fa-5c2bfd912518-var-lib-kubelet\") pod \"tuned-dlq9k\" (UID: \"50b3c714-4147-42c8-94fa-5c2bfd912518\") " pod="openshift-cluster-node-tuning-operator/tuned-dlq9k" Jul 22 02:58:11.813867 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:11.811123 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 02:58:11.813867 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.811124 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a0d8c32d-7c06-494e-9b06-af164d19eb20-etc-selinux\") pod \"aws-ebs-csi-driver-node-wcq24\" (UID: \"a0d8c32d-7c06-494e-9b06-af164d19eb20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wcq24" Jul 22 02:58:11.813867 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:11.811238 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75610210-6577-4ea8-b77d-a29a12c9a972-metrics-certs podName:75610210-6577-4ea8-b77d-a29a12c9a972 nodeName:}" failed. No retries permitted until 2026-07-22 02:58:12.31119635 +0000 UTC m=+3.042817192 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/75610210-6577-4ea8-b77d-a29a12c9a972-metrics-certs") pod "network-metrics-daemon-jqnjj" (UID: "75610210-6577-4ea8-b77d-a29a12c9a972") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 02:58:11.813867 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.811240 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a0d8c32d-7c06-494e-9b06-af164d19eb20-etc-selinux\") pod \"aws-ebs-csi-driver-node-wcq24\" (UID: \"a0d8c32d-7c06-494e-9b06-af164d19eb20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wcq24" Jul 22 02:58:11.813867 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.811500 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/620683de-e754-438d-9d96-3e0cb06a80c6-ovnkube-config\") pod \"ovnkube-node-5rdkb\" (UID: \"620683de-e754-438d-9d96-3e0cb06a80c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rdkb" Jul 22 02:58:11.813867 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.811630 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6ded3042-9b57-4211-af1c-6660ad8363ef-cni-binary-copy\") pod \"multus-additional-cni-plugins-6hdd2\" (UID: \"6ded3042-9b57-4211-af1c-6660ad8363ef\") " pod="openshift-multus/multus-additional-cni-plugins-6hdd2" Jul 22 02:58:11.813867 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.812040 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/50b3c714-4147-42c8-94fa-5c2bfd912518-etc-tuned\") pod \"tuned-dlq9k\" (UID: \"50b3c714-4147-42c8-94fa-5c2bfd912518\") " pod="openshift-cluster-node-tuning-operator/tuned-dlq9k" Jul 22 02:58:11.813867 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.812507 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/620683de-e754-438d-9d96-3e0cb06a80c6-ovn-node-metrics-cert\") pod \"ovnkube-node-5rdkb\" (UID: \"620683de-e754-438d-9d96-3e0cb06a80c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rdkb" Jul 22 02:58:11.817523 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.815159 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50b3c714-4147-42c8-94fa-5c2bfd912518-tmp\") pod \"tuned-dlq9k\" (UID: \"50b3c714-4147-42c8-94fa-5c2bfd912518\") " pod="openshift-cluster-node-tuning-operator/tuned-dlq9k" Jul 22 02:58:11.817523 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.816237 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e33233aa-5e20-483c-8bb0-0cd5a577da98-agent-certs\") pod \"konnectivity-agent-28hhd\" (UID: \"e33233aa-5e20-483c-8bb0-0cd5a577da98\") " pod="kube-system/konnectivity-agent-28hhd" Jul 22 02:58:11.822655 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.822605 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prr8p\" (UniqueName: \"kubernetes.io/projected/75610210-6577-4ea8-b77d-a29a12c9a972-kube-api-access-prr8p\") pod \"network-metrics-daemon-jqnjj\" (UID: \"75610210-6577-4ea8-b77d-a29a12c9a972\") " pod="openshift-multus/network-metrics-daemon-jqnjj" Jul 22 02:58:11.827020 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.826990 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-knw87\" (UniqueName: \"kubernetes.io/projected/5ee42689-5bff-46fe-8047-adbe6cb34273-kube-api-access-knw87\") pod \"node-ca-vssht\" (UID: \"5ee42689-5bff-46fe-8047-adbe6cb34273\") " pod="openshift-image-registry/node-ca-vssht" Jul 22 02:58:11.828325 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.828298 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mxfm\" (UniqueName: \"kubernetes.io/projected/60fd8d72-791a-4e19-9867-6256650aa2da-kube-api-access-4mxfm\") pod \"multus-vddpp\" (UID: \"60fd8d72-791a-4e19-9867-6256650aa2da\") " pod="openshift-multus/multus-vddpp" Jul 22 02:58:11.829670 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.829619 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkz59\" (UniqueName: \"kubernetes.io/projected/620683de-e754-438d-9d96-3e0cb06a80c6-kube-api-access-qkz59\") pod \"ovnkube-node-5rdkb\" (UID: \"620683de-e754-438d-9d96-3e0cb06a80c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rdkb" Jul 22 02:58:11.829913 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.829865 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rflwk\" (UniqueName: \"kubernetes.io/projected/a0d8c32d-7c06-494e-9b06-af164d19eb20-kube-api-access-rflwk\") pod \"aws-ebs-csi-driver-node-wcq24\" (UID: \"a0d8c32d-7c06-494e-9b06-af164d19eb20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wcq24" Jul 22 02:58:11.830149 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.830122 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkdjn\" (UniqueName: \"kubernetes.io/projected/50b3c714-4147-42c8-94fa-5c2bfd912518-kube-api-access-gkdjn\") pod \"tuned-dlq9k\" (UID: \"50b3c714-4147-42c8-94fa-5c2bfd912518\") " pod="openshift-cluster-node-tuning-operator/tuned-dlq9k" Jul 22 02:58:11.830664 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.830633 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9zqv\" (UniqueName: \"kubernetes.io/projected/8c3db93b-85e3-4f29-a0cc-1b6b3d847a94-kube-api-access-j9zqv\") pod \"node-resolver-5gftd\" (UID: \"8c3db93b-85e3-4f29-a0cc-1b6b3d847a94\") " pod="openshift-dns/node-resolver-5gftd" Jul 22 02:58:11.832372 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.832341 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb8qj\" (UniqueName: \"kubernetes.io/projected/6ded3042-9b57-4211-af1c-6660ad8363ef-kube-api-access-hb8qj\") pod \"multus-additional-cni-plugins-6hdd2\" (UID: \"6ded3042-9b57-4211-af1c-6660ad8363ef\") " pod="openshift-multus/multus-additional-cni-plugins-6hdd2" Jul 22 02:58:11.832769 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.832744 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8fh5\" (UniqueName: \"kubernetes.io/projected/a47f1abd-b70b-402b-9cf6-cdff61276fec-kube-api-access-c8fh5\") pod \"iptables-alerter-7tf82\" (UID: \"a47f1abd-b70b-402b-9cf6-cdff61276fec\") " pod="openshift-network-operator/iptables-alerter-7tf82" Jul 22 02:58:11.899713 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.899622 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-67.ec2.internal" event={"ID":"5d8b9a289a92f6d831f68731b2b2419a","Type":"ContainerStarted","Data":"21a673823e1513bd0d112fdde45265c897e6e0966e4c8d55a63acbd2b7425bfb"} Jul 22 02:58:11.900722 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.900692 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-67.ec2.internal" event={"ID":"7f651843bca4c08d31d08118e2ebefbe","Type":"ContainerStarted","Data":"b2d1ec54e410809c9dbcaa0b72f86676ca77359c9119e75ed3bddaeef8c712bb"} Jul 22 02:58:11.912262 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.912233 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5bd7c5db-509d-4f85-82ac-8e357591ae72-dbus\") pod \"global-pull-secret-syncer-lc5mn\" (UID: \"5bd7c5db-509d-4f85-82ac-8e357591ae72\") " pod="kube-system/global-pull-secret-syncer-lc5mn" Jul 22 02:58:11.912262 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.912273 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5bd7c5db-509d-4f85-82ac-8e357591ae72-kubelet-config\") pod \"global-pull-secret-syncer-lc5mn\" (UID: \"5bd7c5db-509d-4f85-82ac-8e357591ae72\") " pod="kube-system/global-pull-secret-syncer-lc5mn" Jul 22 02:58:11.912475 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.912341 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5bd7c5db-509d-4f85-82ac-8e357591ae72-original-pull-secret\") pod \"global-pull-secret-syncer-lc5mn\" (UID: \"5bd7c5db-509d-4f85-82ac-8e357591ae72\") " pod="kube-system/global-pull-secret-syncer-lc5mn" Jul 22 02:58:11.912475 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.912377 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh7j6\" (UniqueName: \"kubernetes.io/projected/0dc67dcb-94d3-439a-8b1b-6c238ccc621d-kube-api-access-dh7j6\") pod \"network-check-target-5xv9c\" (UID: \"0dc67dcb-94d3-439a-8b1b-6c238ccc621d\") " pod="openshift-network-diagnostics/network-check-target-5xv9c" Jul 22 02:58:11.912475 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.912410 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5bd7c5db-509d-4f85-82ac-8e357591ae72-dbus\") pod \"global-pull-secret-syncer-lc5mn\" (UID: \"5bd7c5db-509d-4f85-82ac-8e357591ae72\") " pod="kube-system/global-pull-secret-syncer-lc5mn" Jul 22 02:58:11.912475 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.912458 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5bd7c5db-509d-4f85-82ac-8e357591ae72-kubelet-config\") pod \"global-pull-secret-syncer-lc5mn\" (UID: \"5bd7c5db-509d-4f85-82ac-8e357591ae72\") " pod="kube-system/global-pull-secret-syncer-lc5mn" Jul 22 02:58:11.912680 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:11.912509 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 02:58:11.912680 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:11.912591 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bd7c5db-509d-4f85-82ac-8e357591ae72-original-pull-secret podName:5bd7c5db-509d-4f85-82ac-8e357591ae72 nodeName:}" failed. No retries permitted until 2026-07-22 02:58:12.412567379 +0000 UTC m=+3.144188223 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5bd7c5db-509d-4f85-82ac-8e357591ae72-original-pull-secret") pod "global-pull-secret-syncer-lc5mn" (UID: "5bd7c5db-509d-4f85-82ac-8e357591ae72") : object "kube-system"/"original-pull-secret" not registered Jul 22 02:58:11.921287 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:11.921265 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 02:58:11.921287 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:11.921290 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 02:58:11.921504 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:11.921303 2582 projected.go:196] Error preparing data for projected volume kube-api-access-dh7j6 for pod openshift-network-diagnostics/network-check-target-5xv9c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 02:58:11.921504 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:11.921371 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0dc67dcb-94d3-439a-8b1b-6c238ccc621d-kube-api-access-dh7j6 podName:0dc67dcb-94d3-439a-8b1b-6c238ccc621d nodeName:}" failed. No retries permitted until 2026-07-22 02:58:12.421351571 +0000 UTC m=+3.152972431 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-dh7j6" (UniqueName: "kubernetes.io/projected/0dc67dcb-94d3-439a-8b1b-6c238ccc621d-kube-api-access-dh7j6") pod "network-check-target-5xv9c" (UID: "0dc67dcb-94d3-439a-8b1b-6c238ccc621d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 02:58:11.991639 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:11.991603 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vddpp" Jul 22 02:58:12.006097 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:12.006063 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wcq24" Jul 22 02:58:12.016260 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:12.016232 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6hdd2" Jul 22 02:58:12.022407 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:12.022379 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-28hhd" Jul 22 02:58:12.031121 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:12.031094 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dlq9k" Jul 22 02:58:12.041894 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:12.041863 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7tf82" Jul 22 02:58:12.048892 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:12.048870 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5gftd" Jul 22 02:58:12.050374 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:12.050355 2582 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 02:58:12.057305 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:12.057285 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vssht" Jul 22 02:58:12.064315 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:12.064292 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5rdkb" Jul 22 02:58:12.316333 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:12.316260 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75610210-6577-4ea8-b77d-a29a12c9a972-metrics-certs\") pod \"network-metrics-daemon-jqnjj\" (UID: \"75610210-6577-4ea8-b77d-a29a12c9a972\") " pod="openshift-multus/network-metrics-daemon-jqnjj" Jul 22 02:58:12.316480 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:12.316429 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 02:58:12.316518 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:12.316504 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75610210-6577-4ea8-b77d-a29a12c9a972-metrics-certs podName:75610210-6577-4ea8-b77d-a29a12c9a972 nodeName:}" failed. No retries permitted until 2026-07-22 02:58:13.316485799 +0000 UTC m=+4.048106625 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/75610210-6577-4ea8-b77d-a29a12c9a972-metrics-certs") pod "network-metrics-daemon-jqnjj" (UID: "75610210-6577-4ea8-b77d-a29a12c9a972") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 02:58:12.417258 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:12.417191 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5bd7c5db-509d-4f85-82ac-8e357591ae72-original-pull-secret\") pod \"global-pull-secret-syncer-lc5mn\" (UID: \"5bd7c5db-509d-4f85-82ac-8e357591ae72\") " pod="kube-system/global-pull-secret-syncer-lc5mn" Jul 22 02:58:12.417458 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:12.417383 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 02:58:12.417517 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:12.417471 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bd7c5db-509d-4f85-82ac-8e357591ae72-original-pull-secret podName:5bd7c5db-509d-4f85-82ac-8e357591ae72 nodeName:}" failed. No retries permitted until 2026-07-22 02:58:13.417449408 +0000 UTC m=+4.149070234 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5bd7c5db-509d-4f85-82ac-8e357591ae72-original-pull-secret") pod "global-pull-secret-syncer-lc5mn" (UID: "5bd7c5db-509d-4f85-82ac-8e357591ae72") : object "kube-system"/"original-pull-secret" not registered Jul 22 02:58:12.517851 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:12.517803 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh7j6\" (UniqueName: \"kubernetes.io/projected/0dc67dcb-94d3-439a-8b1b-6c238ccc621d-kube-api-access-dh7j6\") pod \"network-check-target-5xv9c\" (UID: \"0dc67dcb-94d3-439a-8b1b-6c238ccc621d\") " pod="openshift-network-diagnostics/network-check-target-5xv9c" Jul 22 02:58:12.518041 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:12.518006 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 02:58:12.518111 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:12.518050 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 02:58:12.518111 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:12.518062 2582 projected.go:196] Error preparing data for projected volume kube-api-access-dh7j6 for pod openshift-network-diagnostics/network-check-target-5xv9c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 02:58:12.518208 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:12.518127 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0dc67dcb-94d3-439a-8b1b-6c238ccc621d-kube-api-access-dh7j6 podName:0dc67dcb-94d3-439a-8b1b-6c238ccc621d nodeName:}" failed. No retries permitted until 2026-07-22 02:58:13.518109264 +0000 UTC m=+4.249730093 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-dh7j6" (UniqueName: "kubernetes.io/projected/0dc67dcb-94d3-439a-8b1b-6c238ccc621d-kube-api-access-dh7j6") pod "network-check-target-5xv9c" (UID: "0dc67dcb-94d3-439a-8b1b-6c238ccc621d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 02:58:12.741482 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:12.741385 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-21 02:53:12 +0000 UTC" deadline="2027-12-20 08:02:48.25795724 +0000 UTC" Jul 22 02:58:12.741482 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:12.741419 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12389h4m35.516542092s" Jul 22 02:58:12.895547 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:12.895516 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jqnjj" Jul 22 02:58:12.895710 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:12.895518 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lc5mn" Jul 22 02:58:12.895710 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:12.895644 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jqnjj" podUID="75610210-6577-4ea8-b77d-a29a12c9a972" Jul 22 02:58:12.895787 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:12.895751 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lc5mn" podUID="5bd7c5db-509d-4f85-82ac-8e357591ae72" Jul 22 02:58:12.929319 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:12.929285 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod620683de_e754_438d_9d96_3e0cb06a80c6.slice/crio-025cd33a46e0f7eb2590fb7039209b3fccbb8b2b5941d3658f20a43d7a8ca396 WatchSource:0}: Error finding container 025cd33a46e0f7eb2590fb7039209b3fccbb8b2b5941d3658f20a43d7a8ca396: Status 404 returned error can't find the container with id 025cd33a46e0f7eb2590fb7039209b3fccbb8b2b5941d3658f20a43d7a8ca396 Jul 22 02:58:12.932912 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:12.932882 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50b3c714_4147_42c8_94fa_5c2bfd912518.slice/crio-026a1afd1472b30a9536d418a669c5142e8d4499b0f2b208a0f8d265fb89812f WatchSource:0}: Error finding container 026a1afd1472b30a9536d418a669c5142e8d4499b0f2b208a0f8d265fb89812f: Status 404 returned error can't find the container with id 026a1afd1472b30a9536d418a669c5142e8d4499b0f2b208a0f8d265fb89812f Jul 22 02:58:13.196888 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:13.196855 2582 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 02:58:13.323800 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:13.323683 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75610210-6577-4ea8-b77d-a29a12c9a972-metrics-certs\") pod \"network-metrics-daemon-jqnjj\" (UID: \"75610210-6577-4ea8-b77d-a29a12c9a972\") " pod="openshift-multus/network-metrics-daemon-jqnjj" Jul 22 02:58:13.323980 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:13.323844 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 02:58:13.323980 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:13.323916 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75610210-6577-4ea8-b77d-a29a12c9a972-metrics-certs podName:75610210-6577-4ea8-b77d-a29a12c9a972 nodeName:}" failed. No retries permitted until 2026-07-22 02:58:15.323891832 +0000 UTC m=+6.055512675 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/75610210-6577-4ea8-b77d-a29a12c9a972-metrics-certs") pod "network-metrics-daemon-jqnjj" (UID: "75610210-6577-4ea8-b77d-a29a12c9a972") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 02:58:13.424795 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:13.424750 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5bd7c5db-509d-4f85-82ac-8e357591ae72-original-pull-secret\") pod \"global-pull-secret-syncer-lc5mn\" (UID: \"5bd7c5db-509d-4f85-82ac-8e357591ae72\") " pod="kube-system/global-pull-secret-syncer-lc5mn" Jul 22 02:58:13.425006 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:13.424967 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 02:58:13.425104 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:13.425030 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bd7c5db-509d-4f85-82ac-8e357591ae72-original-pull-secret podName:5bd7c5db-509d-4f85-82ac-8e357591ae72 nodeName:}" failed. No retries permitted until 2026-07-22 02:58:15.425010936 +0000 UTC m=+6.156631771 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5bd7c5db-509d-4f85-82ac-8e357591ae72-original-pull-secret") pod "global-pull-secret-syncer-lc5mn" (UID: "5bd7c5db-509d-4f85-82ac-8e357591ae72") : object "kube-system"/"original-pull-secret" not registered Jul 22 02:58:13.525912 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:13.525845 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh7j6\" (UniqueName: \"kubernetes.io/projected/0dc67dcb-94d3-439a-8b1b-6c238ccc621d-kube-api-access-dh7j6\") pod \"network-check-target-5xv9c\" (UID: \"0dc67dcb-94d3-439a-8b1b-6c238ccc621d\") " pod="openshift-network-diagnostics/network-check-target-5xv9c" Jul 22 02:58:13.526111 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:13.526099 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 02:58:13.526191 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:13.526125 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 02:58:13.526191 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:13.526140 2582 projected.go:196] Error preparing data for projected volume kube-api-access-dh7j6 for pod openshift-network-diagnostics/network-check-target-5xv9c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 02:58:13.526302 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:13.526202 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0dc67dcb-94d3-439a-8b1b-6c238ccc621d-kube-api-access-dh7j6 podName:0dc67dcb-94d3-439a-8b1b-6c238ccc621d nodeName:}" failed. No retries permitted until 2026-07-22 02:58:15.52618245 +0000 UTC m=+6.257803278 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-dh7j6" (UniqueName: "kubernetes.io/projected/0dc67dcb-94d3-439a-8b1b-6c238ccc621d-kube-api-access-dh7j6") pod "network-check-target-5xv9c" (UID: "0dc67dcb-94d3-439a-8b1b-6c238ccc621d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 02:58:13.742419 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:13.742345 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-21 02:53:12 +0000 UTC" deadline="2028-01-06 01:42:03.630875836 +0000 UTC" Jul 22 02:58:13.742419 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:13.742416 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12790h43m49.888464739s" Jul 22 02:58:13.900510 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:13.900473 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5xv9c" Jul 22 02:58:13.900692 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:13.900615 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5xv9c" podUID="0dc67dcb-94d3-439a-8b1b-6c238ccc621d" Jul 22 02:58:13.909114 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:13.909004 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6hdd2" event={"ID":"6ded3042-9b57-4211-af1c-6660ad8363ef","Type":"ContainerStarted","Data":"eb62e629b1818bc32726dbbb5455155818760c9a38128ae5591699a6e7613298"} Jul 22 02:58:13.912685 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:13.912646 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7tf82" event={"ID":"a47f1abd-b70b-402b-9cf6-cdff61276fec","Type":"ContainerStarted","Data":"b5780fadc674a23e00ccad3e9986fc7dbe26fc98620885b23fb2caf8e326cf4d"} Jul 22 02:58:13.920565 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:13.920522 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dlq9k" event={"ID":"50b3c714-4147-42c8-94fa-5c2bfd912518","Type":"ContainerStarted","Data":"026a1afd1472b30a9536d418a669c5142e8d4499b0f2b208a0f8d265fb89812f"} Jul 22 02:58:13.924110 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:13.923977 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5rdkb" event={"ID":"620683de-e754-438d-9d96-3e0cb06a80c6","Type":"ContainerStarted","Data":"025cd33a46e0f7eb2590fb7039209b3fccbb8b2b5941d3658f20a43d7a8ca396"} Jul 22 02:58:13.936222 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:13.936107 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-67.ec2.internal" event={"ID":"5d8b9a289a92f6d831f68731b2b2419a","Type":"ContainerStarted","Data":"d8f4413d6e8956323fa01ea93baabe82a383f19b6e189b4b81698bf845cba98c"} Jul 22 02:58:13.952173 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:13.950675 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vddpp" event={"ID":"60fd8d72-791a-4e19-9867-6256650aa2da","Type":"ContainerStarted","Data":"f840e70d3b0521a6d8806b590a4ff517d2004a9922d1d6ad0cb4603860f7482f"} Jul 22 02:58:13.963133 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:13.963080 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wcq24" event={"ID":"a0d8c32d-7c06-494e-9b06-af164d19eb20","Type":"ContainerStarted","Data":"6c0b477ef7865e6a649d8360b98ea3fc3d9a1321c500ffa4143bd0c5a9c0772c"} Jul 22 02:58:13.983461 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:13.983297 2582 generic.go:356] "Generic (PLEG): container finished" podID="7f651843bca4c08d31d08118e2ebefbe" containerID="f2c6e64e4801cf6c5119581afddddede8cda5157274aec65fd331d9704346b12" exitCode=0 Jul 22 02:58:13.983461 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:13.983397 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-67.ec2.internal" event={"ID":"7f651843bca4c08d31d08118e2ebefbe","Type":"ContainerDied","Data":"f2c6e64e4801cf6c5119581afddddede8cda5157274aec65fd331d9704346b12"} Jul 22 02:58:13.990166 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:13.990083 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5gftd" event={"ID":"8c3db93b-85e3-4f29-a0cc-1b6b3d847a94","Type":"ContainerStarted","Data":"4196645e897ad75f7c0934dd382f28180a1f2431e89993d93226c4e3f2f6885f"} Jul 22 02:58:13.995355 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:13.995323 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-28hhd" event={"ID":"e33233aa-5e20-483c-8bb0-0cd5a577da98","Type":"ContainerStarted","Data":"0df082359f8b3e5c5266cc39f8095eb3b4cc0c61dcf58238c65ae2c5c300cec2"} Jul 22 02:58:14.001116 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:14.000800 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-67.ec2.internal" podStartSLOduration=3.000779587 podStartE2EDuration="3.000779587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 02:58:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:58:13.961347615 +0000 UTC m=+4.692968468" watchObservedRunningTime="2026-07-22 02:58:14.000779587 +0000 UTC m=+4.732400440" Jul 22 02:58:14.011849 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:14.011757 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vssht" event={"ID":"5ee42689-5bff-46fe-8047-adbe6cb34273","Type":"ContainerStarted","Data":"d6343bcfc1c1b5680fc3ada623c7786d72e2c13835be168793997c9e3763d6cd"} Jul 22 02:58:14.895748 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:14.895712 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jqnjj" Jul 22 02:58:14.896265 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:14.895870 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jqnjj" podUID="75610210-6577-4ea8-b77d-a29a12c9a972" Jul 22 02:58:14.896355 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:14.896321 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lc5mn" Jul 22 02:58:14.896459 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:14.896435 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lc5mn" podUID="5bd7c5db-509d-4f85-82ac-8e357591ae72" Jul 22 02:58:15.047023 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:15.046989 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-67.ec2.internal" event={"ID":"7f651843bca4c08d31d08118e2ebefbe","Type":"ContainerStarted","Data":"855b95e3d03a01b0c0e8146128e9870526de6a56a300186822901dc0b11f31d6"} Jul 22 02:58:15.351051 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:15.350366 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75610210-6577-4ea8-b77d-a29a12c9a972-metrics-certs\") pod \"network-metrics-daemon-jqnjj\" (UID: \"75610210-6577-4ea8-b77d-a29a12c9a972\") " pod="openshift-multus/network-metrics-daemon-jqnjj" Jul 22 02:58:15.351051 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:15.350516 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 02:58:15.351051 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:15.350587 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75610210-6577-4ea8-b77d-a29a12c9a972-metrics-certs podName:75610210-6577-4ea8-b77d-a29a12c9a972 nodeName:}" failed. No retries permitted until 2026-07-22 02:58:19.350565807 +0000 UTC m=+10.082186638 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/75610210-6577-4ea8-b77d-a29a12c9a972-metrics-certs") pod "network-metrics-daemon-jqnjj" (UID: "75610210-6577-4ea8-b77d-a29a12c9a972") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 02:58:15.452197 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:15.451437 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5bd7c5db-509d-4f85-82ac-8e357591ae72-original-pull-secret\") pod \"global-pull-secret-syncer-lc5mn\" (UID: \"5bd7c5db-509d-4f85-82ac-8e357591ae72\") " pod="kube-system/global-pull-secret-syncer-lc5mn" Jul 22 02:58:15.452197 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:15.451720 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 02:58:15.452197 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:15.451795 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bd7c5db-509d-4f85-82ac-8e357591ae72-original-pull-secret podName:5bd7c5db-509d-4f85-82ac-8e357591ae72 nodeName:}" failed. No retries permitted until 2026-07-22 02:58:19.451767006 +0000 UTC m=+10.183387833 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5bd7c5db-509d-4f85-82ac-8e357591ae72-original-pull-secret") pod "global-pull-secret-syncer-lc5mn" (UID: "5bd7c5db-509d-4f85-82ac-8e357591ae72") : object "kube-system"/"original-pull-secret" not registered Jul 22 02:58:15.552991 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:15.552947 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh7j6\" (UniqueName: \"kubernetes.io/projected/0dc67dcb-94d3-439a-8b1b-6c238ccc621d-kube-api-access-dh7j6\") pod \"network-check-target-5xv9c\" (UID: \"0dc67dcb-94d3-439a-8b1b-6c238ccc621d\") " pod="openshift-network-diagnostics/network-check-target-5xv9c" Jul 22 02:58:15.553246 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:15.553219 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 02:58:15.553246 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:15.553242 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 02:58:15.553385 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:15.553255 2582 projected.go:196] Error preparing data for projected volume kube-api-access-dh7j6 for pod openshift-network-diagnostics/network-check-target-5xv9c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 02:58:15.553385 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:15.553321 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0dc67dcb-94d3-439a-8b1b-6c238ccc621d-kube-api-access-dh7j6 podName:0dc67dcb-94d3-439a-8b1b-6c238ccc621d nodeName:}" failed. No retries permitted until 2026-07-22 02:58:19.553299856 +0000 UTC m=+10.284920704 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-dh7j6" (UniqueName: "kubernetes.io/projected/0dc67dcb-94d3-439a-8b1b-6c238ccc621d-kube-api-access-dh7j6") pod "network-check-target-5xv9c" (UID: "0dc67dcb-94d3-439a-8b1b-6c238ccc621d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 02:58:15.897521 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:15.897482 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5xv9c" Jul 22 02:58:15.898058 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:15.897620 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5xv9c" podUID="0dc67dcb-94d3-439a-8b1b-6c238ccc621d" Jul 22 02:58:16.895579 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:16.895540 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jqnjj" Jul 22 02:58:16.895780 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:16.895699 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jqnjj" podUID="75610210-6577-4ea8-b77d-a29a12c9a972" Jul 22 02:58:16.896230 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:16.896207 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lc5mn" Jul 22 02:58:16.896438 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:16.896358 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lc5mn" podUID="5bd7c5db-509d-4f85-82ac-8e357591ae72" Jul 22 02:58:17.895861 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:17.895217 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5xv9c" Jul 22 02:58:17.895861 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:17.895375 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5xv9c" podUID="0dc67dcb-94d3-439a-8b1b-6c238ccc621d" Jul 22 02:58:18.895525 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:18.895317 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jqnjj" Jul 22 02:58:18.895525 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:18.895483 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jqnjj" podUID="75610210-6577-4ea8-b77d-a29a12c9a972" Jul 22 02:58:18.895938 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:18.895912 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lc5mn" Jul 22 02:58:18.896353 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:18.896039 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lc5mn" podUID="5bd7c5db-509d-4f85-82ac-8e357591ae72" Jul 22 02:58:19.388313 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:19.388226 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75610210-6577-4ea8-b77d-a29a12c9a972-metrics-certs\") pod \"network-metrics-daemon-jqnjj\" (UID: \"75610210-6577-4ea8-b77d-a29a12c9a972\") " pod="openshift-multus/network-metrics-daemon-jqnjj" Jul 22 02:58:19.388482 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:19.388413 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 02:58:19.388539 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:19.388490 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75610210-6577-4ea8-b77d-a29a12c9a972-metrics-certs podName:75610210-6577-4ea8-b77d-a29a12c9a972 nodeName:}" failed. No retries permitted until 2026-07-22 02:58:27.38846833 +0000 UTC m=+18.120089172 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/75610210-6577-4ea8-b77d-a29a12c9a972-metrics-certs") pod "network-metrics-daemon-jqnjj" (UID: "75610210-6577-4ea8-b77d-a29a12c9a972") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 02:58:19.489054 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:19.488998 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5bd7c5db-509d-4f85-82ac-8e357591ae72-original-pull-secret\") pod \"global-pull-secret-syncer-lc5mn\" (UID: \"5bd7c5db-509d-4f85-82ac-8e357591ae72\") " pod="kube-system/global-pull-secret-syncer-lc5mn" Jul 22 02:58:19.489222 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:19.489144 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 02:58:19.489283 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:19.489225 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bd7c5db-509d-4f85-82ac-8e357591ae72-original-pull-secret podName:5bd7c5db-509d-4f85-82ac-8e357591ae72 nodeName:}" failed. No retries permitted until 2026-07-22 02:58:27.489201644 +0000 UTC m=+18.220822471 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5bd7c5db-509d-4f85-82ac-8e357591ae72-original-pull-secret") pod "global-pull-secret-syncer-lc5mn" (UID: "5bd7c5db-509d-4f85-82ac-8e357591ae72") : object "kube-system"/"original-pull-secret" not registered Jul 22 02:58:19.589734 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:19.589695 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh7j6\" (UniqueName: \"kubernetes.io/projected/0dc67dcb-94d3-439a-8b1b-6c238ccc621d-kube-api-access-dh7j6\") pod \"network-check-target-5xv9c\" (UID: \"0dc67dcb-94d3-439a-8b1b-6c238ccc621d\") " pod="openshift-network-diagnostics/network-check-target-5xv9c" Jul 22 02:58:19.589919 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:19.589841 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 02:58:19.589919 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:19.589856 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 02:58:19.589919 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:19.589866 2582 projected.go:196] Error preparing data for projected volume kube-api-access-dh7j6 for pod openshift-network-diagnostics/network-check-target-5xv9c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 02:58:19.589919 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:19.589913 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0dc67dcb-94d3-439a-8b1b-6c238ccc621d-kube-api-access-dh7j6 podName:0dc67dcb-94d3-439a-8b1b-6c238ccc621d nodeName:}" failed. No retries permitted until 2026-07-22 02:58:27.589900458 +0000 UTC m=+18.321521283 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-dh7j6" (UniqueName: "kubernetes.io/projected/0dc67dcb-94d3-439a-8b1b-6c238ccc621d-kube-api-access-dh7j6") pod "network-check-target-5xv9c" (UID: "0dc67dcb-94d3-439a-8b1b-6c238ccc621d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 02:58:19.896501 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:19.896048 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5xv9c" Jul 22 02:58:19.896501 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:19.896164 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5xv9c" podUID="0dc67dcb-94d3-439a-8b1b-6c238ccc621d" Jul 22 02:58:20.895980 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:20.895945 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jqnjj" Jul 22 02:58:20.896186 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:20.896101 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jqnjj" podUID="75610210-6577-4ea8-b77d-a29a12c9a972" Jul 22 02:58:20.896615 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:20.896487 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lc5mn" Jul 22 02:58:20.896615 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:20.896575 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lc5mn" podUID="5bd7c5db-509d-4f85-82ac-8e357591ae72" Jul 22 02:58:21.895936 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:21.895904 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5xv9c" Jul 22 02:58:21.896132 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:21.896047 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5xv9c" podUID="0dc67dcb-94d3-439a-8b1b-6c238ccc621d" Jul 22 02:58:22.895327 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:22.895109 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jqnjj" Jul 22 02:58:22.895327 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:22.895157 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lc5mn" Jul 22 02:58:22.895327 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:22.895258 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jqnjj" podUID="75610210-6577-4ea8-b77d-a29a12c9a972" Jul 22 02:58:22.895829 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:22.895680 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lc5mn" podUID="5bd7c5db-509d-4f85-82ac-8e357591ae72" Jul 22 02:58:23.895472 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:23.895439 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5xv9c" Jul 22 02:58:23.895908 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:23.895558 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5xv9c" podUID="0dc67dcb-94d3-439a-8b1b-6c238ccc621d" Jul 22 02:58:24.895109 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:24.895069 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jqnjj" Jul 22 02:58:24.895109 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:24.895100 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lc5mn" Jul 22 02:58:24.895336 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:24.895219 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jqnjj" podUID="75610210-6577-4ea8-b77d-a29a12c9a972" Jul 22 02:58:24.895414 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:24.895339 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lc5mn" podUID="5bd7c5db-509d-4f85-82ac-8e357591ae72" Jul 22 02:58:25.895726 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:25.895687 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5xv9c" Jul 22 02:58:25.896241 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:25.895822 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5xv9c" podUID="0dc67dcb-94d3-439a-8b1b-6c238ccc621d" Jul 22 02:58:26.895318 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:26.895285 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lc5mn" Jul 22 02:58:26.895607 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:26.895293 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jqnjj" Jul 22 02:58:26.895607 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:26.895401 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lc5mn" podUID="5bd7c5db-509d-4f85-82ac-8e357591ae72" Jul 22 02:58:26.895607 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:26.895527 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jqnjj" podUID="75610210-6577-4ea8-b77d-a29a12c9a972" Jul 22 02:58:27.449615 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:27.449581 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75610210-6577-4ea8-b77d-a29a12c9a972-metrics-certs\") pod \"network-metrics-daemon-jqnjj\" (UID: \"75610210-6577-4ea8-b77d-a29a12c9a972\") " pod="openshift-multus/network-metrics-daemon-jqnjj" Jul 22 02:58:27.449834 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:27.449747 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 02:58:27.449834 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:27.449820 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75610210-6577-4ea8-b77d-a29a12c9a972-metrics-certs podName:75610210-6577-4ea8-b77d-a29a12c9a972 nodeName:}" failed. No retries permitted until 2026-07-22 02:58:43.449799634 +0000 UTC m=+34.181420473 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/75610210-6577-4ea8-b77d-a29a12c9a972-metrics-certs") pod "network-metrics-daemon-jqnjj" (UID: "75610210-6577-4ea8-b77d-a29a12c9a972") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 02:58:27.549906 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:27.549875 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5bd7c5db-509d-4f85-82ac-8e357591ae72-original-pull-secret\") pod \"global-pull-secret-syncer-lc5mn\" (UID: \"5bd7c5db-509d-4f85-82ac-8e357591ae72\") " pod="kube-system/global-pull-secret-syncer-lc5mn" Jul 22 02:58:27.550084 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:27.549995 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 02:58:27.550084 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:27.550066 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bd7c5db-509d-4f85-82ac-8e357591ae72-original-pull-secret podName:5bd7c5db-509d-4f85-82ac-8e357591ae72 nodeName:}" failed. No retries permitted until 2026-07-22 02:58:43.5500526 +0000 UTC m=+34.281673428 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5bd7c5db-509d-4f85-82ac-8e357591ae72-original-pull-secret") pod "global-pull-secret-syncer-lc5mn" (UID: "5bd7c5db-509d-4f85-82ac-8e357591ae72") : object "kube-system"/"original-pull-secret" not registered Jul 22 02:58:27.651044 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:27.650949 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh7j6\" (UniqueName: \"kubernetes.io/projected/0dc67dcb-94d3-439a-8b1b-6c238ccc621d-kube-api-access-dh7j6\") pod \"network-check-target-5xv9c\" (UID: \"0dc67dcb-94d3-439a-8b1b-6c238ccc621d\") " pod="openshift-network-diagnostics/network-check-target-5xv9c" Jul 22 02:58:27.651242 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:27.651109 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 02:58:27.651242 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:27.651131 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 02:58:27.651242 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:27.651145 2582 projected.go:196] Error preparing data for projected volume kube-api-access-dh7j6 for pod openshift-network-diagnostics/network-check-target-5xv9c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 02:58:27.651242 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:27.651207 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0dc67dcb-94d3-439a-8b1b-6c238ccc621d-kube-api-access-dh7j6 podName:0dc67dcb-94d3-439a-8b1b-6c238ccc621d nodeName:}" failed. No retries permitted until 2026-07-22 02:58:43.651190627 +0000 UTC m=+34.382811454 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-dh7j6" (UniqueName: "kubernetes.io/projected/0dc67dcb-94d3-439a-8b1b-6c238ccc621d-kube-api-access-dh7j6") pod "network-check-target-5xv9c" (UID: "0dc67dcb-94d3-439a-8b1b-6c238ccc621d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 02:58:27.895645 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:27.895614 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5xv9c" Jul 22 02:58:27.895853 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:27.895735 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5xv9c" podUID="0dc67dcb-94d3-439a-8b1b-6c238ccc621d" Jul 22 02:58:28.895970 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:28.895933 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jqnjj" Jul 22 02:58:28.896419 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:28.895973 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lc5mn" Jul 22 02:58:28.896419 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:28.896072 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jqnjj" podUID="75610210-6577-4ea8-b77d-a29a12c9a972" Jul 22 02:58:28.896419 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:28.896212 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lc5mn" podUID="5bd7c5db-509d-4f85-82ac-8e357591ae72" Jul 22 02:58:29.898752 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:29.898160 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5xv9c" Jul 22 02:58:29.899209 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:29.899111 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5xv9c" podUID="0dc67dcb-94d3-439a-8b1b-6c238ccc621d" Jul 22 02:58:30.895391 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:30.895345 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jqnjj" Jul 22 02:58:30.895391 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:30.895380 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lc5mn" Jul 22 02:58:30.895649 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:30.895489 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jqnjj" podUID="75610210-6577-4ea8-b77d-a29a12c9a972" Jul 22 02:58:30.895649 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:30.895623 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lc5mn" podUID="5bd7c5db-509d-4f85-82ac-8e357591ae72" Jul 22 02:58:31.899241 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:31.898973 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5xv9c" Jul 22 02:58:31.899722 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:31.899258 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5xv9c" podUID="0dc67dcb-94d3-439a-8b1b-6c238ccc621d" Jul 22 02:58:32.089263 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:32.088912 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5gftd" event={"ID":"8c3db93b-85e3-4f29-a0cc-1b6b3d847a94","Type":"ContainerStarted","Data":"fde7895d7fd56e92a603cce9cdd16d40abfc849d3c2d181b610655c526d2f4ef"} Jul 22 02:58:32.090577 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:32.090547 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-28hhd" event={"ID":"e33233aa-5e20-483c-8bb0-0cd5a577da98","Type":"ContainerStarted","Data":"73817069e3ace8da9b1443df690f1c8e582696b6bf7b7ae8466dec0295d6d840"} Jul 22 02:58:32.092248 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:32.092201 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vssht" event={"ID":"5ee42689-5bff-46fe-8047-adbe6cb34273","Type":"ContainerStarted","Data":"0b50e080090865920bcf0cbf0514c2fc3aa3af5774e9980d3442e04b36a0aa98"} Jul 22 02:58:32.093974 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:32.093935 2582 generic.go:356] "Generic (PLEG): container finished" podID="6ded3042-9b57-4211-af1c-6660ad8363ef" containerID="4dcc0d26b1f3acef6f3ed0257bb86deb5a55d0b21c54918d7d507754dd5deb79" exitCode=0 Jul 22 02:58:32.094122 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:32.094029 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6hdd2" event={"ID":"6ded3042-9b57-4211-af1c-6660ad8363ef","Type":"ContainerDied","Data":"4dcc0d26b1f3acef6f3ed0257bb86deb5a55d0b21c54918d7d507754dd5deb79"} Jul 22 02:58:32.095628 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:32.095602 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dlq9k" event={"ID":"50b3c714-4147-42c8-94fa-5c2bfd912518","Type":"ContainerStarted","Data":"1e637edf6246891dc8d3ad1522aac039c980090fb6d30b6fe7c8895bc6e11cec"} Jul 22 02:58:32.097866 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:32.097842 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5rdkb" event={"ID":"620683de-e754-438d-9d96-3e0cb06a80c6","Type":"ContainerStarted","Data":"ccd21b7650fc227bb40c8174bac6f2836c48d54d2964294c5b057b069c2dabd8"} Jul 22 02:58:32.098029 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:32.097914 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5rdkb" event={"ID":"620683de-e754-438d-9d96-3e0cb06a80c6","Type":"ContainerStarted","Data":"9d82a0a6a65583362404d751bdcd779ab813838a84ddc4e89d39eef77041371b"} Jul 22 02:58:32.098029 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:32.097964 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5rdkb" event={"ID":"620683de-e754-438d-9d96-3e0cb06a80c6","Type":"ContainerStarted","Data":"80a40ae70e4ada3995a215ad00c8d81226502de89f0bd3572d60ad0a81ce7360"} Jul 22 02:58:32.099349 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:32.099316 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vddpp" event={"ID":"60fd8d72-791a-4e19-9867-6256650aa2da","Type":"ContainerStarted","Data":"10344057a5a785694b287ae5c0da51aa83ced39fceb9312ae3924ec584e6d445"} Jul 22 02:58:32.100713 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:32.100689 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wcq24" event={"ID":"a0d8c32d-7c06-494e-9b06-af164d19eb20","Type":"ContainerStarted","Data":"37319419e800824cb44734dfa2e9a3aed17419acd6d84f4144407e5bae6daec1"} Jul 22 02:58:32.104026 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:32.103976 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-67.ec2.internal" podStartSLOduration=21.103963805 podStartE2EDuration="21.103963805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 02:58:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:58:15.06366685 +0000 UTC m=+5.795287718" watchObservedRunningTime="2026-07-22 02:58:32.103963805 +0000 UTC m=+22.835584651" Jul 22 02:58:32.104511 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:32.104458 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-5gftd" podStartSLOduration=3.328218405 podStartE2EDuration="22.104445615s" totalImagesPullingTime="18.77622721s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 02:58:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:58:32.103706745 +0000 UTC m=+22.835327616" watchObservedRunningTime="2026-07-22 02:58:32.104445615 +0000 UTC m=+22.836066462" Jul 22 02:58:32.117229 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:32.117185 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-vssht" podStartSLOduration=3.347107442 podStartE2EDuration="22.11716904s" totalImagesPullingTime="18.770061598s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 02:58:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:58:32.116754774 +0000 UTC m=+22.848375622" watchObservedRunningTime="2026-07-22 02:58:32.11716904 +0000 UTC m=+22.848789936" Jul 22 02:58:32.130964 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:32.130919 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-28hhd" podStartSLOduration=7.701709163 podStartE2EDuration="22.130901115s" totalImagesPullingTime="14.429191952s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:58:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:58:32.130744625 +0000 UTC m=+22.862365473" watchObservedRunningTime="2026-07-22 02:58:32.130901115 +0000 UTC m=+22.862521961" Jul 22 02:58:32.147798 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:32.147730 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-vddpp" podStartSLOduration=3.319243155 podStartE2EDuration="22.147713474s" totalImagesPullingTime="18.828470319s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 02:58:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:58:32.147160377 +0000 UTC m=+22.878781236" watchObservedRunningTime="2026-07-22 02:58:32.147713474 +0000 UTC m=+22.879334320" Jul 22 02:58:32.188319 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:32.188273 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-dlq9k" podStartSLOduration=3.406741071 podStartE2EDuration="22.188258096s" totalImagesPullingTime="18.781517025s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 02:58:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:58:32.188097301 +0000 UTC m=+22.919718158" watchObservedRunningTime="2026-07-22 02:58:32.188258096 +0000 UTC m=+22.919878942" Jul 22 02:58:32.295191 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:32.295165 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:58:32.368694 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:32.368667 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:58:32.578386 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:32.578306 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:58:32.641209 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:32.641162 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:58:32.895851 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:32.895760 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lc5mn" Jul 22 02:58:32.896059 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:32.895875 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lc5mn" podUID="5bd7c5db-509d-4f85-82ac-8e357591ae72" Jul 22 02:58:32.896059 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:32.895935 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jqnjj" Jul 22 02:58:32.896167 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:32.896077 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jqnjj" podUID="75610210-6577-4ea8-b77d-a29a12c9a972" Jul 22 02:58:32.983878 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:32.983845 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:58:33.049825 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:33.049793 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:58:33.107188 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:33.107110 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5rdkb" event={"ID":"620683de-e754-438d-9d96-3e0cb06a80c6","Type":"ContainerStarted","Data":"85b0dc7beeb87f6a62296ffaa08a2390e3f5e44ebbcad5698eb93cf7c067780b"} Jul 22 02:58:33.107363 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:33.107203 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5rdkb" event={"ID":"620683de-e754-438d-9d96-3e0cb06a80c6","Type":"ContainerStarted","Data":"55a33a1665968f419933e35a3340f9c92b43c5d3a5d5d101ba9032d273ebf87c"} Jul 22 02:58:33.107363 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:33.107251 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5rdkb" event={"ID":"620683de-e754-438d-9d96-3e0cb06a80c6","Type":"ContainerStarted","Data":"74df2a136a774b46a89df6e7a0046b9d12b7298763d8e4c436d00b854f28dd50"} Jul 22 02:58:33.269085 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:33.269054 2582 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 22 02:58:33.773187 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:33.773044 2582 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-22T02:58:33.269078777Z","UUID":"6c76b750-cfd7-4e6a-a324-85fede1bd644","Handler":null,"Name":"","Endpoint":""} Jul 22 02:58:33.774910 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:33.774887 2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 22 02:58:33.775059 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:33.774941 2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 22 02:58:33.791069 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:33.791033 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:58:33.895245 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:33.895204 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5xv9c" Jul 22 02:58:33.895516 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:33.895328 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5xv9c" podUID="0dc67dcb-94d3-439a-8b1b-6c238ccc621d" Jul 22 02:58:34.111214 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:34.111126 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7tf82" event={"ID":"a47f1abd-b70b-402b-9cf6-cdff61276fec","Type":"ContainerStarted","Data":"68b9aa70d5a87c61f2bbc5050d30d23494c69388e79a1c3706c680556a72e1a8"} Jul 22 02:58:34.113317 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:34.113289 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wcq24" event={"ID":"a0d8c32d-7c06-494e-9b06-af164d19eb20","Type":"ContainerStarted","Data":"de3382af4591d70b7cfb4d445fec23592a315735dc595e9aa341375b1ba2f159"} Jul 22 02:58:34.126139 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:34.126090 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-7tf82" podStartSLOduration=5.348605304 podStartE2EDuration="24.126076507s" totalImagesPullingTime="18.777471203s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 02:58:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:58:34.125916069 +0000 UTC m=+24.857536935" watchObservedRunningTime="2026-07-22 02:58:34.126076507 +0000 UTC m=+24.857697354" Jul 22 02:58:34.195470 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:34.195431 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:58:34.296361 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:34.296329 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:58:34.895881 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:34.895850 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jqnjj" Jul 22 02:58:34.895881 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:34.895882 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lc5mn" Jul 22 02:58:34.896136 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:34.895986 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jqnjj" podUID="75610210-6577-4ea8-b77d-a29a12c9a972" Jul 22 02:58:34.896136 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:34.896111 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lc5mn" podUID="5bd7c5db-509d-4f85-82ac-8e357591ae72" Jul 22 02:58:34.998054 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:34.998003 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:58:35.122586 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:35.122475 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5rdkb" event={"ID":"620683de-e754-438d-9d96-3e0cb06a80c6","Type":"ContainerStarted","Data":"997189cb364cd2fa2249364e015ef9a8bd94a9fcf9bb1cbfd96114839baed2d1"} Jul 22 02:58:35.124867 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:35.124834 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wcq24" event={"ID":"a0d8c32d-7c06-494e-9b06-af164d19eb20","Type":"ContainerStarted","Data":"35dc3bbfbb9416e6ffd291e56607dd3978b77addfd8d2f895102a8e6d2e9db70"} Jul 22 02:58:35.377007 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:35.376926 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:58:35.450207 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:35.450173 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:58:35.895637 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:35.895603 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5xv9c" Jul 22 02:58:35.895834 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:35.895735 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5xv9c" podUID="0dc67dcb-94d3-439a-8b1b-6c238ccc621d" Jul 22 02:58:36.602244 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:36.602078 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:58:36.895342 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:36.895306 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lc5mn" Jul 22 02:58:36.895573 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:36.895310 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jqnjj" Jul 22 02:58:36.895573 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:36.895423 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lc5mn" podUID="5bd7c5db-509d-4f85-82ac-8e357591ae72" Jul 22 02:58:36.895573 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:36.895534 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jqnjj" podUID="75610210-6577-4ea8-b77d-a29a12c9a972" Jul 22 02:58:36.975996 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:36.975914 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-28hhd" Jul 22 02:58:36.976601 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:36.976581 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-28hhd" Jul 22 02:58:36.992416 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:36.992363 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wcq24" podStartSLOduration=6.016110061 podStartE2EDuration="26.99234681s" totalImagesPullingTime="20.976236749s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 02:58:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:58:35.142055153 +0000 UTC m=+25.873676000" watchObservedRunningTime="2026-07-22 02:58:36.99234681 +0000 UTC m=+27.723967693" Jul 22 02:58:37.130940 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:37.130911 2582 generic.go:356] "Generic (PLEG): container finished" podID="6ded3042-9b57-4211-af1c-6660ad8363ef" containerID="568988faadeefd90635d72aa0e5fbfb22e9dca70427f934aaf409365eba47344" exitCode=0 Jul 22 02:58:37.131123 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:37.130998 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6hdd2" event={"ID":"6ded3042-9b57-4211-af1c-6660ad8363ef","Type":"ContainerDied","Data":"568988faadeefd90635d72aa0e5fbfb22e9dca70427f934aaf409365eba47344"} Jul 22 02:58:37.134751 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:37.134723 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5rdkb" event={"ID":"620683de-e754-438d-9d96-3e0cb06a80c6","Type":"ContainerStarted","Data":"4fd7a615a98b0e6a41054c513d9e6969cfbde8a8480cd8ef23cc2850b8b89750"} Jul 22 02:58:37.135116 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:37.134834 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-28hhd" Jul 22 02:58:37.135330 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:37.135313 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-28hhd" Jul 22 02:58:37.194968 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:37.194917 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-5rdkb" podStartSLOduration=8.339028248 podStartE2EDuration="27.194901588s" totalImagesPullingTime="18.85587334s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 02:58:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:58:37.194212492 +0000 UTC m=+27.925833339" watchObservedRunningTime="2026-07-22 02:58:37.194901588 +0000 UTC m=+27.926522434" Jul 22 02:58:37.234110 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:37.234031 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:58:37.896150 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:37.895868 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5xv9c" Jul 22 02:58:37.896699 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:37.896164 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5xv9c" podUID="0dc67dcb-94d3-439a-8b1b-6c238ccc621d" Jul 22 02:58:38.137973 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:38.137941 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5rdkb" Jul 22 02:58:38.138167 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:38.138057 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5rdkb" Jul 22 02:58:38.138167 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:38.138107 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5rdkb" Jul 22 02:58:38.153954 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:38.153856 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5rdkb" Jul 22 02:58:38.155035 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:38.154970 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5rdkb" Jul 22 02:58:38.620465 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:38.620383 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5xv9c"] Jul 22 02:58:38.620654 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:38.620481 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5xv9c" Jul 22 02:58:38.620654 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:38.620559 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5xv9c" podUID="0dc67dcb-94d3-439a-8b1b-6c238ccc621d" Jul 22 02:58:38.626609 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:38.625561 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jqnjj"] Jul 22 02:58:38.626609 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:38.625660 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lc5mn"] Jul 22 02:58:38.626609 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:38.625761 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lc5mn" Jul 22 02:58:38.626609 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:38.625888 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lc5mn" podUID="5bd7c5db-509d-4f85-82ac-8e357591ae72" Jul 22 02:58:38.626609 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:38.625978 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jqnjj" Jul 22 02:58:38.626609 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:38.626119 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jqnjj" podUID="75610210-6577-4ea8-b77d-a29a12c9a972" Jul 22 02:58:39.141256 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:39.141220 2582 generic.go:356] "Generic (PLEG): container finished" podID="6ded3042-9b57-4211-af1c-6660ad8363ef" containerID="94e710c25b605ec60bfc6b4094a28b2e22a008bfb9e03363db8f12ae6349e848" exitCode=0 Jul 22 02:58:39.141788 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:39.141276 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6hdd2" event={"ID":"6ded3042-9b57-4211-af1c-6660ad8363ef","Type":"ContainerDied","Data":"94e710c25b605ec60bfc6b4094a28b2e22a008bfb9e03363db8f12ae6349e848"} Jul 22 02:58:39.896379 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:39.896342 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lc5mn" Jul 22 02:58:39.896573 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:39.896417 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5xv9c" Jul 22 02:58:39.896573 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:39.896442 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lc5mn" podUID="5bd7c5db-509d-4f85-82ac-8e357591ae72" Jul 22 02:58:39.896573 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:39.896499 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5xv9c" podUID="0dc67dcb-94d3-439a-8b1b-6c238ccc621d" Jul 22 02:58:40.895901 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:40.895732 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jqnjj" Jul 22 02:58:40.896336 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:40.896006 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jqnjj" podUID="75610210-6577-4ea8-b77d-a29a12c9a972" Jul 22 02:58:41.010520 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:41.010442 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:58:41.111589 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:41.111560 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:58:41.149189 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:41.149146 2582 generic.go:356] "Generic (PLEG): container finished" podID="6ded3042-9b57-4211-af1c-6660ad8363ef" containerID="f1a8c76d67886b4d873334e163238ec5a680402e8cb777ecf4fb3e452ddf1df7" exitCode=0 Jul 22 02:58:41.149345 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:41.149214 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6hdd2" event={"ID":"6ded3042-9b57-4211-af1c-6660ad8363ef","Type":"ContainerDied","Data":"f1a8c76d67886b4d873334e163238ec5a680402e8cb777ecf4fb3e452ddf1df7"} Jul 22 02:58:41.895417 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:41.895382 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5xv9c" Jul 22 02:58:41.895635 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:41.895388 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lc5mn" Jul 22 02:58:41.895635 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:41.895507 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5xv9c" podUID="0dc67dcb-94d3-439a-8b1b-6c238ccc621d" Jul 22 02:58:41.895635 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:41.895593 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lc5mn" podUID="5bd7c5db-509d-4f85-82ac-8e357591ae72" Jul 22 02:58:42.895813 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:42.895780 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jqnjj" Jul 22 02:58:42.896302 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:42.895903 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jqnjj" podUID="75610210-6577-4ea8-b77d-a29a12c9a972" Jul 22 02:58:43.469794 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:43.469759 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75610210-6577-4ea8-b77d-a29a12c9a972-metrics-certs\") pod \"network-metrics-daemon-jqnjj\" (UID: \"75610210-6577-4ea8-b77d-a29a12c9a972\") " pod="openshift-multus/network-metrics-daemon-jqnjj" Jul 22 02:58:43.469985 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:43.469932 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 02:58:43.470068 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:43.470034 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75610210-6577-4ea8-b77d-a29a12c9a972-metrics-certs podName:75610210-6577-4ea8-b77d-a29a12c9a972 nodeName:}" failed. No retries permitted until 2026-07-22 02:59:15.469989952 +0000 UTC m=+66.201610794 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/75610210-6577-4ea8-b77d-a29a12c9a972-metrics-certs") pod "network-metrics-daemon-jqnjj" (UID: "75610210-6577-4ea8-b77d-a29a12c9a972") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 02:58:43.570965 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:43.570929 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5bd7c5db-509d-4f85-82ac-8e357591ae72-original-pull-secret\") pod \"global-pull-secret-syncer-lc5mn\" (UID: \"5bd7c5db-509d-4f85-82ac-8e357591ae72\") " pod="kube-system/global-pull-secret-syncer-lc5mn" Jul 22 02:58:43.571156 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:43.571064 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 02:58:43.571156 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:43.571118 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bd7c5db-509d-4f85-82ac-8e357591ae72-original-pull-secret podName:5bd7c5db-509d-4f85-82ac-8e357591ae72 nodeName:}" failed. No retries permitted until 2026-07-22 02:59:15.571104319 +0000 UTC m=+66.302725147 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5bd7c5db-509d-4f85-82ac-8e357591ae72-original-pull-secret") pod "global-pull-secret-syncer-lc5mn" (UID: "5bd7c5db-509d-4f85-82ac-8e357591ae72") : object "kube-system"/"original-pull-secret" not registered Jul 22 02:58:43.671694 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:43.671654 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh7j6\" (UniqueName: \"kubernetes.io/projected/0dc67dcb-94d3-439a-8b1b-6c238ccc621d-kube-api-access-dh7j6\") pod \"network-check-target-5xv9c\" (UID: \"0dc67dcb-94d3-439a-8b1b-6c238ccc621d\") " pod="openshift-network-diagnostics/network-check-target-5xv9c" Jul 22 02:58:43.671889 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:43.671845 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 02:58:43.671889 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:43.671870 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 02:58:43.671889 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:43.671883 2582 projected.go:196] Error preparing data for projected volume kube-api-access-dh7j6 for pod openshift-network-diagnostics/network-check-target-5xv9c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 02:58:43.672065 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:43.671950 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0dc67dcb-94d3-439a-8b1b-6c238ccc621d-kube-api-access-dh7j6 podName:0dc67dcb-94d3-439a-8b1b-6c238ccc621d nodeName:}" failed. No retries permitted until 2026-07-22 02:59:15.671931577 +0000 UTC m=+66.403552422 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-dh7j6" (UniqueName: "kubernetes.io/projected/0dc67dcb-94d3-439a-8b1b-6c238ccc621d-kube-api-access-dh7j6") pod "network-check-target-5xv9c" (UID: "0dc67dcb-94d3-439a-8b1b-6c238ccc621d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 02:58:43.895060 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:43.894969 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lc5mn" Jul 22 02:58:43.895060 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:43.895002 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5xv9c" Jul 22 02:58:43.895313 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:43.895123 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lc5mn" podUID="5bd7c5db-509d-4f85-82ac-8e357591ae72" Jul 22 02:58:43.895313 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:43.895242 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5xv9c" podUID="0dc67dcb-94d3-439a-8b1b-6c238ccc621d" Jul 22 02:58:44.516557 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.516483 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-67.ec2.internal" event="NodeReady" Jul 22 02:58:44.517114 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.516640 2582 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 22 02:58:44.583694 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.583660 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-67vqv"] Jul 22 02:58:44.610005 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.609969 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-czh64"] Jul 22 02:58:44.610197 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.610047 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-67vqv" Jul 22 02:58:44.612550 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.612524 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 22 02:58:44.612550 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.612539 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-sn76t\"" Jul 22 02:58:44.612808 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.612796 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 22 02:58:44.626092 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.626066 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-czh64"] Jul 22 02:58:44.626092 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.626098 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-67vqv"] Jul 22 02:58:44.626303 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.626188 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-czh64" Jul 22 02:58:44.626655 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.626633 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-2nk8j"] Jul 22 02:58:44.628779 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.628752 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 22 02:58:44.628884 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.628824 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 22 02:58:44.628948 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.628914 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 22 02:58:44.629056 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.629039 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-h84cw\"" Jul 22 02:58:44.644078 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.644051 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7dd8f5b7fd-2c4kr"] Jul 22 02:58:44.644232 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.644096 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-2nk8j" Jul 22 02:58:44.646706 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.646681 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 22 02:58:44.646863 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.646681 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 22 02:58:44.646863 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.646688 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-gw7q5\"" Jul 22 02:58:44.647000 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.646944 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 22 02:58:44.647212 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.647172 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 22 02:58:44.666145 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.666121 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-2nk8j"] Jul 22 02:58:44.666317 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.666166 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7dd8f5b7fd-2c4kr"] Jul 22 02:58:44.666317 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.666267 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7dd8f5b7fd-2c4kr" Jul 22 02:58:44.668678 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.668647 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 22 02:58:44.668814 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.668693 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 22 02:58:44.668814 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.668708 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 22 02:58:44.668814 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.668655 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-7vzzs\"" Jul 22 02:58:44.673626 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.673605 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 22 02:58:44.694419 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.694392 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:58:44.762888 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.762855 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:58:44.779476 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.779395 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59gwx\" (UniqueName: \"kubernetes.io/projected/34cca172-635b-4437-a6e0-fa79ada1fd74-kube-api-access-59gwx\") pod \"kube-storage-version-migrator-operator-7658d9b749-2nk8j\" (UID: \"34cca172-635b-4437-a6e0-fa79ada1fd74\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-2nk8j" Jul 22 02:58:44.779476 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.779440 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2835f18b-f325-418a-bf5c-5600606720f5-tmp-dir\") pod \"dns-default-67vqv\" (UID: \"2835f18b-f325-418a-bf5c-5600606720f5\") " pod="openshift-dns/dns-default-67vqv" Jul 22 02:58:44.779476 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.779460 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24q82\" (UniqueName: \"kubernetes.io/projected/2835f18b-f325-418a-bf5c-5600606720f5-kube-api-access-24q82\") pod \"dns-default-67vqv\" (UID: \"2835f18b-f325-418a-bf5c-5600606720f5\") " pod="openshift-dns/dns-default-67vqv" Jul 22 02:58:44.779476 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.779482 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fe5ad29f-dbb7-4353-80ea-60bf10139703-ca-trust-extracted\") pod \"image-registry-7dd8f5b7fd-2c4kr\" (UID: \"fe5ad29f-dbb7-4353-80ea-60bf10139703\") " pod="openshift-image-registry/image-registry-7dd8f5b7fd-2c4kr" Jul 22 02:58:44.779779 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.779502 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fe5ad29f-dbb7-4353-80ea-60bf10139703-bound-sa-token\") pod \"image-registry-7dd8f5b7fd-2c4kr\" (UID: \"fe5ad29f-dbb7-4353-80ea-60bf10139703\") " pod="openshift-image-registry/image-registry-7dd8f5b7fd-2c4kr" Jul 22 02:58:44.779779 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.779534 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fe5ad29f-dbb7-4353-80ea-60bf10139703-registry-tls\") pod \"image-registry-7dd8f5b7fd-2c4kr\" (UID: \"fe5ad29f-dbb7-4353-80ea-60bf10139703\") " pod="openshift-image-registry/image-registry-7dd8f5b7fd-2c4kr" Jul 22 02:58:44.779779 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.779550 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fe5ad29f-dbb7-4353-80ea-60bf10139703-installation-pull-secrets\") pod \"image-registry-7dd8f5b7fd-2c4kr\" (UID: \"fe5ad29f-dbb7-4353-80ea-60bf10139703\") " pod="openshift-image-registry/image-registry-7dd8f5b7fd-2c4kr" Jul 22 02:58:44.779779 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.779570 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a31582c5-7c16-40ce-818e-a45d99137e90-cert\") pod \"ingress-canary-czh64\" (UID: \"a31582c5-7c16-40ce-818e-a45d99137e90\") " pod="openshift-ingress-canary/ingress-canary-czh64" Jul 22 02:58:44.779779 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.779588 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/34cca172-635b-4437-a6e0-fa79ada1fd74-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-2nk8j\" (UID: \"34cca172-635b-4437-a6e0-fa79ada1fd74\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-2nk8j" Jul 22 02:58:44.779779 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.779612 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fe5ad29f-dbb7-4353-80ea-60bf10139703-image-registry-private-configuration\") pod \"image-registry-7dd8f5b7fd-2c4kr\" (UID: \"fe5ad29f-dbb7-4353-80ea-60bf10139703\") " pod="openshift-image-registry/image-registry-7dd8f5b7fd-2c4kr" Jul 22 02:58:44.779779 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.779673 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fe5ad29f-dbb7-4353-80ea-60bf10139703-trusted-ca\") pod \"image-registry-7dd8f5b7fd-2c4kr\" (UID: \"fe5ad29f-dbb7-4353-80ea-60bf10139703\") " pod="openshift-image-registry/image-registry-7dd8f5b7fd-2c4kr" Jul 22 02:58:44.779779 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.779743 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2ttk\" (UniqueName: \"kubernetes.io/projected/fe5ad29f-dbb7-4353-80ea-60bf10139703-kube-api-access-w2ttk\") pod \"image-registry-7dd8f5b7fd-2c4kr\" (UID: \"fe5ad29f-dbb7-4353-80ea-60bf10139703\") " pod="openshift-image-registry/image-registry-7dd8f5b7fd-2c4kr" Jul 22 02:58:44.780142 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.779831 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/34cca172-635b-4437-a6e0-fa79ada1fd74-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-2nk8j\" (UID: \"34cca172-635b-4437-a6e0-fa79ada1fd74\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-2nk8j" Jul 22 02:58:44.780142 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.779861 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2835f18b-f325-418a-bf5c-5600606720f5-metrics-tls\") pod \"dns-default-67vqv\" (UID: \"2835f18b-f325-418a-bf5c-5600606720f5\") " pod="openshift-dns/dns-default-67vqv" Jul 22 02:58:44.780142 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.779887 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fe5ad29f-dbb7-4353-80ea-60bf10139703-registry-certificates\") pod \"image-registry-7dd8f5b7fd-2c4kr\" (UID: \"fe5ad29f-dbb7-4353-80ea-60bf10139703\") " pod="openshift-image-registry/image-registry-7dd8f5b7fd-2c4kr" Jul 22 02:58:44.780142 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.779969 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5g4r\" (UniqueName: \"kubernetes.io/projected/a31582c5-7c16-40ce-818e-a45d99137e90-kube-api-access-t5g4r\") pod \"ingress-canary-czh64\" (UID: \"a31582c5-7c16-40ce-818e-a45d99137e90\") " pod="openshift-ingress-canary/ingress-canary-czh64" Jul 22 02:58:44.780142 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.779996 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2835f18b-f325-418a-bf5c-5600606720f5-config-volume\") pod \"dns-default-67vqv\" (UID: \"2835f18b-f325-418a-bf5c-5600606720f5\") " pod="openshift-dns/dns-default-67vqv" Jul 22 02:58:44.880742 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.880698 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59gwx\" (UniqueName: \"kubernetes.io/projected/34cca172-635b-4437-a6e0-fa79ada1fd74-kube-api-access-59gwx\") pod \"kube-storage-version-migrator-operator-7658d9b749-2nk8j\" (UID: \"34cca172-635b-4437-a6e0-fa79ada1fd74\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-2nk8j" Jul 22 02:58:44.880742 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.880738 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2835f18b-f325-418a-bf5c-5600606720f5-tmp-dir\") pod \"dns-default-67vqv\" (UID: \"2835f18b-f325-418a-bf5c-5600606720f5\") " pod="openshift-dns/dns-default-67vqv" Jul 22 02:58:44.880983 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.880755 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24q82\" (UniqueName: \"kubernetes.io/projected/2835f18b-f325-418a-bf5c-5600606720f5-kube-api-access-24q82\") pod \"dns-default-67vqv\" (UID: \"2835f18b-f325-418a-bf5c-5600606720f5\") " pod="openshift-dns/dns-default-67vqv" Jul 22 02:58:44.880983 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.880779 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fe5ad29f-dbb7-4353-80ea-60bf10139703-ca-trust-extracted\") pod \"image-registry-7dd8f5b7fd-2c4kr\" (UID: \"fe5ad29f-dbb7-4353-80ea-60bf10139703\") " pod="openshift-image-registry/image-registry-7dd8f5b7fd-2c4kr" Jul 22 02:58:44.880983 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.880806 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fe5ad29f-dbb7-4353-80ea-60bf10139703-bound-sa-token\") pod \"image-registry-7dd8f5b7fd-2c4kr\" (UID: \"fe5ad29f-dbb7-4353-80ea-60bf10139703\") " pod="openshift-image-registry/image-registry-7dd8f5b7fd-2c4kr" Jul 22 02:58:44.880983 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.880830 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fe5ad29f-dbb7-4353-80ea-60bf10139703-registry-tls\") pod \"image-registry-7dd8f5b7fd-2c4kr\" (UID: \"fe5ad29f-dbb7-4353-80ea-60bf10139703\") " pod="openshift-image-registry/image-registry-7dd8f5b7fd-2c4kr" Jul 22 02:58:44.880983 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.880854 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fe5ad29f-dbb7-4353-80ea-60bf10139703-installation-pull-secrets\") pod \"image-registry-7dd8f5b7fd-2c4kr\" (UID: \"fe5ad29f-dbb7-4353-80ea-60bf10139703\") " pod="openshift-image-registry/image-registry-7dd8f5b7fd-2c4kr" Jul 22 02:58:44.880983 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.880881 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a31582c5-7c16-40ce-818e-a45d99137e90-cert\") pod \"ingress-canary-czh64\" (UID: \"a31582c5-7c16-40ce-818e-a45d99137e90\") " pod="openshift-ingress-canary/ingress-canary-czh64" Jul 22 02:58:44.880983 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.880904 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/34cca172-635b-4437-a6e0-fa79ada1fd74-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-2nk8j\" (UID: \"34cca172-635b-4437-a6e0-fa79ada1fd74\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-2nk8j" Jul 22 02:58:44.880983 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.880928 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fe5ad29f-dbb7-4353-80ea-60bf10139703-image-registry-private-configuration\") pod \"image-registry-7dd8f5b7fd-2c4kr\" (UID: \"fe5ad29f-dbb7-4353-80ea-60bf10139703\") " pod="openshift-image-registry/image-registry-7dd8f5b7fd-2c4kr" Jul 22 02:58:44.880983 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:44.880954 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 02:58:44.880983 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:44.880978 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7dd8f5b7fd-2c4kr: secret "image-registry-tls" not found Jul 22 02:58:44.881474 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:44.881066 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fe5ad29f-dbb7-4353-80ea-60bf10139703-registry-tls podName:fe5ad29f-dbb7-4353-80ea-60bf10139703 nodeName:}" failed. No retries permitted until 2026-07-22 02:58:45.38103775 +0000 UTC m=+36.112658576 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fe5ad29f-dbb7-4353-80ea-60bf10139703-registry-tls") pod "image-registry-7dd8f5b7fd-2c4kr" (UID: "fe5ad29f-dbb7-4353-80ea-60bf10139703") : secret "image-registry-tls" not found Jul 22 02:58:44.881474 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.881142 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2835f18b-f325-418a-bf5c-5600606720f5-tmp-dir\") pod \"dns-default-67vqv\" (UID: \"2835f18b-f325-418a-bf5c-5600606720f5\") " pod="openshift-dns/dns-default-67vqv" Jul 22 02:58:44.881474 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.880963 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fe5ad29f-dbb7-4353-80ea-60bf10139703-trusted-ca\") pod \"image-registry-7dd8f5b7fd-2c4kr\" (UID: \"fe5ad29f-dbb7-4353-80ea-60bf10139703\") " pod="openshift-image-registry/image-registry-7dd8f5b7fd-2c4kr" Jul 22 02:58:44.881474 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.881207 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2ttk\" (UniqueName: \"kubernetes.io/projected/fe5ad29f-dbb7-4353-80ea-60bf10139703-kube-api-access-w2ttk\") pod \"image-registry-7dd8f5b7fd-2c4kr\" (UID: \"fe5ad29f-dbb7-4353-80ea-60bf10139703\") " pod="openshift-image-registry/image-registry-7dd8f5b7fd-2c4kr" Jul 22 02:58:44.881474 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.881241 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fe5ad29f-dbb7-4353-80ea-60bf10139703-ca-trust-extracted\") pod \"image-registry-7dd8f5b7fd-2c4kr\" (UID: \"fe5ad29f-dbb7-4353-80ea-60bf10139703\") " pod="openshift-image-registry/image-registry-7dd8f5b7fd-2c4kr" Jul 22 02:58:44.881474 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.881305 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/34cca172-635b-4437-a6e0-fa79ada1fd74-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-2nk8j\" (UID: \"34cca172-635b-4437-a6e0-fa79ada1fd74\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-2nk8j" Jul 22 02:58:44.881474 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.881340 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2835f18b-f325-418a-bf5c-5600606720f5-metrics-tls\") pod \"dns-default-67vqv\" (UID: \"2835f18b-f325-418a-bf5c-5600606720f5\") " pod="openshift-dns/dns-default-67vqv" Jul 22 02:58:44.881474 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.881374 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fe5ad29f-dbb7-4353-80ea-60bf10139703-registry-certificates\") pod \"image-registry-7dd8f5b7fd-2c4kr\" (UID: \"fe5ad29f-dbb7-4353-80ea-60bf10139703\") " pod="openshift-image-registry/image-registry-7dd8f5b7fd-2c4kr" Jul 22 02:58:44.881474 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.881450 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5g4r\" (UniqueName: \"kubernetes.io/projected/a31582c5-7c16-40ce-818e-a45d99137e90-kube-api-access-t5g4r\") pod \"ingress-canary-czh64\" (UID: \"a31582c5-7c16-40ce-818e-a45d99137e90\") " pod="openshift-ingress-canary/ingress-canary-czh64" Jul 22 02:58:44.881907 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.881481 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2835f18b-f325-418a-bf5c-5600606720f5-config-volume\") pod \"dns-default-67vqv\" (UID: \"2835f18b-f325-418a-bf5c-5600606720f5\") " pod="openshift-dns/dns-default-67vqv" Jul 22 02:58:44.881907 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:44.881567 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 02:58:44.881907 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:44.881679 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2835f18b-f325-418a-bf5c-5600606720f5-metrics-tls podName:2835f18b-f325-418a-bf5c-5600606720f5 nodeName:}" failed. No retries permitted until 2026-07-22 02:58:45.381660462 +0000 UTC m=+36.113281302 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2835f18b-f325-418a-bf5c-5600606720f5-metrics-tls") pod "dns-default-67vqv" (UID: "2835f18b-f325-418a-bf5c-5600606720f5") : secret "dns-default-metrics-tls" not found Jul 22 02:58:44.882083 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.882064 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2835f18b-f325-418a-bf5c-5600606720f5-config-volume\") pod \"dns-default-67vqv\" (UID: \"2835f18b-f325-418a-bf5c-5600606720f5\") " pod="openshift-dns/dns-default-67vqv" Jul 22 02:58:44.882473 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.882428 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/34cca172-635b-4437-a6e0-fa79ada1fd74-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-2nk8j\" (UID: \"34cca172-635b-4437-a6e0-fa79ada1fd74\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-2nk8j" Jul 22 02:58:44.882598 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.882556 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fe5ad29f-dbb7-4353-80ea-60bf10139703-registry-certificates\") pod \"image-registry-7dd8f5b7fd-2c4kr\" (UID: \"fe5ad29f-dbb7-4353-80ea-60bf10139703\") " pod="openshift-image-registry/image-registry-7dd8f5b7fd-2c4kr" Jul 22 02:58:44.882759 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.882737 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fe5ad29f-dbb7-4353-80ea-60bf10139703-trusted-ca\") pod \"image-registry-7dd8f5b7fd-2c4kr\" (UID: \"fe5ad29f-dbb7-4353-80ea-60bf10139703\") " pod="openshift-image-registry/image-registry-7dd8f5b7fd-2c4kr" Jul 22 02:58:44.886203 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.886181 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/34cca172-635b-4437-a6e0-fa79ada1fd74-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-2nk8j\" (UID: \"34cca172-635b-4437-a6e0-fa79ada1fd74\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-2nk8j" Jul 22 02:58:44.886383 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.886363 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fe5ad29f-dbb7-4353-80ea-60bf10139703-image-registry-private-configuration\") pod \"image-registry-7dd8f5b7fd-2c4kr\" (UID: \"fe5ad29f-dbb7-4353-80ea-60bf10139703\") " pod="openshift-image-registry/image-registry-7dd8f5b7fd-2c4kr" Jul 22 02:58:44.886431 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.886416 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a31582c5-7c16-40ce-818e-a45d99137e90-cert\") pod \"ingress-canary-czh64\" (UID: \"a31582c5-7c16-40ce-818e-a45d99137e90\") " pod="openshift-ingress-canary/ingress-canary-czh64" Jul 22 02:58:44.886474 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.886425 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fe5ad29f-dbb7-4353-80ea-60bf10139703-installation-pull-secrets\") pod \"image-registry-7dd8f5b7fd-2c4kr\" (UID: \"fe5ad29f-dbb7-4353-80ea-60bf10139703\") " pod="openshift-image-registry/image-registry-7dd8f5b7fd-2c4kr" Jul 22 02:58:44.892400 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.891989 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59gwx\" (UniqueName: \"kubernetes.io/projected/34cca172-635b-4437-a6e0-fa79ada1fd74-kube-api-access-59gwx\") pod \"kube-storage-version-migrator-operator-7658d9b749-2nk8j\" (UID: \"34cca172-635b-4437-a6e0-fa79ada1fd74\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-2nk8j" Jul 22 02:58:44.892400 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.892376 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2ttk\" (UniqueName: \"kubernetes.io/projected/fe5ad29f-dbb7-4353-80ea-60bf10139703-kube-api-access-w2ttk\") pod \"image-registry-7dd8f5b7fd-2c4kr\" (UID: \"fe5ad29f-dbb7-4353-80ea-60bf10139703\") " pod="openshift-image-registry/image-registry-7dd8f5b7fd-2c4kr" Jul 22 02:58:44.892595 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.892376 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fe5ad29f-dbb7-4353-80ea-60bf10139703-bound-sa-token\") pod \"image-registry-7dd8f5b7fd-2c4kr\" (UID: \"fe5ad29f-dbb7-4353-80ea-60bf10139703\") " pod="openshift-image-registry/image-registry-7dd8f5b7fd-2c4kr" Jul 22 02:58:44.892961 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.892937 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5g4r\" (UniqueName: \"kubernetes.io/projected/a31582c5-7c16-40ce-818e-a45d99137e90-kube-api-access-t5g4r\") pod \"ingress-canary-czh64\" (UID: \"a31582c5-7c16-40ce-818e-a45d99137e90\") " pod="openshift-ingress-canary/ingress-canary-czh64" Jul 22 02:58:44.893288 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.893269 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24q82\" (UniqueName: \"kubernetes.io/projected/2835f18b-f325-418a-bf5c-5600606720f5-kube-api-access-24q82\") pod \"dns-default-67vqv\" (UID: \"2835f18b-f325-418a-bf5c-5600606720f5\") " pod="openshift-dns/dns-default-67vqv" Jul 22 02:58:44.895240 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.895218 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jqnjj" Jul 22 02:58:44.897919 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.897898 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 22 02:58:44.898376 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.898358 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sqq4d\"" Jul 22 02:58:44.935635 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.935599 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-czh64" Jul 22 02:58:44.953868 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:44.953830 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-2nk8j" Jul 22 02:58:45.320654 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:45.320618 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-2nk8j"] Jul 22 02:58:45.323338 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:45.323122 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-czh64"] Jul 22 02:58:45.370619 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:45.370587 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34cca172_635b_4437_a6e0_fa79ada1fd74.slice/crio-3b4e2c6e016350f6b65eb6421b578761f4cdcef9b8561f4acce84e29d66289fa WatchSource:0}: Error finding container 3b4e2c6e016350f6b65eb6421b578761f4cdcef9b8561f4acce84e29d66289fa: Status 404 returned error can't find the container with id 3b4e2c6e016350f6b65eb6421b578761f4cdcef9b8561f4acce84e29d66289fa Jul 22 02:58:45.370789 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:45.370774 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda31582c5_7c16_40ce_818e_a45d99137e90.slice/crio-743f37bb6fd0f6bae42c4c5db30b2b9ce73408a911bf80a0c77c7077fdde217c WatchSource:0}: Error finding container 743f37bb6fd0f6bae42c4c5db30b2b9ce73408a911bf80a0c77c7077fdde217c: Status 404 returned error can't find the container with id 743f37bb6fd0f6bae42c4c5db30b2b9ce73408a911bf80a0c77c7077fdde217c Jul 22 02:58:45.386711 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:45.386603 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fe5ad29f-dbb7-4353-80ea-60bf10139703-registry-tls\") pod \"image-registry-7dd8f5b7fd-2c4kr\" (UID: \"fe5ad29f-dbb7-4353-80ea-60bf10139703\") " pod="openshift-image-registry/image-registry-7dd8f5b7fd-2c4kr" Jul 22 02:58:45.386861 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:45.386745 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2835f18b-f325-418a-bf5c-5600606720f5-metrics-tls\") pod \"dns-default-67vqv\" (UID: \"2835f18b-f325-418a-bf5c-5600606720f5\") " pod="openshift-dns/dns-default-67vqv" Jul 22 02:58:45.386861 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:45.386750 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 02:58:45.386861 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:45.386769 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7dd8f5b7fd-2c4kr: secret "image-registry-tls" not found Jul 22 02:58:45.386861 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:45.386832 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fe5ad29f-dbb7-4353-80ea-60bf10139703-registry-tls podName:fe5ad29f-dbb7-4353-80ea-60bf10139703 nodeName:}" failed. No retries permitted until 2026-07-22 02:58:46.386816624 +0000 UTC m=+37.118437452 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fe5ad29f-dbb7-4353-80ea-60bf10139703-registry-tls") pod "image-registry-7dd8f5b7fd-2c4kr" (UID: "fe5ad29f-dbb7-4353-80ea-60bf10139703") : secret "image-registry-tls" not found Jul 22 02:58:45.386861 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:45.386859 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 02:58:45.387158 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:45.386900 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2835f18b-f325-418a-bf5c-5600606720f5-metrics-tls podName:2835f18b-f325-418a-bf5c-5600606720f5 nodeName:}" failed. No retries permitted until 2026-07-22 02:58:46.386889307 +0000 UTC m=+37.118510131 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2835f18b-f325-418a-bf5c-5600606720f5-metrics-tls") pod "dns-default-67vqv" (UID: "2835f18b-f325-418a-bf5c-5600606720f5") : secret "dns-default-metrics-tls" not found Jul 22 02:58:45.895080 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:45.895046 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5xv9c" Jul 22 02:58:45.895785 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:45.895099 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lc5mn" Jul 22 02:58:45.898265 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:45.898240 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 22 02:58:45.898408 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:45.898243 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 22 02:58:45.898956 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:45.898933 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nztgp\"" Jul 22 02:58:45.899082 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:45.898961 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 22 02:58:46.165620 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:46.165534 2582 generic.go:356] "Generic (PLEG): container finished" podID="6ded3042-9b57-4211-af1c-6660ad8363ef" containerID="b46df9a49a8f75e1741dde808caa0c87c6808b7490543e7d15f6b2b4d494b932" exitCode=0 Jul 22 02:58:46.165784 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:46.165626 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6hdd2" event={"ID":"6ded3042-9b57-4211-af1c-6660ad8363ef","Type":"ContainerDied","Data":"b46df9a49a8f75e1741dde808caa0c87c6808b7490543e7d15f6b2b4d494b932"} Jul 22 02:58:46.167076 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:46.167043 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-czh64" event={"ID":"a31582c5-7c16-40ce-818e-a45d99137e90","Type":"ContainerStarted","Data":"743f37bb6fd0f6bae42c4c5db30b2b9ce73408a911bf80a0c77c7077fdde217c"} Jul 22 02:58:46.168423 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:46.168400 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-2nk8j" event={"ID":"34cca172-635b-4437-a6e0-fa79ada1fd74","Type":"ContainerStarted","Data":"3b4e2c6e016350f6b65eb6421b578761f4cdcef9b8561f4acce84e29d66289fa"} Jul 22 02:58:46.395031 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:46.394974 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2835f18b-f325-418a-bf5c-5600606720f5-metrics-tls\") pod \"dns-default-67vqv\" (UID: \"2835f18b-f325-418a-bf5c-5600606720f5\") " pod="openshift-dns/dns-default-67vqv" Jul 22 02:58:46.395218 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:46.395090 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fe5ad29f-dbb7-4353-80ea-60bf10139703-registry-tls\") pod \"image-registry-7dd8f5b7fd-2c4kr\" (UID: \"fe5ad29f-dbb7-4353-80ea-60bf10139703\") " pod="openshift-image-registry/image-registry-7dd8f5b7fd-2c4kr" Jul 22 02:58:46.395218 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:46.395133 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 02:58:46.395218 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:46.395206 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2835f18b-f325-418a-bf5c-5600606720f5-metrics-tls podName:2835f18b-f325-418a-bf5c-5600606720f5 nodeName:}" failed. No retries permitted until 2026-07-22 02:58:48.395185453 +0000 UTC m=+39.126806284 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2835f18b-f325-418a-bf5c-5600606720f5-metrics-tls") pod "dns-default-67vqv" (UID: "2835f18b-f325-418a-bf5c-5600606720f5") : secret "dns-default-metrics-tls" not found Jul 22 02:58:46.395395 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:46.395222 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 02:58:46.395395 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:46.395237 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7dd8f5b7fd-2c4kr: secret "image-registry-tls" not found Jul 22 02:58:46.395395 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:46.395288 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fe5ad29f-dbb7-4353-80ea-60bf10139703-registry-tls podName:fe5ad29f-dbb7-4353-80ea-60bf10139703 nodeName:}" failed. No retries permitted until 2026-07-22 02:58:48.39527321 +0000 UTC m=+39.126894037 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fe5ad29f-dbb7-4353-80ea-60bf10139703-registry-tls") pod "image-registry-7dd8f5b7fd-2c4kr" (UID: "fe5ad29f-dbb7-4353-80ea-60bf10139703") : secret "image-registry-tls" not found Jul 22 02:58:46.818251 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:46.818217 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:58:46.890248 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:46.890201 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:58:47.174936 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:47.174894 2582 generic.go:356] "Generic (PLEG): container finished" podID="6ded3042-9b57-4211-af1c-6660ad8363ef" containerID="b58e143db839d7ab2259440728eb8bba70b7a9818663a4f930ab6c66639cd56b" exitCode=0 Jul 22 02:58:47.175405 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:47.174951 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6hdd2" event={"ID":"6ded3042-9b57-4211-af1c-6660ad8363ef","Type":"ContainerDied","Data":"b58e143db839d7ab2259440728eb8bba70b7a9818663a4f930ab6c66639cd56b"} Jul 22 02:58:47.708293 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:47.708255 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:58:47.805978 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:47.805939 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:58:48.045549 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:48.045469 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:58:48.415206 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:48.415172 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fe5ad29f-dbb7-4353-80ea-60bf10139703-registry-tls\") pod \"image-registry-7dd8f5b7fd-2c4kr\" (UID: \"fe5ad29f-dbb7-4353-80ea-60bf10139703\") " pod="openshift-image-registry/image-registry-7dd8f5b7fd-2c4kr" Jul 22 02:58:48.415579 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:48.415233 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2835f18b-f325-418a-bf5c-5600606720f5-metrics-tls\") pod \"dns-default-67vqv\" (UID: \"2835f18b-f325-418a-bf5c-5600606720f5\") " pod="openshift-dns/dns-default-67vqv" Jul 22 02:58:48.415579 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:48.415329 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 02:58:48.415579 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:48.415359 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 02:58:48.415579 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:48.415409 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2835f18b-f325-418a-bf5c-5600606720f5-metrics-tls podName:2835f18b-f325-418a-bf5c-5600606720f5 nodeName:}" failed. No retries permitted until 2026-07-22 02:58:52.415395784 +0000 UTC m=+43.147016609 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2835f18b-f325-418a-bf5c-5600606720f5-metrics-tls") pod "dns-default-67vqv" (UID: "2835f18b-f325-418a-bf5c-5600606720f5") : secret "dns-default-metrics-tls" not found Jul 22 02:58:48.415579 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:48.415361 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7dd8f5b7fd-2c4kr: secret "image-registry-tls" not found Jul 22 02:58:48.415579 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:48.415489 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fe5ad29f-dbb7-4353-80ea-60bf10139703-registry-tls podName:fe5ad29f-dbb7-4353-80ea-60bf10139703 nodeName:}" failed. No retries permitted until 2026-07-22 02:58:52.415461998 +0000 UTC m=+43.147082826 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fe5ad29f-dbb7-4353-80ea-60bf10139703-registry-tls") pod "image-registry-7dd8f5b7fd-2c4kr" (UID: "fe5ad29f-dbb7-4353-80ea-60bf10139703") : secret "image-registry-tls" not found Jul 22 02:58:48.946063 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:48.945864 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:58:49.032849 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:49.032812 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:58:49.184002 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:49.183967 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6hdd2" event={"ID":"6ded3042-9b57-4211-af1c-6660ad8363ef","Type":"ContainerStarted","Data":"dcd500743c91ca25c28cba7d11a0ffd3ca866d9bcdbc4dd4592c47956698a4d3"} Jul 22 02:58:49.184173 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:49.184090 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-vs55f"] Jul 22 02:58:49.208554 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:49.208454 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-6hdd2" podStartSLOduration=9.718730399 podStartE2EDuration="39.208438818s" totalImagesPullingTime="29.489708419s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 02:58:10 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:58:49.207467596 +0000 UTC m=+39.939088463" watchObservedRunningTime="2026-07-22 02:58:49.208438818 +0000 UTC m=+39.940059664" Jul 22 02:58:49.218456 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:49.218425 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-vs55f"] Jul 22 02:58:49.218605 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:49.218464 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-czh64" event={"ID":"a31582c5-7c16-40ce-818e-a45d99137e90","Type":"ContainerStarted","Data":"212660e1b6acc0d31cf495b4216c166b9eb86563cc9561691359eecee72b1a16"} Jul 22 02:58:49.218605 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:49.218489 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-2nk8j" event={"ID":"34cca172-635b-4437-a6e0-fa79ada1fd74","Type":"ContainerStarted","Data":"669e13f5da7b3bfc013ac3dbd4cd71a2e2e17bf7feb165b152f4447bd8cf2b54"} Jul 22 02:58:49.218605 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:49.218540 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-vs55f" Jul 22 02:58:49.221123 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:49.221102 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-ndgkp\"" Jul 22 02:58:49.222742 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:49.222700 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-czh64" podStartSLOduration=2.1155399089999998 podStartE2EDuration="5.222687104s" totalImagesPullingTime="3.107147195s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:58:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:58:49.222057495 +0000 UTC m=+39.953678369" watchObservedRunningTime="2026-07-22 02:58:49.222687104 +0000 UTC m=+39.954307952" Jul 22 02:58:49.239995 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:49.239940 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-2nk8j" podStartSLOduration=2.139965695 podStartE2EDuration="5.239922855s" totalImagesPullingTime="3.09995716s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:58:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:58:49.238824556 +0000 UTC m=+39.970445404" watchObservedRunningTime="2026-07-22 02:58:49.239922855 +0000 UTC m=+39.971543702" Jul 22 02:58:49.323199 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:49.323154 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xgrc\" (UniqueName: \"kubernetes.io/projected/7484f78f-4c8b-42bf-b5f5-30df1527ada6-kube-api-access-9xgrc\") pod \"network-check-source-7bc5b6bf85-vs55f\" (UID: \"7484f78f-4c8b-42bf-b5f5-30df1527ada6\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-vs55f" Jul 22 02:58:49.333889 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:49.333855 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:58:49.424208 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:49.424175 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xgrc\" (UniqueName: \"kubernetes.io/projected/7484f78f-4c8b-42bf-b5f5-30df1527ada6-kube-api-access-9xgrc\") pod \"network-check-source-7bc5b6bf85-vs55f\" (UID: \"7484f78f-4c8b-42bf-b5f5-30df1527ada6\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-vs55f" Jul 22 02:58:49.450485 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:49.450457 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xgrc\" (UniqueName: \"kubernetes.io/projected/7484f78f-4c8b-42bf-b5f5-30df1527ada6-kube-api-access-9xgrc\") pod \"network-check-source-7bc5b6bf85-vs55f\" (UID: \"7484f78f-4c8b-42bf-b5f5-30df1527ada6\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-vs55f" Jul 22 02:58:49.451942 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:49.451914 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:58:49.519557 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:49.519484 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-h5q8g"] Jul 22 02:58:49.527544 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:49.527516 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-vs55f" Jul 22 02:58:49.541023 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:49.540982 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-h5q8g"] Jul 22 02:58:49.541151 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:49.541113 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-h5q8g" Jul 22 02:58:49.543695 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:49.543641 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 22 02:58:49.543695 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:49.543692 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-ln7kj\"" Jul 22 02:58:49.543918 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:49.543648 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 22 02:58:49.588726 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:49.588673 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:58:49.675888 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:49.675860 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-vs55f"] Jul 22 02:58:49.679398 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:49.679372 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7484f78f_4c8b_42bf_b5f5_30df1527ada6.slice/crio-9968b61a16c9c05d4f46a3592e9f50341ae73f265a041cb453c188501084b4eb WatchSource:0}: Error finding container 9968b61a16c9c05d4f46a3592e9f50341ae73f265a041cb453c188501084b4eb: Status 404 returned error can't find the container with id 9968b61a16c9c05d4f46a3592e9f50341ae73f265a041cb453c188501084b4eb Jul 22 02:58:49.680949 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:49.680907 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:58:49.726327 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:49.726296 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gphv4\" (UniqueName: \"kubernetes.io/projected/46945b18-460c-4da9-b5d3-eeeaf93b3a67-kube-api-access-gphv4\") pod \"migrator-7cc9c7ffd7-h5q8g\" (UID: \"46945b18-460c-4da9-b5d3-eeeaf93b3a67\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-h5q8g" Jul 22 02:58:49.827504 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:49.827407 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gphv4\" (UniqueName: \"kubernetes.io/projected/46945b18-460c-4da9-b5d3-eeeaf93b3a67-kube-api-access-gphv4\") pod \"migrator-7cc9c7ffd7-h5q8g\" (UID: \"46945b18-460c-4da9-b5d3-eeeaf93b3a67\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-h5q8g" Jul 22 02:58:49.836375 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:49.836351 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gphv4\" (UniqueName: \"kubernetes.io/projected/46945b18-460c-4da9-b5d3-eeeaf93b3a67-kube-api-access-gphv4\") pod \"migrator-7cc9c7ffd7-h5q8g\" (UID: \"46945b18-460c-4da9-b5d3-eeeaf93b3a67\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-h5q8g" Jul 22 02:58:49.850298 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:49.850258 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-h5q8g" Jul 22 02:58:50.025801 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:50.025768 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-h5q8g"] Jul 22 02:58:50.029362 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:58:50.029335 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46945b18_460c_4da9_b5d3_eeeaf93b3a67.slice/crio-4e73ecc52c63196852beaf2e2f45b00dde4f96f87fcc9869deb66f095271111b WatchSource:0}: Error finding container 4e73ecc52c63196852beaf2e2f45b00dde4f96f87fcc9869deb66f095271111b: Status 404 returned error can't find the container with id 4e73ecc52c63196852beaf2e2f45b00dde4f96f87fcc9869deb66f095271111b Jul 22 02:58:50.190834 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:50.190791 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-h5q8g" event={"ID":"46945b18-460c-4da9-b5d3-eeeaf93b3a67","Type":"ContainerStarted","Data":"4e73ecc52c63196852beaf2e2f45b00dde4f96f87fcc9869deb66f095271111b"} Jul 22 02:58:50.192043 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:50.191995 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-vs55f" event={"ID":"7484f78f-4c8b-42bf-b5f5-30df1527ada6","Type":"ContainerStarted","Data":"9968b61a16c9c05d4f46a3592e9f50341ae73f265a041cb453c188501084b4eb"} Jul 22 02:58:50.237161 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:50.237129 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-zhrpq"] Jul 22 02:58:50.254172 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:50.254107 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zhrpq"] Jul 22 02:58:50.254348 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:50.254224 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zhrpq" Jul 22 02:58:50.256686 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:50.256657 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 22 02:58:50.256812 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:50.256764 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 22 02:58:50.256942 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:50.256922 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-zh2d9\"" Jul 22 02:58:50.257107 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:50.257077 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 22 02:58:50.257390 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:50.257359 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 22 02:58:50.284524 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:50.284485 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:58:50.359507 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:50.359446 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:58:50.433332 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:50.433294 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c9f901fc-712c-4bc6-a5c8-2471837fcc18-crio-socket\") pod \"insights-runtime-extractor-zhrpq\" (UID: \"c9f901fc-712c-4bc6-a5c8-2471837fcc18\") " pod="openshift-insights/insights-runtime-extractor-zhrpq" Jul 22 02:58:50.433822 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:50.433340 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c9f901fc-712c-4bc6-a5c8-2471837fcc18-data-volume\") pod \"insights-runtime-extractor-zhrpq\" (UID: \"c9f901fc-712c-4bc6-a5c8-2471837fcc18\") " pod="openshift-insights/insights-runtime-extractor-zhrpq" Jul 22 02:58:50.433822 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:50.433368 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv6kh\" (UniqueName: \"kubernetes.io/projected/c9f901fc-712c-4bc6-a5c8-2471837fcc18-kube-api-access-bv6kh\") pod \"insights-runtime-extractor-zhrpq\" (UID: \"c9f901fc-712c-4bc6-a5c8-2471837fcc18\") " pod="openshift-insights/insights-runtime-extractor-zhrpq" Jul 22 02:58:50.433822 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:50.433460 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c9f901fc-712c-4bc6-a5c8-2471837fcc18-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zhrpq\" (UID: \"c9f901fc-712c-4bc6-a5c8-2471837fcc18\") " pod="openshift-insights/insights-runtime-extractor-zhrpq" Jul 22 02:58:50.433822 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:50.433496 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c9f901fc-712c-4bc6-a5c8-2471837fcc18-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zhrpq\" (UID: \"c9f901fc-712c-4bc6-a5c8-2471837fcc18\") " pod="openshift-insights/insights-runtime-extractor-zhrpq" Jul 22 02:58:50.534767 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:50.534673 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c9f901fc-712c-4bc6-a5c8-2471837fcc18-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zhrpq\" (UID: \"c9f901fc-712c-4bc6-a5c8-2471837fcc18\") " pod="openshift-insights/insights-runtime-extractor-zhrpq" Jul 22 02:58:50.534767 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:50.534724 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c9f901fc-712c-4bc6-a5c8-2471837fcc18-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zhrpq\" (UID: \"c9f901fc-712c-4bc6-a5c8-2471837fcc18\") " pod="openshift-insights/insights-runtime-extractor-zhrpq" Jul 22 02:58:50.534995 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:50.534821 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c9f901fc-712c-4bc6-a5c8-2471837fcc18-crio-socket\") pod \"insights-runtime-extractor-zhrpq\" (UID: \"c9f901fc-712c-4bc6-a5c8-2471837fcc18\") " pod="openshift-insights/insights-runtime-extractor-zhrpq" Jul 22 02:58:50.534995 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:50.534834 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 02:58:50.534995 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:50.534853 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 02:58:50.534995 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:50.534902 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9f901fc-712c-4bc6-a5c8-2471837fcc18-insights-runtime-extractor-tls podName:c9f901fc-712c-4bc6-a5c8-2471837fcc18 nodeName:}" failed. No retries permitted until 2026-07-22 02:58:51.034881999 +0000 UTC m=+41.766502824 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c9f901fc-712c-4bc6-a5c8-2471837fcc18-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zhrpq" (UID: "c9f901fc-712c-4bc6-a5c8-2471837fcc18") : secret "insights-runtime-extractor-tls" not found Jul 22 02:58:50.534995 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:50.534917 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c9f901fc-712c-4bc6-a5c8-2471837fcc18-kube-rbac-proxy-cm podName:c9f901fc-712c-4bc6-a5c8-2471837fcc18 nodeName:}" failed. No retries permitted until 2026-07-22 02:58:51.034910456 +0000 UTC m=+41.766531281 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c9f901fc-712c-4bc6-a5c8-2471837fcc18-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zhrpq" (UID: "c9f901fc-712c-4bc6-a5c8-2471837fcc18") : configmap "kube-rbac-proxy" not found Jul 22 02:58:50.534995 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:50.534931 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c9f901fc-712c-4bc6-a5c8-2471837fcc18-data-volume\") pod \"insights-runtime-extractor-zhrpq\" (UID: \"c9f901fc-712c-4bc6-a5c8-2471837fcc18\") " pod="openshift-insights/insights-runtime-extractor-zhrpq" Jul 22 02:58:50.534995 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:50.534955 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c9f901fc-712c-4bc6-a5c8-2471837fcc18-crio-socket\") pod \"insights-runtime-extractor-zhrpq\" (UID: \"c9f901fc-712c-4bc6-a5c8-2471837fcc18\") " pod="openshift-insights/insights-runtime-extractor-zhrpq" Jul 22 02:58:50.534995 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:50.534959 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv6kh\" (UniqueName: \"kubernetes.io/projected/c9f901fc-712c-4bc6-a5c8-2471837fcc18-kube-api-access-bv6kh\") pod \"insights-runtime-extractor-zhrpq\" (UID: \"c9f901fc-712c-4bc6-a5c8-2471837fcc18\") " pod="openshift-insights/insights-runtime-extractor-zhrpq" Jul 22 02:58:50.535397 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:50.535379 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c9f901fc-712c-4bc6-a5c8-2471837fcc18-data-volume\") pod \"insights-runtime-extractor-zhrpq\" (UID: \"c9f901fc-712c-4bc6-a5c8-2471837fcc18\") " pod="openshift-insights/insights-runtime-extractor-zhrpq" Jul 22 02:58:50.543755 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:50.543727 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv6kh\" (UniqueName: \"kubernetes.io/projected/c9f901fc-712c-4bc6-a5c8-2471837fcc18-kube-api-access-bv6kh\") pod \"insights-runtime-extractor-zhrpq\" (UID: \"c9f901fc-712c-4bc6-a5c8-2471837fcc18\") " pod="openshift-insights/insights-runtime-extractor-zhrpq" Jul 22 02:58:50.584027 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:50.583992 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:58:50.663277 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:50.663238 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:58:51.040145 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:51.040115 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c9f901fc-712c-4bc6-a5c8-2471837fcc18-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zhrpq\" (UID: \"c9f901fc-712c-4bc6-a5c8-2471837fcc18\") " pod="openshift-insights/insights-runtime-extractor-zhrpq" Jul 22 02:58:51.040145 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:51.040150 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c9f901fc-712c-4bc6-a5c8-2471837fcc18-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zhrpq\" (UID: \"c9f901fc-712c-4bc6-a5c8-2471837fcc18\") " pod="openshift-insights/insights-runtime-extractor-zhrpq" Jul 22 02:58:51.040443 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:51.040294 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 02:58:51.040443 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:51.040363 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9f901fc-712c-4bc6-a5c8-2471837fcc18-insights-runtime-extractor-tls podName:c9f901fc-712c-4bc6-a5c8-2471837fcc18 nodeName:}" failed. No retries permitted until 2026-07-22 02:58:52.04034551 +0000 UTC m=+42.771966335 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c9f901fc-712c-4bc6-a5c8-2471837fcc18-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zhrpq" (UID: "c9f901fc-712c-4bc6-a5c8-2471837fcc18") : secret "insights-runtime-extractor-tls" not found Jul 22 02:58:51.040443 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:51.040297 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 02:58:51.040443 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:51.040435 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c9f901fc-712c-4bc6-a5c8-2471837fcc18-kube-rbac-proxy-cm podName:c9f901fc-712c-4bc6-a5c8-2471837fcc18 nodeName:}" failed. No retries permitted until 2026-07-22 02:58:52.040423916 +0000 UTC m=+42.772044742 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c9f901fc-712c-4bc6-a5c8-2471837fcc18-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zhrpq" (UID: "c9f901fc-712c-4bc6-a5c8-2471837fcc18") : configmap "kube-rbac-proxy" not found Jul 22 02:58:51.160458 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:51.160423 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:58:51.364441 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:51.364359 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:58:51.473030 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:51.472982 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:58:52.048974 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:52.048936 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c9f901fc-712c-4bc6-a5c8-2471837fcc18-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zhrpq\" (UID: \"c9f901fc-712c-4bc6-a5c8-2471837fcc18\") " pod="openshift-insights/insights-runtime-extractor-zhrpq" Jul 22 02:58:52.049187 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:52.048983 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c9f901fc-712c-4bc6-a5c8-2471837fcc18-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zhrpq\" (UID: \"c9f901fc-712c-4bc6-a5c8-2471837fcc18\") " pod="openshift-insights/insights-runtime-extractor-zhrpq" Jul 22 02:58:52.049187 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:52.049084 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 02:58:52.049187 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:52.049106 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 02:58:52.049187 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:52.049163 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9f901fc-712c-4bc6-a5c8-2471837fcc18-insights-runtime-extractor-tls podName:c9f901fc-712c-4bc6-a5c8-2471837fcc18 nodeName:}" failed. No retries permitted until 2026-07-22 02:58:54.049142632 +0000 UTC m=+44.780763470 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c9f901fc-712c-4bc6-a5c8-2471837fcc18-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zhrpq" (UID: "c9f901fc-712c-4bc6-a5c8-2471837fcc18") : secret "insights-runtime-extractor-tls" not found Jul 22 02:58:52.049187 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:52.049186 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c9f901fc-712c-4bc6-a5c8-2471837fcc18-kube-rbac-proxy-cm podName:c9f901fc-712c-4bc6-a5c8-2471837fcc18 nodeName:}" failed. No retries permitted until 2026-07-22 02:58:54.049174641 +0000 UTC m=+44.780795472 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c9f901fc-712c-4bc6-a5c8-2471837fcc18-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zhrpq" (UID: "c9f901fc-712c-4bc6-a5c8-2471837fcc18") : configmap "kube-rbac-proxy" not found Jul 22 02:58:52.453190 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:52.453145 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2835f18b-f325-418a-bf5c-5600606720f5-metrics-tls\") pod \"dns-default-67vqv\" (UID: \"2835f18b-f325-418a-bf5c-5600606720f5\") " pod="openshift-dns/dns-default-67vqv" Jul 22 02:58:52.453400 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:52.453253 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fe5ad29f-dbb7-4353-80ea-60bf10139703-registry-tls\") pod \"image-registry-7dd8f5b7fd-2c4kr\" (UID: \"fe5ad29f-dbb7-4353-80ea-60bf10139703\") " pod="openshift-image-registry/image-registry-7dd8f5b7fd-2c4kr" Jul 22 02:58:52.453400 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:52.453327 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 02:58:52.453512 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:52.453409 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2835f18b-f325-418a-bf5c-5600606720f5-metrics-tls podName:2835f18b-f325-418a-bf5c-5600606720f5 nodeName:}" failed. No retries permitted until 2026-07-22 02:59:00.453389125 +0000 UTC m=+51.185009965 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2835f18b-f325-418a-bf5c-5600606720f5-metrics-tls") pod "dns-default-67vqv" (UID: "2835f18b-f325-418a-bf5c-5600606720f5") : secret "dns-default-metrics-tls" not found Jul 22 02:58:52.453512 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:52.453415 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 02:58:52.453512 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:52.453431 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7dd8f5b7fd-2c4kr: secret "image-registry-tls" not found Jul 22 02:58:52.453512 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:52.453473 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fe5ad29f-dbb7-4353-80ea-60bf10139703-registry-tls podName:fe5ad29f-dbb7-4353-80ea-60bf10139703 nodeName:}" failed. No retries permitted until 2026-07-22 02:59:00.453458042 +0000 UTC m=+51.185078872 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fe5ad29f-dbb7-4353-80ea-60bf10139703-registry-tls") pod "image-registry-7dd8f5b7fd-2c4kr" (UID: "fe5ad29f-dbb7-4353-80ea-60bf10139703") : secret "image-registry-tls" not found Jul 22 02:58:52.779560 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:52.779485 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:58:52.822413 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:52.822391 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5gftd_8c3db93b-85e3-4f29-a0cc-1b6b3d847a94/dns-node-resolver/0.log" Jul 22 02:58:52.879718 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:52.879690 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:58:52.951463 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:52.951432 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:58:53.206274 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:53.206226 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-h5q8g" event={"ID":"46945b18-460c-4da9-b5d3-eeeaf93b3a67","Type":"ContainerStarted","Data":"e8c6ba2037542c514094b376b818b35886783c2661c8c6d7e9edd7f341da567e"} Jul 22 02:58:53.471693 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:53.471644 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:58:53.876061 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:53.876026 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:58:54.066610 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:54.066572 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c9f901fc-712c-4bc6-a5c8-2471837fcc18-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zhrpq\" (UID: \"c9f901fc-712c-4bc6-a5c8-2471837fcc18\") " pod="openshift-insights/insights-runtime-extractor-zhrpq" Jul 22 02:58:54.066610 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:54.066614 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c9f901fc-712c-4bc6-a5c8-2471837fcc18-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zhrpq\" (UID: \"c9f901fc-712c-4bc6-a5c8-2471837fcc18\") " pod="openshift-insights/insights-runtime-extractor-zhrpq" Jul 22 02:58:54.066783 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:54.066684 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 02:58:54.066783 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:54.066736 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c9f901fc-712c-4bc6-a5c8-2471837fcc18-kube-rbac-proxy-cm podName:c9f901fc-712c-4bc6-a5c8-2471837fcc18 nodeName:}" failed. No retries permitted until 2026-07-22 02:58:58.066722401 +0000 UTC m=+48.798343227 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c9f901fc-712c-4bc6-a5c8-2471837fcc18-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zhrpq" (UID: "c9f901fc-712c-4bc6-a5c8-2471837fcc18") : configmap "kube-rbac-proxy" not found Jul 22 02:58:54.066783 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:54.066750 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 02:58:54.067065 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:54.066815 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9f901fc-712c-4bc6-a5c8-2471837fcc18-insights-runtime-extractor-tls podName:c9f901fc-712c-4bc6-a5c8-2471837fcc18 nodeName:}" failed. No retries permitted until 2026-07-22 02:58:58.066796572 +0000 UTC m=+48.798417409 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c9f901fc-712c-4bc6-a5c8-2471837fcc18-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zhrpq" (UID: "c9f901fc-712c-4bc6-a5c8-2471837fcc18") : secret "insights-runtime-extractor-tls" not found Jul 22 02:58:54.210188 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:54.210067 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-vs55f" event={"ID":"7484f78f-4c8b-42bf-b5f5-30df1527ada6","Type":"ContainerStarted","Data":"ff081d9637a145daa202e50ab009ebd9461a0b58e5f54444f7c588579cbfdc1f"} Jul 22 02:58:54.211822 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:54.211780 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-h5q8g" event={"ID":"46945b18-460c-4da9-b5d3-eeeaf93b3a67","Type":"ContainerStarted","Data":"4a8baee182b78eb4b3e2d7c80935b80ecaddfed728be1ddf0c48503d35e8a4ea"} Jul 22 02:58:54.223112 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:54.223079 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vssht_5ee42689-5bff-46fe-8047-adbe6cb34273/node-ca/0.log" Jul 22 02:58:54.228098 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:54.228056 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-vs55f" podStartSLOduration=1.776204974 podStartE2EDuration="5.228040684s" totalImagesPullingTime="3.45183571s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:58:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:58:54.226878244 +0000 UTC m=+44.958499102" watchObservedRunningTime="2026-07-22 02:58:54.228040684 +0000 UTC m=+44.959661532" Jul 22 02:58:54.242340 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:54.242285 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-h5q8g" podStartSLOduration=2.469504963 podStartE2EDuration="5.242267815s" totalImagesPullingTime="2.772762852s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:58:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:58:54.241440845 +0000 UTC m=+44.973061714" watchObservedRunningTime="2026-07-22 02:58:54.242267815 +0000 UTC m=+44.973888658" Jul 22 02:58:54.622284 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:54.622246 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-czh64_a31582c5-7c16-40ce-818e-a45d99137e90/serve-healthcheck-canary/0.log" Jul 22 02:58:55.223332 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:55.223305 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-h5q8g_46945b18-460c-4da9-b5d3-eeeaf93b3a67/migrator/0.log" Jul 22 02:58:55.422811 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:55.422786 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-h5q8g_46945b18-460c-4da9-b5d3-eeeaf93b3a67/graceful-termination/0.log" Jul 22 02:58:55.623989 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:55.623964 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-2nk8j_34cca172-635b-4437-a6e0-fa79ada1fd74/kube-storage-version-migrator-operator/0.log" Jul 22 02:58:55.813595 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:55.813565 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:58:55.935583 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:55.935505 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:58:56.064570 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:56.064539 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:58:56.212381 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:56.212297 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:58:56.373770 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:56.373738 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:58:56.479294 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:56.479186 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:58:56.574792 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:56.574761 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:58:57.137899 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:57.137865 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:58:57.535866 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:57.535771 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:58:57.914887 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:57.914851 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:58:58.100292 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:58.100255 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c9f901fc-712c-4bc6-a5c8-2471837fcc18-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zhrpq\" (UID: \"c9f901fc-712c-4bc6-a5c8-2471837fcc18\") " pod="openshift-insights/insights-runtime-extractor-zhrpq" Jul 22 02:58:58.100292 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:58.100296 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c9f901fc-712c-4bc6-a5c8-2471837fcc18-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zhrpq\" (UID: \"c9f901fc-712c-4bc6-a5c8-2471837fcc18\") " pod="openshift-insights/insights-runtime-extractor-zhrpq" Jul 22 02:58:58.100506 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:58.100370 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 02:58:58.100506 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:58.100405 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 02:58:58.100506 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:58.100470 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9f901fc-712c-4bc6-a5c8-2471837fcc18-insights-runtime-extractor-tls podName:c9f901fc-712c-4bc6-a5c8-2471837fcc18 nodeName:}" failed. No retries permitted until 2026-07-22 02:59:06.100450917 +0000 UTC m=+56.832071751 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c9f901fc-712c-4bc6-a5c8-2471837fcc18-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zhrpq" (UID: "c9f901fc-712c-4bc6-a5c8-2471837fcc18") : secret "insights-runtime-extractor-tls" not found Jul 22 02:58:58.100506 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:58:58.100490 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c9f901fc-712c-4bc6-a5c8-2471837fcc18-kube-rbac-proxy-cm podName:c9f901fc-712c-4bc6-a5c8-2471837fcc18 nodeName:}" failed. No retries permitted until 2026-07-22 02:59:06.100483718 +0000 UTC m=+56.832104543 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c9f901fc-712c-4bc6-a5c8-2471837fcc18-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zhrpq" (UID: "c9f901fc-712c-4bc6-a5c8-2471837fcc18") : configmap "kube-rbac-proxy" not found Jul 22 02:58:58.125044 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:58.125000 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:58:58.317251 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:58.317168 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:58:58.720982 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:58.720945 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:58:58.925407 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:58.925374 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:58:59.520214 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:59.520181 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:58:59.620836 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:59.620803 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:58:59.721181 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:59.721157 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:58:59.913853 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:59.913823 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:58:59.988214 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:58:59.988188 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:00.516525 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:00.516493 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:00.519613 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:00.519588 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2835f18b-f325-418a-bf5c-5600606720f5-metrics-tls\") pod \"dns-default-67vqv\" (UID: \"2835f18b-f325-418a-bf5c-5600606720f5\") " pod="openshift-dns/dns-default-67vqv" Jul 22 02:59:00.519676 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:00.519657 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fe5ad29f-dbb7-4353-80ea-60bf10139703-registry-tls\") pod \"image-registry-7dd8f5b7fd-2c4kr\" (UID: \"fe5ad29f-dbb7-4353-80ea-60bf10139703\") " pod="openshift-image-registry/image-registry-7dd8f5b7fd-2c4kr" Jul 22 02:59:00.521932 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:00.521906 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2835f18b-f325-418a-bf5c-5600606720f5-metrics-tls\") pod \"dns-default-67vqv\" (UID: \"2835f18b-f325-418a-bf5c-5600606720f5\") " pod="openshift-dns/dns-default-67vqv" Jul 22 02:59:00.522056 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:00.521981 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fe5ad29f-dbb7-4353-80ea-60bf10139703-registry-tls\") pod \"image-registry-7dd8f5b7fd-2c4kr\" (UID: \"fe5ad29f-dbb7-4353-80ea-60bf10139703\") " pod="openshift-image-registry/image-registry-7dd8f5b7fd-2c4kr" Jul 22 02:59:00.577988 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:00.577949 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7dd8f5b7fd-2c4kr" Jul 22 02:59:00.722086 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:00.722054 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7dd8f5b7fd-2c4kr"] Jul 22 02:59:00.725282 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:59:00.725256 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe5ad29f_dbb7_4353_80ea_60bf10139703.slice/crio-efdcc6e09fda7b23aada5f9aeacba2e9718f3e17832059d893625c99746ef72d WatchSource:0}: Error finding container efdcc6e09fda7b23aada5f9aeacba2e9718f3e17832059d893625c99746ef72d: Status 404 returned error can't find the container with id efdcc6e09fda7b23aada5f9aeacba2e9718f3e17832059d893625c99746ef72d Jul 22 02:59:00.821315 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:00.821273 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-67vqv" Jul 22 02:59:00.996154 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:00.996120 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-67vqv"] Jul 22 02:59:00.999499 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:59:00.999472 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2835f18b_f325_418a_bf5c_5600606720f5.slice/crio-fab0ba5ca3c69bef98ba9b988bee7285520337450b797eb0548750dd8d47e709 WatchSource:0}: Error finding container fab0ba5ca3c69bef98ba9b988bee7285520337450b797eb0548750dd8d47e709: Status 404 returned error can't find the container with id fab0ba5ca3c69bef98ba9b988bee7285520337450b797eb0548750dd8d47e709 Jul 22 02:59:01.116481 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:01.116400 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:01.230768 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:01.230730 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-67vqv" event={"ID":"2835f18b-f325-418a-bf5c-5600606720f5","Type":"ContainerStarted","Data":"fab0ba5ca3c69bef98ba9b988bee7285520337450b797eb0548750dd8d47e709"} Jul 22 02:59:01.232305 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:01.232279 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7dd8f5b7fd-2c4kr" event={"ID":"fe5ad29f-dbb7-4353-80ea-60bf10139703","Type":"ContainerStarted","Data":"867ed54c34375c1718567a2a46c44db03422263a33ff0ddcd81e259e84b9b4af"} Jul 22 02:59:01.232413 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:01.232322 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7dd8f5b7fd-2c4kr" event={"ID":"fe5ad29f-dbb7-4353-80ea-60bf10139703","Type":"ContainerStarted","Data":"efdcc6e09fda7b23aada5f9aeacba2e9718f3e17832059d893625c99746ef72d"} Jul 22 02:59:01.232467 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:01.232411 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7dd8f5b7fd-2c4kr" Jul 22 02:59:01.253819 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:01.253764 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7dd8f5b7fd-2c4kr" podStartSLOduration=17.253740944 podStartE2EDuration="17.253740944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:58:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:59:01.25346304 +0000 UTC m=+51.985083887" watchObservedRunningTime="2026-07-22 02:59:01.253740944 +0000 UTC m=+51.985361789" Jul 22 02:59:01.314625 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:01.314593 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:01.513225 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:01.513172 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:01.929534 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:01.929507 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:02.043694 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:02.043662 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:02.510528 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:02.510495 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:02.615048 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:02.615008 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:02.918797 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:02.918768 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:03.242647 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:03.242557 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-67vqv" event={"ID":"2835f18b-f325-418a-bf5c-5600606720f5","Type":"ContainerStarted","Data":"5a6935043176d6b350a211d15063d1d8e3fa039be8423b6e3c5f54d3dedf3ee0"} Jul 22 02:59:03.242647 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:03.242634 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-67vqv" event={"ID":"2835f18b-f325-418a-bf5c-5600606720f5","Type":"ContainerStarted","Data":"3b5642e802ab6ae61851ef4e88f9d098a25604f963e95985437dea54c030a9b6"} Jul 22 02:59:03.243096 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:03.242711 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-67vqv" Jul 22 02:59:03.267918 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:03.267871 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-67vqv" podStartSLOduration=17.813535268 podStartE2EDuration="19.267855757s" totalImagesPullingTime="1.454320489s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:58:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:59:03.267383296 +0000 UTC m=+53.999004165" watchObservedRunningTime="2026-07-22 02:59:03.267855757 +0000 UTC m=+53.999476653" Jul 22 02:59:03.406498 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:03.406467 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:03.723745 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:03.723706 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:03.912785 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:03.912753 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:04.323570 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:04.323536 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:04.942719 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:04.942675 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:05.170134 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:05.170100 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:05.320374 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:05.320289 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:05.718391 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:05.718360 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:06.121265 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:06.121182 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:06.162402 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:06.162367 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c9f901fc-712c-4bc6-a5c8-2471837fcc18-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zhrpq\" (UID: \"c9f901fc-712c-4bc6-a5c8-2471837fcc18\") " pod="openshift-insights/insights-runtime-extractor-zhrpq" Jul 22 02:59:06.162402 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:06.162404 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c9f901fc-712c-4bc6-a5c8-2471837fcc18-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zhrpq\" (UID: \"c9f901fc-712c-4bc6-a5c8-2471837fcc18\") " pod="openshift-insights/insights-runtime-extractor-zhrpq" Jul 22 02:59:06.162983 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:06.162963 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c9f901fc-712c-4bc6-a5c8-2471837fcc18-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zhrpq\" (UID: \"c9f901fc-712c-4bc6-a5c8-2471837fcc18\") " pod="openshift-insights/insights-runtime-extractor-zhrpq" Jul 22 02:59:06.164632 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:06.164611 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c9f901fc-712c-4bc6-a5c8-2471837fcc18-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zhrpq\" (UID: \"c9f901fc-712c-4bc6-a5c8-2471837fcc18\") " pod="openshift-insights/insights-runtime-extractor-zhrpq" Jul 22 02:59:06.166647 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:06.166627 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zhrpq" Jul 22 02:59:06.302602 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:06.302572 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zhrpq"] Jul 22 02:59:06.305394 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:59:06.305372 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9f901fc_712c_4bc6_a5c8_2471837fcc18.slice/crio-09616fa72d20e71a2c2b360c76360449f60fed8ed7028fc3b4a21168c4cf2604 WatchSource:0}: Error finding container 09616fa72d20e71a2c2b360c76360449f60fed8ed7028fc3b4a21168c4cf2604: Status 404 returned error can't find the container with id 09616fa72d20e71a2c2b360c76360449f60fed8ed7028fc3b4a21168c4cf2604 Jul 22 02:59:06.328264 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:06.328229 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:06.533189 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:06.533158 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:07.190388 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:07.190351 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:07.258097 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:07.258049 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zhrpq" event={"ID":"c9f901fc-712c-4bc6-a5c8-2471837fcc18","Type":"ContainerStarted","Data":"09616fa72d20e71a2c2b360c76360449f60fed8ed7028fc3b4a21168c4cf2604"} Jul 22 02:59:07.265917 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:07.265887 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:08.140943 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:08.140902 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:08.244162 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:08.244130 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:08.320206 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:08.320153 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:08.446280 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:08.446169 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:08.650644 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:08.650610 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:08.766803 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:08.766714 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:09.046646 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:09.046563 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:09.136423 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:09.136386 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:09.238744 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:09.238710 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:09.473665 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:09.473631 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:09.575873 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:09.575838 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:09.646952 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:09.646902 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:09.723971 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:09.723892 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:09.834659 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:09.834607 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:09.979178 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:09.979105 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:10.156427 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:10.156398 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5rdkb" Jul 22 02:59:11.258863 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:11.258833 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:11.273248 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:11.273222 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zhrpq" event={"ID":"c9f901fc-712c-4bc6-a5c8-2471837fcc18","Type":"ContainerStarted","Data":"e73c35c258b6c436350053b92cb5f701f103eeb9f4325b9a312202db7d20b379"} Jul 22 02:59:11.363767 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:11.363703 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:12.711290 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:12.711250 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:13.248415 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:13.248390 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-67vqv" Jul 22 02:59:15.288931 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:15.288890 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zhrpq" event={"ID":"c9f901fc-712c-4bc6-a5c8-2471837fcc18","Type":"ContainerStarted","Data":"eb0881c9ccc63ff521a9b6a25ecbbd225c839f954e5739266cd6bb8495afcd28"} Jul 22 02:59:15.540764 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:15.540669 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75610210-6577-4ea8-b77d-a29a12c9a972-metrics-certs\") pod \"network-metrics-daemon-jqnjj\" (UID: \"75610210-6577-4ea8-b77d-a29a12c9a972\") " pod="openshift-multus/network-metrics-daemon-jqnjj" Jul 22 02:59:15.543605 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:15.543583 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 22 02:59:15.554241 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:15.554216 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75610210-6577-4ea8-b77d-a29a12c9a972-metrics-certs\") pod \"network-metrics-daemon-jqnjj\" (UID: \"75610210-6577-4ea8-b77d-a29a12c9a972\") " pod="openshift-multus/network-metrics-daemon-jqnjj" Jul 22 02:59:15.642060 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:15.642000 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5bd7c5db-509d-4f85-82ac-8e357591ae72-original-pull-secret\") pod \"global-pull-secret-syncer-lc5mn\" (UID: \"5bd7c5db-509d-4f85-82ac-8e357591ae72\") " pod="kube-system/global-pull-secret-syncer-lc5mn" Jul 22 02:59:15.645118 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:15.645088 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 22 02:59:15.655119 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:15.655091 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5bd7c5db-509d-4f85-82ac-8e357591ae72-original-pull-secret\") pod \"global-pull-secret-syncer-lc5mn\" (UID: \"5bd7c5db-509d-4f85-82ac-8e357591ae72\") " pod="kube-system/global-pull-secret-syncer-lc5mn" Jul 22 02:59:15.742594 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:15.742546 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh7j6\" (UniqueName: \"kubernetes.io/projected/0dc67dcb-94d3-439a-8b1b-6c238ccc621d-kube-api-access-dh7j6\") pod \"network-check-target-5xv9c\" (UID: \"0dc67dcb-94d3-439a-8b1b-6c238ccc621d\") " pod="openshift-network-diagnostics/network-check-target-5xv9c" Jul 22 02:59:15.745609 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:15.745571 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh7j6\" (UniqueName: \"kubernetes.io/projected/0dc67dcb-94d3-439a-8b1b-6c238ccc621d-kube-api-access-dh7j6\") pod \"network-check-target-5xv9c\" (UID: \"0dc67dcb-94d3-439a-8b1b-6c238ccc621d\") " pod="openshift-network-diagnostics/network-check-target-5xv9c" Jul 22 02:59:15.817603 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:15.817521 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sqq4d\"" Jul 22 02:59:15.824633 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:15.824573 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jqnjj" Jul 22 02:59:15.910616 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:15.910558 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nztgp\"" Jul 22 02:59:15.914380 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:15.914354 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lc5mn" Jul 22 02:59:15.918556 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:15.918505 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5xv9c" Jul 22 02:59:16.027317 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:16.027211 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jqnjj"] Jul 22 02:59:16.109172 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:16.109141 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lc5mn"] Jul 22 02:59:16.112882 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:59:16.112849 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bd7c5db_509d_4f85_82ac_8e357591ae72.slice/crio-3d2a38119b2765272a904ba1574eb45a5bce48c1c2c721caf600bbeb8dcace94 WatchSource:0}: Error finding container 3d2a38119b2765272a904ba1574eb45a5bce48c1c2c721caf600bbeb8dcace94: Status 404 returned error can't find the container with id 3d2a38119b2765272a904ba1574eb45a5bce48c1c2c721caf600bbeb8dcace94 Jul 22 02:59:16.127812 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:16.127783 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5xv9c"] Jul 22 02:59:16.132097 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:59:16.132064 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dc67dcb_94d3_439a_8b1b_6c238ccc621d.slice/crio-e98622d3b2d21a0a61e9bad71421ece725f4e58ebbea9f7d3a3d9a56da80f6a5 WatchSource:0}: Error finding container e98622d3b2d21a0a61e9bad71421ece725f4e58ebbea9f7d3a3d9a56da80f6a5: Status 404 returned error can't find the container with id e98622d3b2d21a0a61e9bad71421ece725f4e58ebbea9f7d3a3d9a56da80f6a5 Jul 22 02:59:16.293406 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:16.293358 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jqnjj" event={"ID":"75610210-6577-4ea8-b77d-a29a12c9a972","Type":"ContainerStarted","Data":"a8aa1e9273c4a587f120a74eab26ef61b525751d716af609905204da00efc802"} Jul 22 02:59:16.295054 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:16.295007 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5xv9c" event={"ID":"0dc67dcb-94d3-439a-8b1b-6c238ccc621d","Type":"ContainerStarted","Data":"e5948a85528136834f47df6dcab464a74beff376f003b39e9071c6a571b92478"} Jul 22 02:59:16.295200 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:16.295080 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5xv9c" event={"ID":"0dc67dcb-94d3-439a-8b1b-6c238ccc621d","Type":"ContainerStarted","Data":"e98622d3b2d21a0a61e9bad71421ece725f4e58ebbea9f7d3a3d9a56da80f6a5"} Jul 22 02:59:16.295200 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:16.295132 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-5xv9c" Jul 22 02:59:16.296480 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:16.296454 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lc5mn" event={"ID":"5bd7c5db-509d-4f85-82ac-8e357591ae72","Type":"ContainerStarted","Data":"3d2a38119b2765272a904ba1574eb45a5bce48c1c2c721caf600bbeb8dcace94"} Jul 22 02:59:16.313714 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:16.313654 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-5xv9c" podStartSLOduration=66.313630657 podStartE2EDuration="1m6.313630657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:58:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:59:16.312660987 +0000 UTC m=+67.044281889" watchObservedRunningTime="2026-07-22 02:59:16.313630657 +0000 UTC m=+67.045251508" Jul 22 02:59:18.475370 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:18.475240 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7dd8f5b7fd-2c4kr"] Jul 22 02:59:18.476307 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:18.476261 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f7dc679dd-6xvrs"] Jul 22 02:59:18.481586 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:18.481472 2582 patch_prober.go:28] interesting pod/image-registry-7dd8f5b7fd-2c4kr container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 22 02:59:18.481820 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:18.481559 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7dd8f5b7fd-2c4kr" podUID="fe5ad29f-dbb7-4353-80ea-60bf10139703" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 22 02:59:18.505313 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:18.505268 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f7dc679dd-6xvrs"] Jul 22 02:59:18.505487 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:18.505395 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f7dc679dd-6xvrs" Jul 22 02:59:18.509347 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:18.509324 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 22 02:59:18.509479 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:18.509365 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-7pplv\"" Jul 22 02:59:18.509575 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:18.509555 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 22 02:59:18.509945 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:18.509927 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 22 02:59:18.515467 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:18.515445 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 22 02:59:18.563741 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:18.563706 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5e6b772d-5a8a-4da1-90bd-5f7fd944f1af-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6f7dc679dd-6xvrs\" (UID: \"5e6b772d-5a8a-4da1-90bd-5f7fd944f1af\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f7dc679dd-6xvrs" Jul 22 02:59:18.563926 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:18.563771 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s45pb\" (UniqueName: \"kubernetes.io/projected/5e6b772d-5a8a-4da1-90bd-5f7fd944f1af-kube-api-access-s45pb\") pod \"managed-serviceaccount-addon-agent-6f7dc679dd-6xvrs\" (UID: \"5e6b772d-5a8a-4da1-90bd-5f7fd944f1af\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f7dc679dd-6xvrs" Jul 22 02:59:18.566636 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:18.566609 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:18.577881 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:18.577843 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-67748fc486-qc6lw"] Jul 22 02:59:18.580071 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:18.580042 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67748fc486-qc6lw" Jul 22 02:59:18.599200 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:18.599171 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-67748fc486-qc6lw"] Jul 22 02:59:18.664362 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:18.664325 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4efe03aa-d6c0-4702-9f2d-517a51158c53-installation-pull-secrets\") pod \"image-registry-67748fc486-qc6lw\" (UID: \"4efe03aa-d6c0-4702-9f2d-517a51158c53\") " pod="openshift-image-registry/image-registry-67748fc486-qc6lw" Jul 22 02:59:18.664564 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:18.664373 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4efe03aa-d6c0-4702-9f2d-517a51158c53-image-registry-private-configuration\") pod \"image-registry-67748fc486-qc6lw\" (UID: \"4efe03aa-d6c0-4702-9f2d-517a51158c53\") " pod="openshift-image-registry/image-registry-67748fc486-qc6lw" Jul 22 02:59:18.664564 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:18.664403 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4efe03aa-d6c0-4702-9f2d-517a51158c53-registry-tls\") pod \"image-registry-67748fc486-qc6lw\" (UID: \"4efe03aa-d6c0-4702-9f2d-517a51158c53\") " pod="openshift-image-registry/image-registry-67748fc486-qc6lw" Jul 22 02:59:18.664564 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:18.664465 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5e6b772d-5a8a-4da1-90bd-5f7fd944f1af-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6f7dc679dd-6xvrs\" (UID: \"5e6b772d-5a8a-4da1-90bd-5f7fd944f1af\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f7dc679dd-6xvrs" Jul 22 02:59:18.664564 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:18.664497 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4efe03aa-d6c0-4702-9f2d-517a51158c53-bound-sa-token\") pod \"image-registry-67748fc486-qc6lw\" (UID: \"4efe03aa-d6c0-4702-9f2d-517a51158c53\") " pod="openshift-image-registry/image-registry-67748fc486-qc6lw" Jul 22 02:59:18.664564 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:18.664525 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4efe03aa-d6c0-4702-9f2d-517a51158c53-trusted-ca\") pod \"image-registry-67748fc486-qc6lw\" (UID: \"4efe03aa-d6c0-4702-9f2d-517a51158c53\") " pod="openshift-image-registry/image-registry-67748fc486-qc6lw" Jul 22 02:59:18.664564 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:18.664554 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s45pb\" (UniqueName: \"kubernetes.io/projected/5e6b772d-5a8a-4da1-90bd-5f7fd944f1af-kube-api-access-s45pb\") pod \"managed-serviceaccount-addon-agent-6f7dc679dd-6xvrs\" (UID: \"5e6b772d-5a8a-4da1-90bd-5f7fd944f1af\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f7dc679dd-6xvrs" Jul 22 02:59:18.664839 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:18.664570 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4efe03aa-d6c0-4702-9f2d-517a51158c53-registry-certificates\") pod \"image-registry-67748fc486-qc6lw\" (UID: \"4efe03aa-d6c0-4702-9f2d-517a51158c53\") " pod="openshift-image-registry/image-registry-67748fc486-qc6lw" Jul 22 02:59:18.664839 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:18.664699 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z68lq\" (UniqueName: \"kubernetes.io/projected/4efe03aa-d6c0-4702-9f2d-517a51158c53-kube-api-access-z68lq\") pod \"image-registry-67748fc486-qc6lw\" (UID: \"4efe03aa-d6c0-4702-9f2d-517a51158c53\") " pod="openshift-image-registry/image-registry-67748fc486-qc6lw" Jul 22 02:59:18.664839 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:18.664757 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4efe03aa-d6c0-4702-9f2d-517a51158c53-ca-trust-extracted\") pod \"image-registry-67748fc486-qc6lw\" (UID: \"4efe03aa-d6c0-4702-9f2d-517a51158c53\") " pod="openshift-image-registry/image-registry-67748fc486-qc6lw" Jul 22 02:59:18.666201 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:18.666156 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:18.667869 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:18.667844 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5e6b772d-5a8a-4da1-90bd-5f7fd944f1af-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6f7dc679dd-6xvrs\" (UID: \"5e6b772d-5a8a-4da1-90bd-5f7fd944f1af\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f7dc679dd-6xvrs" Jul 22 02:59:18.680797 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:18.680767 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s45pb\" (UniqueName: \"kubernetes.io/projected/5e6b772d-5a8a-4da1-90bd-5f7fd944f1af-kube-api-access-s45pb\") pod \"managed-serviceaccount-addon-agent-6f7dc679dd-6xvrs\" (UID: \"5e6b772d-5a8a-4da1-90bd-5f7fd944f1af\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f7dc679dd-6xvrs" Jul 22 02:59:18.763285 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:18.763200 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:18.766053 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:18.766025 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z68lq\" (UniqueName: \"kubernetes.io/projected/4efe03aa-d6c0-4702-9f2d-517a51158c53-kube-api-access-z68lq\") pod \"image-registry-67748fc486-qc6lw\" (UID: \"4efe03aa-d6c0-4702-9f2d-517a51158c53\") " pod="openshift-image-registry/image-registry-67748fc486-qc6lw" Jul 22 02:59:18.766210 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:18.766081 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4efe03aa-d6c0-4702-9f2d-517a51158c53-ca-trust-extracted\") pod \"image-registry-67748fc486-qc6lw\" (UID: \"4efe03aa-d6c0-4702-9f2d-517a51158c53\") " pod="openshift-image-registry/image-registry-67748fc486-qc6lw" Jul 22 02:59:18.766210 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:18.766108 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4efe03aa-d6c0-4702-9f2d-517a51158c53-installation-pull-secrets\") pod \"image-registry-67748fc486-qc6lw\" (UID: \"4efe03aa-d6c0-4702-9f2d-517a51158c53\") " pod="openshift-image-registry/image-registry-67748fc486-qc6lw" Jul 22 02:59:18.766210 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:18.766138 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4efe03aa-d6c0-4702-9f2d-517a51158c53-image-registry-private-configuration\") pod \"image-registry-67748fc486-qc6lw\" (UID: \"4efe03aa-d6c0-4702-9f2d-517a51158c53\") " pod="openshift-image-registry/image-registry-67748fc486-qc6lw" Jul 22 02:59:18.766210 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:18.766166 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4efe03aa-d6c0-4702-9f2d-517a51158c53-registry-tls\") pod \"image-registry-67748fc486-qc6lw\" (UID: \"4efe03aa-d6c0-4702-9f2d-517a51158c53\") " pod="openshift-image-registry/image-registry-67748fc486-qc6lw" Jul 22 02:59:18.766210 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:18.766202 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4efe03aa-d6c0-4702-9f2d-517a51158c53-bound-sa-token\") pod \"image-registry-67748fc486-qc6lw\" (UID: \"4efe03aa-d6c0-4702-9f2d-517a51158c53\") " pod="openshift-image-registry/image-registry-67748fc486-qc6lw" Jul 22 02:59:18.766451 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:18.766228 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4efe03aa-d6c0-4702-9f2d-517a51158c53-trusted-ca\") pod \"image-registry-67748fc486-qc6lw\" (UID: \"4efe03aa-d6c0-4702-9f2d-517a51158c53\") " pod="openshift-image-registry/image-registry-67748fc486-qc6lw" Jul 22 02:59:18.766451 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:18.766256 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4efe03aa-d6c0-4702-9f2d-517a51158c53-registry-certificates\") pod \"image-registry-67748fc486-qc6lw\" (UID: \"4efe03aa-d6c0-4702-9f2d-517a51158c53\") " pod="openshift-image-registry/image-registry-67748fc486-qc6lw" Jul 22 02:59:18.766558 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:18.766454 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4efe03aa-d6c0-4702-9f2d-517a51158c53-ca-trust-extracted\") pod \"image-registry-67748fc486-qc6lw\" (UID: \"4efe03aa-d6c0-4702-9f2d-517a51158c53\") " pod="openshift-image-registry/image-registry-67748fc486-qc6lw" Jul 22 02:59:18.767249 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:18.767200 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4efe03aa-d6c0-4702-9f2d-517a51158c53-registry-certificates\") pod \"image-registry-67748fc486-qc6lw\" (UID: \"4efe03aa-d6c0-4702-9f2d-517a51158c53\") " pod="openshift-image-registry/image-registry-67748fc486-qc6lw" Jul 22 02:59:18.767677 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:18.767652 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4efe03aa-d6c0-4702-9f2d-517a51158c53-trusted-ca\") pod \"image-registry-67748fc486-qc6lw\" (UID: \"4efe03aa-d6c0-4702-9f2d-517a51158c53\") " pod="openshift-image-registry/image-registry-67748fc486-qc6lw" Jul 22 02:59:18.769521 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:18.769469 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4efe03aa-d6c0-4702-9f2d-517a51158c53-installation-pull-secrets\") pod \"image-registry-67748fc486-qc6lw\" (UID: \"4efe03aa-d6c0-4702-9f2d-517a51158c53\") " pod="openshift-image-registry/image-registry-67748fc486-qc6lw" Jul 22 02:59:18.769627 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:18.769547 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4efe03aa-d6c0-4702-9f2d-517a51158c53-image-registry-private-configuration\") pod \"image-registry-67748fc486-qc6lw\" (UID: \"4efe03aa-d6c0-4702-9f2d-517a51158c53\") " pod="openshift-image-registry/image-registry-67748fc486-qc6lw" Jul 22 02:59:18.769852 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:18.769819 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4efe03aa-d6c0-4702-9f2d-517a51158c53-registry-tls\") pod \"image-registry-67748fc486-qc6lw\" (UID: \"4efe03aa-d6c0-4702-9f2d-517a51158c53\") " pod="openshift-image-registry/image-registry-67748fc486-qc6lw" Jul 22 02:59:18.781549 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:18.781521 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z68lq\" (UniqueName: \"kubernetes.io/projected/4efe03aa-d6c0-4702-9f2d-517a51158c53-kube-api-access-z68lq\") pod \"image-registry-67748fc486-qc6lw\" (UID: \"4efe03aa-d6c0-4702-9f2d-517a51158c53\") " pod="openshift-image-registry/image-registry-67748fc486-qc6lw" Jul 22 02:59:18.781549 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:18.781540 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4efe03aa-d6c0-4702-9f2d-517a51158c53-bound-sa-token\") pod \"image-registry-67748fc486-qc6lw\" (UID: \"4efe03aa-d6c0-4702-9f2d-517a51158c53\") " pod="openshift-image-registry/image-registry-67748fc486-qc6lw" Jul 22 02:59:18.830190 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:18.830153 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f7dc679dd-6xvrs" Jul 22 02:59:18.854313 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:18.854266 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:18.891660 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:18.891623 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67748fc486-qc6lw" Jul 22 02:59:18.932240 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:18.932193 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:20.318555 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:20.318528 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zhrpq" event={"ID":"c9f901fc-712c-4bc6-a5c8-2471837fcc18","Type":"ContainerStarted","Data":"15b5365d50f6d0b8bf27e8a28be6c562f9230850b4446e9cd19ddb7319a45671"} Jul 22 02:59:20.320657 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:20.320616 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jqnjj" event={"ID":"75610210-6577-4ea8-b77d-a29a12c9a972","Type":"ContainerStarted","Data":"bb298f034023db124a07bd1487b6776c5a41ca96131b7fe602481d2ec111d0dc"} Jul 22 02:59:20.337023 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:20.336976 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-67748fc486-qc6lw"] Jul 22 02:59:20.343197 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:59:20.343164 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4efe03aa_d6c0_4702_9f2d_517a51158c53.slice/crio-21f80a910219badc71af2b96286399fcc058567469211385d76abbb3730a6598 WatchSource:0}: Error finding container 21f80a910219badc71af2b96286399fcc058567469211385d76abbb3730a6598: Status 404 returned error can't find the container with id 21f80a910219badc71af2b96286399fcc058567469211385d76abbb3730a6598 Jul 22 02:59:20.344007 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:20.343958 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-zhrpq" podStartSLOduration=16.602740987 podStartE2EDuration="30.343940237s" totalImagesPullingTime="13.74119925s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 02:58:50 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:59:20.34309102 +0000 UTC m=+71.074711884" watchObservedRunningTime="2026-07-22 02:59:20.343940237 +0000 UTC m=+71.075561084" Jul 22 02:59:20.349204 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:20.348031 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f7dc679dd-6xvrs"] Jul 22 02:59:20.477402 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:20.477327 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:20.573291 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:20.573254 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:20.654549 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:20.654502 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:21.022269 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:21.022236 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:21.328529 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:21.328441 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lc5mn" event={"ID":"5bd7c5db-509d-4f85-82ac-8e357591ae72","Type":"ContainerStarted","Data":"703dd55835f62aac0e1475b1a7612fe84add770bc2e9cdd0f4a607b784852178"} Jul 22 02:59:21.329702 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:21.329664 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f7dc679dd-6xvrs" event={"ID":"5e6b772d-5a8a-4da1-90bd-5f7fd944f1af","Type":"ContainerStarted","Data":"f772e7b4e6d2d8719bc99e6c07fe4443f28eec9b5d9c5e7189d35fd09d024ab0"} Jul 22 02:59:21.331249 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:21.331224 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67748fc486-qc6lw" event={"ID":"4efe03aa-d6c0-4702-9f2d-517a51158c53","Type":"ContainerStarted","Data":"b0575d0eed78f5c6c918ff99b2bdf62fb35c6414a175d924977de187f009041a"} Jul 22 02:59:21.331397 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:21.331263 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67748fc486-qc6lw" event={"ID":"4efe03aa-d6c0-4702-9f2d-517a51158c53","Type":"ContainerStarted","Data":"21f80a910219badc71af2b96286399fcc058567469211385d76abbb3730a6598"} Jul 22 02:59:21.331397 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:21.331284 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-67748fc486-qc6lw" Jul 22 02:59:21.334601 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:21.334577 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jqnjj" event={"ID":"75610210-6577-4ea8-b77d-a29a12c9a972","Type":"ContainerStarted","Data":"a12b2e6e511675b62abd46aebed01ccde970faaf58564a032b88d545765946da"} Jul 22 02:59:21.347177 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:21.347126 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-lc5mn" podStartSLOduration=67.263769868 podStartE2EDuration="1m11.347109224s" totalImagesPullingTime="4.083339356s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 02:58:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:59:21.346809669 +0000 UTC m=+72.078430535" watchObservedRunningTime="2026-07-22 02:59:21.347109224 +0000 UTC m=+72.078730068" Jul 22 02:59:21.366576 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:21.366529 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-jqnjj" podStartSLOduration=67.244509197 podStartE2EDuration="1m11.366511564s" totalImagesPullingTime="4.122002367s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:58:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:59:21.366134308 +0000 UTC m=+72.097755189" watchObservedRunningTime="2026-07-22 02:59:21.366511564 +0000 UTC m=+72.098132410" Jul 22 02:59:21.385788 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:21.385730 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-67748fc486-qc6lw" podStartSLOduration=3.385716188 podStartE2EDuration="3.385716188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:59:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:59:21.385249538 +0000 UTC m=+72.116870385" watchObservedRunningTime="2026-07-22 02:59:21.385716188 +0000 UTC m=+72.117337034" Jul 22 02:59:21.472033 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:21.471982 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:21.537001 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:21.536967 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:23.847287 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:23.847259 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:23.951318 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:23.951286 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:25.863102 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:25.863069 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5494b569cc-m4xl8"] Jul 22 02:59:25.865659 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:25.865642 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5494b569cc-m4xl8" Jul 22 02:59:25.870025 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:25.869985 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-srtb2\"" Jul 22 02:59:25.870025 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:25.870002 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 22 02:59:25.870197 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:25.870002 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 22 02:59:25.870197 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:25.870051 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 22 02:59:25.870197 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:25.869985 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 22 02:59:25.870197 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:25.870063 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 22 02:59:25.870386 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:25.870281 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 22 02:59:25.870878 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:25.870863 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 22 02:59:25.874051 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:25.874031 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 22 02:59:25.878322 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:25.878302 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5494b569cc-m4xl8"] Jul 22 02:59:26.020900 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:26.020865 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bab72638-4717-4ea0-8a68-b4d9a1138154-console-oauth-config\") pod \"console-5494b569cc-m4xl8\" (UID: \"bab72638-4717-4ea0-8a68-b4d9a1138154\") " pod="openshift-console/console-5494b569cc-m4xl8" Jul 22 02:59:26.021096 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:26.020909 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bab72638-4717-4ea0-8a68-b4d9a1138154-oauth-serving-cert\") pod \"console-5494b569cc-m4xl8\" (UID: \"bab72638-4717-4ea0-8a68-b4d9a1138154\") " pod="openshift-console/console-5494b569cc-m4xl8" Jul 22 02:59:26.021096 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:26.020929 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tts94\" (UniqueName: \"kubernetes.io/projected/bab72638-4717-4ea0-8a68-b4d9a1138154-kube-api-access-tts94\") pod \"console-5494b569cc-m4xl8\" (UID: \"bab72638-4717-4ea0-8a68-b4d9a1138154\") " pod="openshift-console/console-5494b569cc-m4xl8" Jul 22 02:59:26.021096 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:26.020988 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bab72638-4717-4ea0-8a68-b4d9a1138154-trusted-ca-bundle\") pod \"console-5494b569cc-m4xl8\" (UID: \"bab72638-4717-4ea0-8a68-b4d9a1138154\") " pod="openshift-console/console-5494b569cc-m4xl8" Jul 22 02:59:26.021096 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:26.021055 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bab72638-4717-4ea0-8a68-b4d9a1138154-console-serving-cert\") pod \"console-5494b569cc-m4xl8\" (UID: \"bab72638-4717-4ea0-8a68-b4d9a1138154\") " pod="openshift-console/console-5494b569cc-m4xl8" Jul 22 02:59:26.021282 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:26.021099 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bab72638-4717-4ea0-8a68-b4d9a1138154-console-config\") pod \"console-5494b569cc-m4xl8\" (UID: \"bab72638-4717-4ea0-8a68-b4d9a1138154\") " pod="openshift-console/console-5494b569cc-m4xl8" Jul 22 02:59:26.021282 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:26.021125 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bab72638-4717-4ea0-8a68-b4d9a1138154-service-ca\") pod \"console-5494b569cc-m4xl8\" (UID: \"bab72638-4717-4ea0-8a68-b4d9a1138154\") " pod="openshift-console/console-5494b569cc-m4xl8" Jul 22 02:59:26.121901 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:26.121813 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bab72638-4717-4ea0-8a68-b4d9a1138154-console-serving-cert\") pod \"console-5494b569cc-m4xl8\" (UID: \"bab72638-4717-4ea0-8a68-b4d9a1138154\") " pod="openshift-console/console-5494b569cc-m4xl8" Jul 22 02:59:26.121901 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:26.121860 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bab72638-4717-4ea0-8a68-b4d9a1138154-console-config\") pod \"console-5494b569cc-m4xl8\" (UID: \"bab72638-4717-4ea0-8a68-b4d9a1138154\") " pod="openshift-console/console-5494b569cc-m4xl8" Jul 22 02:59:26.121901 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:26.121880 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bab72638-4717-4ea0-8a68-b4d9a1138154-service-ca\") pod \"console-5494b569cc-m4xl8\" (UID: \"bab72638-4717-4ea0-8a68-b4d9a1138154\") " pod="openshift-console/console-5494b569cc-m4xl8" Jul 22 02:59:26.122240 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:26.121914 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bab72638-4717-4ea0-8a68-b4d9a1138154-console-oauth-config\") pod \"console-5494b569cc-m4xl8\" (UID: \"bab72638-4717-4ea0-8a68-b4d9a1138154\") " pod="openshift-console/console-5494b569cc-m4xl8" Jul 22 02:59:26.122240 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:26.121956 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bab72638-4717-4ea0-8a68-b4d9a1138154-oauth-serving-cert\") pod \"console-5494b569cc-m4xl8\" (UID: \"bab72638-4717-4ea0-8a68-b4d9a1138154\") " pod="openshift-console/console-5494b569cc-m4xl8" Jul 22 02:59:26.122240 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:26.121981 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tts94\" (UniqueName: \"kubernetes.io/projected/bab72638-4717-4ea0-8a68-b4d9a1138154-kube-api-access-tts94\") pod \"console-5494b569cc-m4xl8\" (UID: \"bab72638-4717-4ea0-8a68-b4d9a1138154\") " pod="openshift-console/console-5494b569cc-m4xl8" Jul 22 02:59:26.122240 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:26.122029 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bab72638-4717-4ea0-8a68-b4d9a1138154-trusted-ca-bundle\") pod \"console-5494b569cc-m4xl8\" (UID: \"bab72638-4717-4ea0-8a68-b4d9a1138154\") " pod="openshift-console/console-5494b569cc-m4xl8" Jul 22 02:59:26.122748 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:26.122719 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bab72638-4717-4ea0-8a68-b4d9a1138154-oauth-serving-cert\") pod \"console-5494b569cc-m4xl8\" (UID: \"bab72638-4717-4ea0-8a68-b4d9a1138154\") " pod="openshift-console/console-5494b569cc-m4xl8" Jul 22 02:59:26.122859 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:26.122778 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bab72638-4717-4ea0-8a68-b4d9a1138154-console-config\") pod \"console-5494b569cc-m4xl8\" (UID: \"bab72638-4717-4ea0-8a68-b4d9a1138154\") " pod="openshift-console/console-5494b569cc-m4xl8" Jul 22 02:59:26.122859 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:26.122807 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bab72638-4717-4ea0-8a68-b4d9a1138154-service-ca\") pod \"console-5494b569cc-m4xl8\" (UID: \"bab72638-4717-4ea0-8a68-b4d9a1138154\") " pod="openshift-console/console-5494b569cc-m4xl8" Jul 22 02:59:26.123033 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:26.122937 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bab72638-4717-4ea0-8a68-b4d9a1138154-trusted-ca-bundle\") pod \"console-5494b569cc-m4xl8\" (UID: \"bab72638-4717-4ea0-8a68-b4d9a1138154\") " pod="openshift-console/console-5494b569cc-m4xl8" Jul 22 02:59:26.124468 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:26.124440 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bab72638-4717-4ea0-8a68-b4d9a1138154-console-oauth-config\") pod \"console-5494b569cc-m4xl8\" (UID: \"bab72638-4717-4ea0-8a68-b4d9a1138154\") " pod="openshift-console/console-5494b569cc-m4xl8" Jul 22 02:59:26.124571 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:26.124442 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bab72638-4717-4ea0-8a68-b4d9a1138154-console-serving-cert\") pod \"console-5494b569cc-m4xl8\" (UID: \"bab72638-4717-4ea0-8a68-b4d9a1138154\") " pod="openshift-console/console-5494b569cc-m4xl8" Jul 22 02:59:26.135523 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:26.135496 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tts94\" (UniqueName: \"kubernetes.io/projected/bab72638-4717-4ea0-8a68-b4d9a1138154-kube-api-access-tts94\") pod \"console-5494b569cc-m4xl8\" (UID: \"bab72638-4717-4ea0-8a68-b4d9a1138154\") " pod="openshift-console/console-5494b569cc-m4xl8" Jul 22 02:59:26.175231 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:26.175177 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5494b569cc-m4xl8" Jul 22 02:59:26.309035 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:26.308982 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5494b569cc-m4xl8"] Jul 22 02:59:26.312543 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:59:26.312517 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbab72638_4717_4ea0_8a68_b4d9a1138154.slice/crio-32d1deded5b25dfe93b798ffe2bdefe44f4d35cb9c0573086ba7210464ff4cb8 WatchSource:0}: Error finding container 32d1deded5b25dfe93b798ffe2bdefe44f4d35cb9c0573086ba7210464ff4cb8: Status 404 returned error can't find the container with id 32d1deded5b25dfe93b798ffe2bdefe44f4d35cb9c0573086ba7210464ff4cb8 Jul 22 02:59:26.351718 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:26.351680 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5494b569cc-m4xl8" event={"ID":"bab72638-4717-4ea0-8a68-b4d9a1138154","Type":"ContainerStarted","Data":"32d1deded5b25dfe93b798ffe2bdefe44f4d35cb9c0573086ba7210464ff4cb8"} Jul 22 02:59:26.568729 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:26.568699 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:26.672448 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:26.672415 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:26.761399 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:26.761358 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:27.532806 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:27.532770 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:27.647919 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:27.647886 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:27.727735 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:27.727697 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-8dht8"] Jul 22 02:59:27.730043 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:27.730004 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-8dht8" Jul 22 02:59:27.732601 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:27.732574 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 22 02:59:27.733503 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:27.733477 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 22 02:59:27.733620 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:27.733510 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 22 02:59:27.733620 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:27.733492 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 22 02:59:27.733620 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:27.733537 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 22 02:59:27.733620 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:27.733488 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-zcn24\"" Jul 22 02:59:27.741636 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:27.741612 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-8dht8"] Jul 22 02:59:27.750118 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:27.750090 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:27.838366 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:27.838273 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2f209b19-94fe-48ba-9d66-81886ca09b08-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-8dht8\" (UID: \"2f209b19-94fe-48ba-9d66-81886ca09b08\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-8dht8" Jul 22 02:59:27.838366 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:27.838336 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2f209b19-94fe-48ba-9d66-81886ca09b08-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-8dht8\" (UID: \"2f209b19-94fe-48ba-9d66-81886ca09b08\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-8dht8" Jul 22 02:59:27.838587 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:27.838377 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2f209b19-94fe-48ba-9d66-81886ca09b08-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-8dht8\" (UID: \"2f209b19-94fe-48ba-9d66-81886ca09b08\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-8dht8" Jul 22 02:59:27.838587 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:27.838405 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfnfn\" (UniqueName: \"kubernetes.io/projected/2f209b19-94fe-48ba-9d66-81886ca09b08-kube-api-access-nfnfn\") pod \"prometheus-operator-5945b5cfd-8dht8\" (UID: \"2f209b19-94fe-48ba-9d66-81886ca09b08\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-8dht8" Jul 22 02:59:27.861618 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:27.861589 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:27.939420 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:27.939376 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2f209b19-94fe-48ba-9d66-81886ca09b08-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-8dht8\" (UID: \"2f209b19-94fe-48ba-9d66-81886ca09b08\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-8dht8" Jul 22 02:59:27.939601 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:27.939453 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2f209b19-94fe-48ba-9d66-81886ca09b08-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-8dht8\" (UID: \"2f209b19-94fe-48ba-9d66-81886ca09b08\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-8dht8" Jul 22 02:59:27.939601 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:27.939482 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfnfn\" (UniqueName: \"kubernetes.io/projected/2f209b19-94fe-48ba-9d66-81886ca09b08-kube-api-access-nfnfn\") pod \"prometheus-operator-5945b5cfd-8dht8\" (UID: \"2f209b19-94fe-48ba-9d66-81886ca09b08\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-8dht8" Jul 22 02:59:27.939601 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:27.939511 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2f209b19-94fe-48ba-9d66-81886ca09b08-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-8dht8\" (UID: \"2f209b19-94fe-48ba-9d66-81886ca09b08\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-8dht8" Jul 22 02:59:27.940174 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:27.940152 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2f209b19-94fe-48ba-9d66-81886ca09b08-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-8dht8\" (UID: \"2f209b19-94fe-48ba-9d66-81886ca09b08\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-8dht8" Jul 22 02:59:27.941831 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:27.941751 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2f209b19-94fe-48ba-9d66-81886ca09b08-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-8dht8\" (UID: \"2f209b19-94fe-48ba-9d66-81886ca09b08\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-8dht8" Jul 22 02:59:27.941991 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:27.941973 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2f209b19-94fe-48ba-9d66-81886ca09b08-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-8dht8\" (UID: \"2f209b19-94fe-48ba-9d66-81886ca09b08\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-8dht8" Jul 22 02:59:27.948195 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:27.948173 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfnfn\" (UniqueName: \"kubernetes.io/projected/2f209b19-94fe-48ba-9d66-81886ca09b08-kube-api-access-nfnfn\") pod \"prometheus-operator-5945b5cfd-8dht8\" (UID: \"2f209b19-94fe-48ba-9d66-81886ca09b08\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-8dht8" Jul 22 02:59:28.042453 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:28.042414 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-8dht8" Jul 22 02:59:28.481476 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:28.481450 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7dd8f5b7fd-2c4kr" Jul 22 02:59:29.061379 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:29.061346 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:29.127427 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:29.127382 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-8dht8"] Jul 22 02:59:29.130969 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:59:29.130946 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f209b19_94fe_48ba_9d66_81886ca09b08.slice/crio-7227f824c4853b3a974da757cd75e026146f5c62bd48a2a119fe626d48d8a9e8 WatchSource:0}: Error finding container 7227f824c4853b3a974da757cd75e026146f5c62bd48a2a119fe626d48d8a9e8: Status 404 returned error can't find the container with id 7227f824c4853b3a974da757cd75e026146f5c62bd48a2a119fe626d48d8a9e8 Jul 22 02:59:29.364697 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:29.364606 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5494b569cc-m4xl8" event={"ID":"bab72638-4717-4ea0-8a68-b4d9a1138154","Type":"ContainerStarted","Data":"757893d35ddfaa48b8cf9aca3d9862e345a52426c6ed47a7741c3df9ff52bead"} Jul 22 02:59:29.365704 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:29.365680 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-8dht8" event={"ID":"2f209b19-94fe-48ba-9d66-81886ca09b08","Type":"ContainerStarted","Data":"7227f824c4853b3a974da757cd75e026146f5c62bd48a2a119fe626d48d8a9e8"} Jul 22 02:59:29.383964 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:29.383905 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5494b569cc-m4xl8" podStartSLOduration=1.667661616 podStartE2EDuration="4.38388765s" totalImagesPullingTime="2.716226034s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:59:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:59:29.383085759 +0000 UTC m=+80.114706608" watchObservedRunningTime="2026-07-22 02:59:29.38388765 +0000 UTC m=+80.115508494" Jul 22 02:59:30.721505 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:30.721420 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:31.374508 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:31.374391 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-8dht8" event={"ID":"2f209b19-94fe-48ba-9d66-81886ca09b08","Type":"ContainerStarted","Data":"85c6ad9c2ae3766bb774eafec46781d099ca7d80d2969992de0c17f05b25db56"} Jul 22 02:59:31.374508 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:31.374494 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-8dht8" event={"ID":"2f209b19-94fe-48ba-9d66-81886ca09b08","Type":"ContainerStarted","Data":"0d353889d2ef47674ffcfcce8ad8f3f19dc12db25c5493eba05eaa184fc387f9"} Jul 22 02:59:31.396086 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:31.396001 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-5945b5cfd-8dht8" podStartSLOduration=3.120146517 podStartE2EDuration="4.395983807s" totalImagesPullingTime="1.27583729s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:59:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:59:31.395420201 +0000 UTC m=+82.127041068" watchObservedRunningTime="2026-07-22 02:59:31.395983807 +0000 UTC m=+82.127604653" Jul 22 02:59:31.507246 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:31.507208 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:31.640919 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:31.640834 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:32.850291 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:32.850255 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:33.076871 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:33.076830 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:33.087691 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:33.087653 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-lsxjz"] Jul 22 02:59:33.099519 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:33.099480 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-lsxjz" Jul 22 02:59:33.102642 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:33.102566 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 22 02:59:33.103448 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:33.103215 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 22 02:59:33.104180 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:33.104000 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-gqmlh\"" Jul 22 02:59:33.104737 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:33.104644 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-lsxjz"] Jul 22 02:59:33.105771 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:33.105747 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-ffjds"] Jul 22 02:59:33.111307 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:33.111285 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ffjds" Jul 22 02:59:33.113769 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:33.113752 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 22 02:59:33.113953 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:33.113928 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 22 02:59:33.114066 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:33.113981 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 22 02:59:33.114066 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:33.113936 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-ffkrv\"" Jul 22 02:59:33.169561 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:33.169524 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:33.183793 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:33.183755 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6d641c24-367d-4e6b-9402-e61e594f8811-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-lsxjz\" (UID: \"6d641c24-367d-4e6b-9402-e61e594f8811\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-lsxjz" Jul 22 02:59:33.183975 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:33.183810 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6d641c24-367d-4e6b-9402-e61e594f8811-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-lsxjz\" (UID: \"6d641c24-367d-4e6b-9402-e61e594f8811\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-lsxjz" Jul 22 02:59:33.183975 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:33.183888 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8zhl\" (UniqueName: \"kubernetes.io/projected/6d641c24-367d-4e6b-9402-e61e594f8811-kube-api-access-v8zhl\") pod \"openshift-state-metrics-79f6958cdf-lsxjz\" (UID: \"6d641c24-367d-4e6b-9402-e61e594f8811\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-lsxjz" Jul 22 02:59:33.183975 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:33.183950 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6d641c24-367d-4e6b-9402-e61e594f8811-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-lsxjz\" (UID: \"6d641c24-367d-4e6b-9402-e61e594f8811\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-lsxjz" Jul 22 02:59:33.271230 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:33.271192 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:33.285151 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:33.285119 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldw46\" (UniqueName: \"kubernetes.io/projected/ba8f4a56-ce73-4f6a-9215-13a7f32f9342-kube-api-access-ldw46\") pod \"node-exporter-ffjds\" (UID: \"ba8f4a56-ce73-4f6a-9215-13a7f32f9342\") " pod="openshift-monitoring/node-exporter-ffjds" Jul 22 02:59:33.285330 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:33.285169 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ba8f4a56-ce73-4f6a-9215-13a7f32f9342-node-exporter-accelerators-collector-config\") pod \"node-exporter-ffjds\" (UID: \"ba8f4a56-ce73-4f6a-9215-13a7f32f9342\") " pod="openshift-monitoring/node-exporter-ffjds" Jul 22 02:59:33.285330 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:33.285208 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6d641c24-367d-4e6b-9402-e61e594f8811-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-lsxjz\" (UID: \"6d641c24-367d-4e6b-9402-e61e594f8811\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-lsxjz" Jul 22 02:59:33.285330 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:33.285266 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6d641c24-367d-4e6b-9402-e61e594f8811-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-lsxjz\" (UID: \"6d641c24-367d-4e6b-9402-e61e594f8811\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-lsxjz" Jul 22 02:59:33.285330 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:33.285293 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ba8f4a56-ce73-4f6a-9215-13a7f32f9342-metrics-client-ca\") pod \"node-exporter-ffjds\" (UID: \"ba8f4a56-ce73-4f6a-9215-13a7f32f9342\") " pod="openshift-monitoring/node-exporter-ffjds" Jul 22 02:59:33.285330 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:59:33.285311 2582 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 22 02:59:33.285580 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:33.285331 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8zhl\" (UniqueName: \"kubernetes.io/projected/6d641c24-367d-4e6b-9402-e61e594f8811-kube-api-access-v8zhl\") pod \"openshift-state-metrics-79f6958cdf-lsxjz\" (UID: \"6d641c24-367d-4e6b-9402-e61e594f8811\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-lsxjz" Jul 22 02:59:33.285580 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:59:33.285373 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d641c24-367d-4e6b-9402-e61e594f8811-openshift-state-metrics-tls podName:6d641c24-367d-4e6b-9402-e61e594f8811 nodeName:}" failed. No retries permitted until 2026-07-22 02:59:33.785351742 +0000 UTC m=+84.516972567 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/6d641c24-367d-4e6b-9402-e61e594f8811-openshift-state-metrics-tls") pod "openshift-state-metrics-79f6958cdf-lsxjz" (UID: "6d641c24-367d-4e6b-9402-e61e594f8811") : secret "openshift-state-metrics-tls" not found Jul 22 02:59:33.285580 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:33.285448 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ba8f4a56-ce73-4f6a-9215-13a7f32f9342-node-exporter-tls\") pod \"node-exporter-ffjds\" (UID: \"ba8f4a56-ce73-4f6a-9215-13a7f32f9342\") " pod="openshift-monitoring/node-exporter-ffjds" Jul 22 02:59:33.285580 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:33.285491 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6d641c24-367d-4e6b-9402-e61e594f8811-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-lsxjz\" (UID: \"6d641c24-367d-4e6b-9402-e61e594f8811\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-lsxjz" Jul 22 02:59:33.285580 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:33.285515 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ba8f4a56-ce73-4f6a-9215-13a7f32f9342-node-exporter-textfile\") pod \"node-exporter-ffjds\" (UID: \"ba8f4a56-ce73-4f6a-9215-13a7f32f9342\") " pod="openshift-monitoring/node-exporter-ffjds" Jul 22 02:59:33.285580 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:33.285561 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ba8f4a56-ce73-4f6a-9215-13a7f32f9342-root\") pod \"node-exporter-ffjds\" (UID: \"ba8f4a56-ce73-4f6a-9215-13a7f32f9342\") " pod="openshift-monitoring/node-exporter-ffjds" Jul 22 02:59:33.285877 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:33.285586 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ba8f4a56-ce73-4f6a-9215-13a7f32f9342-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ffjds\" (UID: \"ba8f4a56-ce73-4f6a-9215-13a7f32f9342\") " pod="openshift-monitoring/node-exporter-ffjds" Jul 22 02:59:33.285877 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:33.285645 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ba8f4a56-ce73-4f6a-9215-13a7f32f9342-node-exporter-wtmp\") pod \"node-exporter-ffjds\" (UID: \"ba8f4a56-ce73-4f6a-9215-13a7f32f9342\") " pod="openshift-monitoring/node-exporter-ffjds" Jul 22 02:59:33.285877 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:33.285689 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ba8f4a56-ce73-4f6a-9215-13a7f32f9342-sys\") pod \"node-exporter-ffjds\" (UID: \"ba8f4a56-ce73-4f6a-9215-13a7f32f9342\") " pod="openshift-monitoring/node-exporter-ffjds" Jul 22 02:59:33.286202 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:33.286185 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6d641c24-367d-4e6b-9402-e61e594f8811-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-lsxjz\" (UID: \"6d641c24-367d-4e6b-9402-e61e594f8811\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-lsxjz" Jul 22 02:59:33.288066 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:33.288039 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6d641c24-367d-4e6b-9402-e61e594f8811-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-lsxjz\" (UID: \"6d641c24-367d-4e6b-9402-e61e594f8811\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-lsxjz" Jul 22 02:59:33.295162 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:33.295137 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8zhl\" (UniqueName: \"kubernetes.io/projected/6d641c24-367d-4e6b-9402-e61e594f8811-kube-api-access-v8zhl\") pod \"openshift-state-metrics-79f6958cdf-lsxjz\" (UID: \"6d641c24-367d-4e6b-9402-e61e594f8811\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-lsxjz" Jul 22 02:59:33.373052 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:33.373004 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:33.386679 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:33.386643 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ba8f4a56-ce73-4f6a-9215-13a7f32f9342-root\") pod \"node-exporter-ffjds\" (UID: \"ba8f4a56-ce73-4f6a-9215-13a7f32f9342\") " pod="openshift-monitoring/node-exporter-ffjds" Jul 22 02:59:33.386679 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:33.386688 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ba8f4a56-ce73-4f6a-9215-13a7f32f9342-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ffjds\" (UID: \"ba8f4a56-ce73-4f6a-9215-13a7f32f9342\") " pod="openshift-monitoring/node-exporter-ffjds" Jul 22 02:59:33.386931 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:33.386713 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ba8f4a56-ce73-4f6a-9215-13a7f32f9342-node-exporter-wtmp\") pod \"node-exporter-ffjds\" (UID: \"ba8f4a56-ce73-4f6a-9215-13a7f32f9342\") " pod="openshift-monitoring/node-exporter-ffjds" Jul 22 02:59:33.386931 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:33.386736 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ba8f4a56-ce73-4f6a-9215-13a7f32f9342-sys\") pod \"node-exporter-ffjds\" (UID: \"ba8f4a56-ce73-4f6a-9215-13a7f32f9342\") " pod="openshift-monitoring/node-exporter-ffjds" Jul 22 02:59:33.386931 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:33.386772 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ba8f4a56-ce73-4f6a-9215-13a7f32f9342-root\") pod \"node-exporter-ffjds\" (UID: \"ba8f4a56-ce73-4f6a-9215-13a7f32f9342\") " pod="openshift-monitoring/node-exporter-ffjds" Jul 22 02:59:33.386931 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:33.386798 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ba8f4a56-ce73-4f6a-9215-13a7f32f9342-sys\") pod \"node-exporter-ffjds\" (UID: \"ba8f4a56-ce73-4f6a-9215-13a7f32f9342\") " pod="openshift-monitoring/node-exporter-ffjds" Jul 22 02:59:33.386931 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:33.386863 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldw46\" (UniqueName: \"kubernetes.io/projected/ba8f4a56-ce73-4f6a-9215-13a7f32f9342-kube-api-access-ldw46\") pod \"node-exporter-ffjds\" (UID: \"ba8f4a56-ce73-4f6a-9215-13a7f32f9342\") " pod="openshift-monitoring/node-exporter-ffjds" Jul 22 02:59:33.386931 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:33.386878 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ba8f4a56-ce73-4f6a-9215-13a7f32f9342-node-exporter-wtmp\") pod \"node-exporter-ffjds\" (UID: \"ba8f4a56-ce73-4f6a-9215-13a7f32f9342\") " pod="openshift-monitoring/node-exporter-ffjds" Jul 22 02:59:33.386931 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:33.386911 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ba8f4a56-ce73-4f6a-9215-13a7f32f9342-node-exporter-accelerators-collector-config\") pod \"node-exporter-ffjds\" (UID: \"ba8f4a56-ce73-4f6a-9215-13a7f32f9342\") " pod="openshift-monitoring/node-exporter-ffjds" Jul 22 02:59:33.387294 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:33.386977 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ba8f4a56-ce73-4f6a-9215-13a7f32f9342-metrics-client-ca\") pod \"node-exporter-ffjds\" (UID: \"ba8f4a56-ce73-4f6a-9215-13a7f32f9342\") " pod="openshift-monitoring/node-exporter-ffjds" Jul 22 02:59:33.387294 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:33.387049 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ba8f4a56-ce73-4f6a-9215-13a7f32f9342-node-exporter-tls\") pod \"node-exporter-ffjds\" (UID: \"ba8f4a56-ce73-4f6a-9215-13a7f32f9342\") " pod="openshift-monitoring/node-exporter-ffjds" Jul 22 02:59:33.387294 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:33.387079 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ba8f4a56-ce73-4f6a-9215-13a7f32f9342-node-exporter-textfile\") pod \"node-exporter-ffjds\" (UID: \"ba8f4a56-ce73-4f6a-9215-13a7f32f9342\") " pod="openshift-monitoring/node-exporter-ffjds" Jul 22 02:59:33.387439 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:59:33.387314 2582 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 22 02:59:33.387439 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:59:33.387377 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba8f4a56-ce73-4f6a-9215-13a7f32f9342-node-exporter-tls podName:ba8f4a56-ce73-4f6a-9215-13a7f32f9342 nodeName:}" failed. No retries permitted until 2026-07-22 02:59:33.887357245 +0000 UTC m=+84.618978070 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/ba8f4a56-ce73-4f6a-9215-13a7f32f9342-node-exporter-tls") pod "node-exporter-ffjds" (UID: "ba8f4a56-ce73-4f6a-9215-13a7f32f9342") : secret "node-exporter-tls" not found Jul 22 02:59:33.387439 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:33.387394 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ba8f4a56-ce73-4f6a-9215-13a7f32f9342-node-exporter-textfile\") pod \"node-exporter-ffjds\" (UID: \"ba8f4a56-ce73-4f6a-9215-13a7f32f9342\") " pod="openshift-monitoring/node-exporter-ffjds" Jul 22 02:59:33.387597 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:33.387561 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ba8f4a56-ce73-4f6a-9215-13a7f32f9342-node-exporter-accelerators-collector-config\") pod \"node-exporter-ffjds\" (UID: \"ba8f4a56-ce73-4f6a-9215-13a7f32f9342\") " pod="openshift-monitoring/node-exporter-ffjds" Jul 22 02:59:33.387665 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:33.387645 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ba8f4a56-ce73-4f6a-9215-13a7f32f9342-metrics-client-ca\") pod \"node-exporter-ffjds\" (UID: \"ba8f4a56-ce73-4f6a-9215-13a7f32f9342\") " pod="openshift-monitoring/node-exporter-ffjds" Jul 22 02:59:33.389338 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:33.389312 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ba8f4a56-ce73-4f6a-9215-13a7f32f9342-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ffjds\" (UID: \"ba8f4a56-ce73-4f6a-9215-13a7f32f9342\") " pod="openshift-monitoring/node-exporter-ffjds" Jul 22 02:59:33.404443 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:33.404406 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldw46\" (UniqueName: \"kubernetes.io/projected/ba8f4a56-ce73-4f6a-9215-13a7f32f9342-kube-api-access-ldw46\") pod \"node-exporter-ffjds\" (UID: \"ba8f4a56-ce73-4f6a-9215-13a7f32f9342\") " pod="openshift-monitoring/node-exporter-ffjds" Jul 22 02:59:33.473417 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:33.473382 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:33.576801 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:33.576776 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:33.673608 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:33.673518 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:33.791613 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:33.791575 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6d641c24-367d-4e6b-9402-e61e594f8811-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-lsxjz\" (UID: \"6d641c24-367d-4e6b-9402-e61e594f8811\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-lsxjz" Jul 22 02:59:33.793954 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:33.793920 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6d641c24-367d-4e6b-9402-e61e594f8811-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-lsxjz\" (UID: \"6d641c24-367d-4e6b-9402-e61e594f8811\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-lsxjz" Jul 22 02:59:33.892409 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:33.892374 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ba8f4a56-ce73-4f6a-9215-13a7f32f9342-node-exporter-tls\") pod \"node-exporter-ffjds\" (UID: \"ba8f4a56-ce73-4f6a-9215-13a7f32f9342\") " pod="openshift-monitoring/node-exporter-ffjds" Jul 22 02:59:33.894607 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:33.894589 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ba8f4a56-ce73-4f6a-9215-13a7f32f9342-node-exporter-tls\") pod \"node-exporter-ffjds\" (UID: \"ba8f4a56-ce73-4f6a-9215-13a7f32f9342\") " pod="openshift-monitoring/node-exporter-ffjds" Jul 22 02:59:34.010696 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:34.010607 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-lsxjz" Jul 22 02:59:34.022436 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:34.022407 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ffjds" Jul 22 02:59:34.032561 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:59:34.032524 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba8f4a56_ce73_4f6a_9215_13a7f32f9342.slice/crio-a9aad73eef31db90d5aa8cab0a74edcab10462b50a9b74ab3a828d5f37a12921 WatchSource:0}: Error finding container a9aad73eef31db90d5aa8cab0a74edcab10462b50a9b74ab3a828d5f37a12921: Status 404 returned error can't find the container with id a9aad73eef31db90d5aa8cab0a74edcab10462b50a9b74ab3a828d5f37a12921 Jul 22 02:59:34.115672 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:34.115581 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:34.147066 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:34.147033 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-lsxjz"] Jul 22 02:59:34.150192 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:59:34.150154 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d641c24_367d_4e6b_9402_e61e594f8811.slice/crio-5be25324b51f85f4472f54f0094a88436f7beff4b685c33a9e9409719f72b45a WatchSource:0}: Error finding container 5be25324b51f85f4472f54f0094a88436f7beff4b685c33a9e9409719f72b45a: Status 404 returned error can't find the container with id 5be25324b51f85f4472f54f0094a88436f7beff4b685c33a9e9409719f72b45a Jul 22 02:59:34.238357 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:34.238318 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:34.326653 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:34.326618 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:34.387083 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:34.387034 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f7dc679dd-6xvrs" event={"ID":"5e6b772d-5a8a-4da1-90bd-5f7fd944f1af","Type":"ContainerStarted","Data":"39aa90ed8638a1d0c7723f2c71bf8a734c62e74dfae92ab53ee5350a1a0f3ab6"} Jul 22 02:59:34.388770 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:34.388743 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-lsxjz" event={"ID":"6d641c24-367d-4e6b-9402-e61e594f8811","Type":"ContainerStarted","Data":"9e9e6e0c48c19e65638e9c2adab89ee226e0953a6fe81816ebf94aa15760f272"} Jul 22 02:59:34.388890 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:34.388795 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-lsxjz" event={"ID":"6d641c24-367d-4e6b-9402-e61e594f8811","Type":"ContainerStarted","Data":"9e90c530772ba6b1adc72fe671c8efc72f0e6a591855ae2ec2d6043cd00fd10a"} Jul 22 02:59:34.388890 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:34.388816 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-lsxjz" event={"ID":"6d641c24-367d-4e6b-9402-e61e594f8811","Type":"ContainerStarted","Data":"5be25324b51f85f4472f54f0094a88436f7beff4b685c33a9e9409719f72b45a"} Jul 22 02:59:34.389853 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:34.389837 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ffjds" event={"ID":"ba8f4a56-ce73-4f6a-9215-13a7f32f9342","Type":"ContainerStarted","Data":"a9aad73eef31db90d5aa8cab0a74edcab10462b50a9b74ab3a828d5f37a12921"} Jul 22 02:59:34.404656 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:34.404611 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f7dc679dd-6xvrs" podStartSLOduration=3.216509684 podStartE2EDuration="16.404596154s" totalImagesPullingTime="13.18808647s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:59:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:59:34.403681082 +0000 UTC m=+85.135301929" watchObservedRunningTime="2026-07-22 02:59:34.404596154 +0000 UTC m=+85.136217001" Jul 22 02:59:34.529177 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:34.529094 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:34.628791 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:34.628762 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:35.395288 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:35.395254 2582 generic.go:356] "Generic (PLEG): container finished" podID="ba8f4a56-ce73-4f6a-9215-13a7f32f9342" containerID="1eb5f29e416fdcf13d35f8bed77c21a31bbb1b38b928210ee73ce70073c5999b" exitCode=0 Jul 22 02:59:35.395728 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:35.395336 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ffjds" event={"ID":"ba8f4a56-ce73-4f6a-9215-13a7f32f9342","Type":"ContainerDied","Data":"1eb5f29e416fdcf13d35f8bed77c21a31bbb1b38b928210ee73ce70073c5999b"} Jul 22 02:59:35.712956 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:35.712879 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:35.792611 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:35.792578 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:36.116440 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:36.116366 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:36.176227 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:36.176189 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5494b569cc-m4xl8" Jul 22 02:59:36.176421 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:36.176271 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5494b569cc-m4xl8" Jul 22 02:59:36.181605 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:36.181577 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5494b569cc-m4xl8" Jul 22 02:59:36.185160 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:36.185138 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5c648fddd6-v4m7c"] Jul 22 02:59:36.189633 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:36.189609 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5c648fddd6-v4m7c" Jul 22 02:59:36.192560 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:36.192533 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 22 02:59:36.192841 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:36.192822 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 22 02:59:36.193078 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:36.193057 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 22 02:59:36.193330 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:36.193301 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 22 02:59:36.193434 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:36.193306 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 22 02:59:36.193846 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:36.193646 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-cailkbl0uf55d\"" Jul 22 02:59:36.193846 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:36.193792 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-m5g64\"" Jul 22 02:59:36.201927 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:36.201906 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5c648fddd6-v4m7c"] Jul 22 02:59:36.217738 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:36.217707 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:36.297954 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:36.297920 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:36.314039 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:36.313984 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/00475f40-7c5a-49d4-abd5-c4893b108673-secret-grpc-tls\") pod \"thanos-querier-5c648fddd6-v4m7c\" (UID: \"00475f40-7c5a-49d4-abd5-c4893b108673\") " pod="openshift-monitoring/thanos-querier-5c648fddd6-v4m7c" Jul 22 02:59:36.314187 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:36.314069 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/00475f40-7c5a-49d4-abd5-c4893b108673-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5c648fddd6-v4m7c\" (UID: \"00475f40-7c5a-49d4-abd5-c4893b108673\") " pod="openshift-monitoring/thanos-querier-5c648fddd6-v4m7c" Jul 22 02:59:36.314187 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:36.314140 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/00475f40-7c5a-49d4-abd5-c4893b108673-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5c648fddd6-v4m7c\" (UID: \"00475f40-7c5a-49d4-abd5-c4893b108673\") " pod="openshift-monitoring/thanos-querier-5c648fddd6-v4m7c" Jul 22 02:59:36.314318 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:36.314207 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/00475f40-7c5a-49d4-abd5-c4893b108673-secret-thanos-querier-tls\") pod \"thanos-querier-5c648fddd6-v4m7c\" (UID: \"00475f40-7c5a-49d4-abd5-c4893b108673\") " pod="openshift-monitoring/thanos-querier-5c648fddd6-v4m7c" Jul 22 02:59:36.314318 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:36.314235 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc4lv\" (UniqueName: \"kubernetes.io/projected/00475f40-7c5a-49d4-abd5-c4893b108673-kube-api-access-zc4lv\") pod \"thanos-querier-5c648fddd6-v4m7c\" (UID: \"00475f40-7c5a-49d4-abd5-c4893b108673\") " pod="openshift-monitoring/thanos-querier-5c648fddd6-v4m7c" Jul 22 02:59:36.314318 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:36.314265 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/00475f40-7c5a-49d4-abd5-c4893b108673-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5c648fddd6-v4m7c\" (UID: \"00475f40-7c5a-49d4-abd5-c4893b108673\") " pod="openshift-monitoring/thanos-querier-5c648fddd6-v4m7c" Jul 22 02:59:36.314440 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:36.314361 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/00475f40-7c5a-49d4-abd5-c4893b108673-metrics-client-ca\") pod \"thanos-querier-5c648fddd6-v4m7c\" (UID: \"00475f40-7c5a-49d4-abd5-c4893b108673\") " pod="openshift-monitoring/thanos-querier-5c648fddd6-v4m7c" Jul 22 02:59:36.314793 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:36.314755 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/00475f40-7c5a-49d4-abd5-c4893b108673-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5c648fddd6-v4m7c\" (UID: \"00475f40-7c5a-49d4-abd5-c4893b108673\") " pod="openshift-monitoring/thanos-querier-5c648fddd6-v4m7c" Jul 22 02:59:36.401315 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:36.401276 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ffjds" event={"ID":"ba8f4a56-ce73-4f6a-9215-13a7f32f9342","Type":"ContainerStarted","Data":"e4e8936ae736904ad76d2106bb3671f317d8a0ba56016d91ef20e0b4277d3c16"} Jul 22 02:59:36.401895 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:36.401381 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ffjds" event={"ID":"ba8f4a56-ce73-4f6a-9215-13a7f32f9342","Type":"ContainerStarted","Data":"e0e5995607dfe6a0958db83dec0db9e6b700403e94bad0e57ef94477d7c4464f"} Jul 22 02:59:36.403427 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:36.403382 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-lsxjz" event={"ID":"6d641c24-367d-4e6b-9402-e61e594f8811","Type":"ContainerStarted","Data":"69bbe5935ae9adf81f277018226514fd79339105df4ddf81db74711e8d112d0b"} Jul 22 02:59:36.407589 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:36.407560 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5494b569cc-m4xl8" Jul 22 02:59:36.415955 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:36.415931 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/00475f40-7c5a-49d4-abd5-c4893b108673-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5c648fddd6-v4m7c\" (UID: \"00475f40-7c5a-49d4-abd5-c4893b108673\") " pod="openshift-monitoring/thanos-querier-5c648fddd6-v4m7c" Jul 22 02:59:36.416182 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:36.415968 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/00475f40-7c5a-49d4-abd5-c4893b108673-secret-thanos-querier-tls\") pod \"thanos-querier-5c648fddd6-v4m7c\" (UID: \"00475f40-7c5a-49d4-abd5-c4893b108673\") " pod="openshift-monitoring/thanos-querier-5c648fddd6-v4m7c" Jul 22 02:59:36.416182 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:36.415989 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc4lv\" (UniqueName: \"kubernetes.io/projected/00475f40-7c5a-49d4-abd5-c4893b108673-kube-api-access-zc4lv\") pod \"thanos-querier-5c648fddd6-v4m7c\" (UID: \"00475f40-7c5a-49d4-abd5-c4893b108673\") " pod="openshift-monitoring/thanos-querier-5c648fddd6-v4m7c" Jul 22 02:59:36.416182 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:36.416007 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/00475f40-7c5a-49d4-abd5-c4893b108673-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5c648fddd6-v4m7c\" (UID: \"00475f40-7c5a-49d4-abd5-c4893b108673\") " pod="openshift-monitoring/thanos-querier-5c648fddd6-v4m7c" Jul 22 02:59:36.416182 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:36.416055 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/00475f40-7c5a-49d4-abd5-c4893b108673-metrics-client-ca\") pod \"thanos-querier-5c648fddd6-v4m7c\" (UID: \"00475f40-7c5a-49d4-abd5-c4893b108673\") " pod="openshift-monitoring/thanos-querier-5c648fddd6-v4m7c" Jul 22 02:59:36.416182 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:36.416087 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/00475f40-7c5a-49d4-abd5-c4893b108673-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5c648fddd6-v4m7c\" (UID: \"00475f40-7c5a-49d4-abd5-c4893b108673\") " pod="openshift-monitoring/thanos-querier-5c648fddd6-v4m7c" Jul 22 02:59:36.416182 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:36.416113 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/00475f40-7c5a-49d4-abd5-c4893b108673-secret-grpc-tls\") pod \"thanos-querier-5c648fddd6-v4m7c\" (UID: \"00475f40-7c5a-49d4-abd5-c4893b108673\") " pod="openshift-monitoring/thanos-querier-5c648fddd6-v4m7c" Jul 22 02:59:36.416182 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:36.416148 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/00475f40-7c5a-49d4-abd5-c4893b108673-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5c648fddd6-v4m7c\" (UID: \"00475f40-7c5a-49d4-abd5-c4893b108673\") " pod="openshift-monitoring/thanos-querier-5c648fddd6-v4m7c" Jul 22 02:59:36.417215 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:36.417187 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/00475f40-7c5a-49d4-abd5-c4893b108673-metrics-client-ca\") pod \"thanos-querier-5c648fddd6-v4m7c\" (UID: \"00475f40-7c5a-49d4-abd5-c4893b108673\") " pod="openshift-monitoring/thanos-querier-5c648fddd6-v4m7c" Jul 22 02:59:36.418706 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:36.418638 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/00475f40-7c5a-49d4-abd5-c4893b108673-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5c648fddd6-v4m7c\" (UID: \"00475f40-7c5a-49d4-abd5-c4893b108673\") " pod="openshift-monitoring/thanos-querier-5c648fddd6-v4m7c" Jul 22 02:59:36.418806 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:36.418726 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/00475f40-7c5a-49d4-abd5-c4893b108673-secret-grpc-tls\") pod \"thanos-querier-5c648fddd6-v4m7c\" (UID: \"00475f40-7c5a-49d4-abd5-c4893b108673\") " pod="openshift-monitoring/thanos-querier-5c648fddd6-v4m7c" Jul 22 02:59:36.418864 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:36.418841 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/00475f40-7c5a-49d4-abd5-c4893b108673-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5c648fddd6-v4m7c\" (UID: \"00475f40-7c5a-49d4-abd5-c4893b108673\") " pod="openshift-monitoring/thanos-querier-5c648fddd6-v4m7c" Jul 22 02:59:36.418952 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:36.418934 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/00475f40-7c5a-49d4-abd5-c4893b108673-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5c648fddd6-v4m7c\" (UID: \"00475f40-7c5a-49d4-abd5-c4893b108673\") " pod="openshift-monitoring/thanos-querier-5c648fddd6-v4m7c" Jul 22 02:59:36.419031 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:36.418948 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/00475f40-7c5a-49d4-abd5-c4893b108673-secret-thanos-querier-tls\") pod \"thanos-querier-5c648fddd6-v4m7c\" (UID: \"00475f40-7c5a-49d4-abd5-c4893b108673\") " pod="openshift-monitoring/thanos-querier-5c648fddd6-v4m7c" Jul 22 02:59:36.419031 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:36.418960 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/00475f40-7c5a-49d4-abd5-c4893b108673-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5c648fddd6-v4m7c\" (UID: \"00475f40-7c5a-49d4-abd5-c4893b108673\") " pod="openshift-monitoring/thanos-querier-5c648fddd6-v4m7c" Jul 22 02:59:36.425721 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:36.425508 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-ffjds" podStartSLOduration=2.506696148 podStartE2EDuration="3.425490143s" totalImagesPullingTime="849.476174ms" totalInitContainerRuntime="69.317821ms" isStatefulPod=true podCreationTimestamp="2026-07-22 02:59:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:59:36.423989647 +0000 UTC m=+87.155610540" watchObservedRunningTime="2026-07-22 02:59:36.425490143 +0000 UTC m=+87.157110994" Jul 22 02:59:36.428760 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:36.428740 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc4lv\" (UniqueName: \"kubernetes.io/projected/00475f40-7c5a-49d4-abd5-c4893b108673-kube-api-access-zc4lv\") pod \"thanos-querier-5c648fddd6-v4m7c\" (UID: \"00475f40-7c5a-49d4-abd5-c4893b108673\") " pod="openshift-monitoring/thanos-querier-5c648fddd6-v4m7c" Jul 22 02:59:36.442323 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:36.442271 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-lsxjz" podStartSLOduration=2.237013637 podStartE2EDuration="3.442254307s" totalImagesPullingTime="1.20524067s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:59:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:59:36.441587548 +0000 UTC m=+87.173208396" watchObservedRunningTime="2026-07-22 02:59:36.442254307 +0000 UTC m=+87.173875155" Jul 22 02:59:36.499560 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:36.499529 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5c648fddd6-v4m7c" Jul 22 02:59:36.617634 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:36.617604 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:36.641752 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:36.641719 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5c648fddd6-v4m7c"] Jul 22 02:59:36.644787 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:59:36.644757 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00475f40_7c5a_49d4_abd5_c4893b108673.slice/crio-73951fdeb2c0bf3500a9e69fa864ebc03e3aaf60cfaaf148dd29d4f5e64fbf6c WatchSource:0}: Error finding container 73951fdeb2c0bf3500a9e69fa864ebc03e3aaf60cfaaf148dd29d4f5e64fbf6c: Status 404 returned error can't find the container with id 73951fdeb2c0bf3500a9e69fa864ebc03e3aaf60cfaaf148dd29d4f5e64fbf6c Jul 22 02:59:36.719704 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:36.719603 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:36.820917 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:36.820879 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:36.921309 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:36.921275 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:37.408987 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:37.408938 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5c648fddd6-v4m7c" event={"ID":"00475f40-7c5a-49d4-abd5-c4893b108673","Type":"ContainerStarted","Data":"73951fdeb2c0bf3500a9e69fa864ebc03e3aaf60cfaaf148dd29d4f5e64fbf6c"} Jul 22 02:59:37.923312 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:37.923266 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:37.943952 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:37.943917 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-n42kf"] Jul 22 02:59:37.947063 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:37.947038 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-n42kf" Jul 22 02:59:37.951093 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:37.951069 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 22 02:59:37.951230 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:37.951092 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-6rvpc\"" Jul 22 02:59:37.956731 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:37.956481 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-n42kf"] Jul 22 02:59:38.020771 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:38.020735 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:38.134559 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:38.134527 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1737a1fb-ebbd-4aab-8a32-595697912b60-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-n42kf\" (UID: \"1737a1fb-ebbd-4aab-8a32-595697912b60\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-n42kf" Jul 22 02:59:38.134741 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:38.134602 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltcf4\" (UniqueName: \"kubernetes.io/projected/1737a1fb-ebbd-4aab-8a32-595697912b60-kube-api-access-ltcf4\") pod \"monitoring-plugin-7b7b7bc8bf-n42kf\" (UID: \"1737a1fb-ebbd-4aab-8a32-595697912b60\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-n42kf" Jul 22 02:59:38.235478 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:38.235381 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1737a1fb-ebbd-4aab-8a32-595697912b60-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-n42kf\" (UID: \"1737a1fb-ebbd-4aab-8a32-595697912b60\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-n42kf" Jul 22 02:59:38.235648 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:38.235493 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltcf4\" (UniqueName: \"kubernetes.io/projected/1737a1fb-ebbd-4aab-8a32-595697912b60-kube-api-access-ltcf4\") pod \"monitoring-plugin-7b7b7bc8bf-n42kf\" (UID: \"1737a1fb-ebbd-4aab-8a32-595697912b60\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-n42kf" Jul 22 02:59:38.235648 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:59:38.235566 2582 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 22 02:59:38.235648 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:59:38.235643 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1737a1fb-ebbd-4aab-8a32-595697912b60-monitoring-plugin-cert podName:1737a1fb-ebbd-4aab-8a32-595697912b60 nodeName:}" failed. No retries permitted until 2026-07-22 02:59:38.735621057 +0000 UTC m=+89.467241882 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/1737a1fb-ebbd-4aab-8a32-595697912b60-monitoring-plugin-cert") pod "monitoring-plugin-7b7b7bc8bf-n42kf" (UID: "1737a1fb-ebbd-4aab-8a32-595697912b60") : secret "monitoring-plugin-cert" not found Jul 22 02:59:38.244774 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:38.244745 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltcf4\" (UniqueName: \"kubernetes.io/projected/1737a1fb-ebbd-4aab-8a32-595697912b60-kube-api-access-ltcf4\") pod \"monitoring-plugin-7b7b7bc8bf-n42kf\" (UID: \"1737a1fb-ebbd-4aab-8a32-595697912b60\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-n42kf" Jul 22 02:59:38.433938 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:38.433905 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:38.739231 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:38.739197 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1737a1fb-ebbd-4aab-8a32-595697912b60-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-n42kf\" (UID: \"1737a1fb-ebbd-4aab-8a32-595697912b60\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-n42kf" Jul 22 02:59:38.741918 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:38.741891 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1737a1fb-ebbd-4aab-8a32-595697912b60-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-n42kf\" (UID: \"1737a1fb-ebbd-4aab-8a32-595697912b60\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-n42kf" Jul 22 02:59:38.860854 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:38.860822 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-n42kf" Jul 22 02:59:38.989927 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:38.989893 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-n42kf"] Jul 22 02:59:38.992966 ip-10-0-131-67 kubenswrapper[2582]: W0722 02:59:38.992942 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1737a1fb_ebbd_4aab_8a32_595697912b60.slice/crio-cb1061fe8b8a8e50e611ffa534c87f9901741f57ba6b39cd82d33d5951b54bac WatchSource:0}: Error finding container cb1061fe8b8a8e50e611ffa534c87f9901741f57ba6b39cd82d33d5951b54bac: Status 404 returned error can't find the container with id cb1061fe8b8a8e50e611ffa534c87f9901741f57ba6b39cd82d33d5951b54bac Jul 22 02:59:39.365665 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:39.365588 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:39.420076 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:39.420033 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-n42kf" event={"ID":"1737a1fb-ebbd-4aab-8a32-595697912b60","Type":"ContainerStarted","Data":"cb1061fe8b8a8e50e611ffa534c87f9901741f57ba6b39cd82d33d5951b54bac"} Jul 22 02:59:39.422767 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:39.422732 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5c648fddd6-v4m7c" event={"ID":"00475f40-7c5a-49d4-abd5-c4893b108673","Type":"ContainerStarted","Data":"1913a9ff9daac7329646f04b3839bead1b439dcc48333a1eeffda394b99666b3"} Jul 22 02:59:39.422904 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:39.422795 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5c648fddd6-v4m7c" event={"ID":"00475f40-7c5a-49d4-abd5-c4893b108673","Type":"ContainerStarted","Data":"8625e42f871b33cf1ff64840a9e7502c2d518ebabe4aff689bdb4e1adaaf23aa"} Jul 22 02:59:39.422904 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:39.422823 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5c648fddd6-v4m7c" event={"ID":"00475f40-7c5a-49d4-abd5-c4893b108673","Type":"ContainerStarted","Data":"d112bf47ff297fc7dfb44d7684c79b45f259d520ee37f66a5166587a13ef6beb"} Jul 22 02:59:39.451552 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:39.451495 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:40.431949 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:40.431899 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5c648fddd6-v4m7c" event={"ID":"00475f40-7c5a-49d4-abd5-c4893b108673","Type":"ContainerStarted","Data":"82243252775a858361e81fa07a07791823962c9e0329b2d4d3e1320c748f795e"} Jul 22 02:59:40.432148 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:40.431986 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5c648fddd6-v4m7c" event={"ID":"00475f40-7c5a-49d4-abd5-c4893b108673","Type":"ContainerStarted","Data":"40fde316f68d63212c21922d7dc2bb52438d6e8ab520ca0ff03c843efaf38568"} Jul 22 02:59:40.432148 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:40.432040 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5c648fddd6-v4m7c" event={"ID":"00475f40-7c5a-49d4-abd5-c4893b108673","Type":"ContainerStarted","Data":"878946bb3fffaeefc4be827dadce4cce9feae984a5a093c83c7ae1bf63c85973"} Jul 22 02:59:40.432148 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:40.432086 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5c648fddd6-v4m7c" Jul 22 02:59:40.460279 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:40.459956 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5c648fddd6-v4m7c" podStartSLOduration=1.466654103 podStartE2EDuration="4.459931437s" totalImagesPullingTime="2.993277334s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:59:36 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:59:40.459109195 +0000 UTC m=+91.190730042" watchObservedRunningTime="2026-07-22 02:59:40.459931437 +0000 UTC m=+91.191552284" Jul 22 02:59:41.014920 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:41.014847 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:41.106560 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:41.106530 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:41.438200 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:41.438170 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-n42kf" event={"ID":"1737a1fb-ebbd-4aab-8a32-595697912b60","Type":"ContainerStarted","Data":"90c7cd3d62b1a97762608cf471ab96b64fe0116497d7daac323f3f1f0688140c"} Jul 22 02:59:41.438459 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:41.438444 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-n42kf" Jul 22 02:59:41.443505 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:41.443483 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-n42kf" Jul 22 02:59:41.458353 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:41.458300 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-n42kf" podStartSLOduration=2.852827005 podStartE2EDuration="4.458284233s" totalImagesPullingTime="1.605457228s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:59:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:59:41.4574418 +0000 UTC m=+92.189062689" watchObservedRunningTime="2026-07-22 02:59:41.458284233 +0000 UTC m=+92.189905080" Jul 22 02:59:42.342644 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:42.342614 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-67748fc486-qc6lw" Jul 22 02:59:42.490606 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:42.490569 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:42.578855 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:42.578820 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:43.503829 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:43.503779 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7dd8f5b7fd-2c4kr" podUID="fe5ad29f-dbb7-4353-80ea-60bf10139703" containerName="registry" containerID="cri-o://867ed54c34375c1718567a2a46c44db03422263a33ff0ddcd81e259e84b9b4af" gracePeriod=30 Jul 22 02:59:44.317122 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:44.317086 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7dd8f5b7fd-2c4kr" Jul 22 02:59:44.387630 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:44.387593 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fe5ad29f-dbb7-4353-80ea-60bf10139703-installation-pull-secrets\") pod \"fe5ad29f-dbb7-4353-80ea-60bf10139703\" (UID: \"fe5ad29f-dbb7-4353-80ea-60bf10139703\") " Jul 22 02:59:44.387630 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:44.387629 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fe5ad29f-dbb7-4353-80ea-60bf10139703-image-registry-private-configuration\") pod \"fe5ad29f-dbb7-4353-80ea-60bf10139703\" (UID: \"fe5ad29f-dbb7-4353-80ea-60bf10139703\") " Jul 22 02:59:44.387853 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:44.387660 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fe5ad29f-dbb7-4353-80ea-60bf10139703-ca-trust-extracted\") pod \"fe5ad29f-dbb7-4353-80ea-60bf10139703\" (UID: \"fe5ad29f-dbb7-4353-80ea-60bf10139703\") " Jul 22 02:59:44.387853 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:44.387702 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fe5ad29f-dbb7-4353-80ea-60bf10139703-registry-tls\") pod \"fe5ad29f-dbb7-4353-80ea-60bf10139703\" (UID: \"fe5ad29f-dbb7-4353-80ea-60bf10139703\") " Jul 22 02:59:44.387853 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:44.387732 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fe5ad29f-dbb7-4353-80ea-60bf10139703-registry-certificates\") pod \"fe5ad29f-dbb7-4353-80ea-60bf10139703\" (UID: \"fe5ad29f-dbb7-4353-80ea-60bf10139703\") " Jul 22 02:59:44.387853 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:44.387760 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w2ttk\" (UniqueName: \"kubernetes.io/projected/fe5ad29f-dbb7-4353-80ea-60bf10139703-kube-api-access-w2ttk\") pod \"fe5ad29f-dbb7-4353-80ea-60bf10139703\" (UID: \"fe5ad29f-dbb7-4353-80ea-60bf10139703\") " Jul 22 02:59:44.387853 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:44.387810 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fe5ad29f-dbb7-4353-80ea-60bf10139703-bound-sa-token\") pod \"fe5ad29f-dbb7-4353-80ea-60bf10139703\" (UID: \"fe5ad29f-dbb7-4353-80ea-60bf10139703\") " Jul 22 02:59:44.390155 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:44.388577 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe5ad29f-dbb7-4353-80ea-60bf10139703-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "fe5ad29f-dbb7-4353-80ea-60bf10139703" (UID: "fe5ad29f-dbb7-4353-80ea-60bf10139703"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 02:59:44.390155 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:44.389004 2582 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fe5ad29f-dbb7-4353-80ea-60bf10139703-registry-certificates\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 02:59:44.393802 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:44.393568 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe5ad29f-dbb7-4353-80ea-60bf10139703-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "fe5ad29f-dbb7-4353-80ea-60bf10139703" (UID: "fe5ad29f-dbb7-4353-80ea-60bf10139703"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 02:59:44.394587 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:44.394089 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe5ad29f-dbb7-4353-80ea-60bf10139703-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "fe5ad29f-dbb7-4353-80ea-60bf10139703" (UID: "fe5ad29f-dbb7-4353-80ea-60bf10139703"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:59:44.394686 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:44.394589 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe5ad29f-dbb7-4353-80ea-60bf10139703-kube-api-access-w2ttk" (OuterVolumeSpecName: "kube-api-access-w2ttk") pod "fe5ad29f-dbb7-4353-80ea-60bf10139703" (UID: "fe5ad29f-dbb7-4353-80ea-60bf10139703"). InnerVolumeSpecName "kube-api-access-w2ttk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:59:44.394686 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:44.394601 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe5ad29f-dbb7-4353-80ea-60bf10139703-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "fe5ad29f-dbb7-4353-80ea-60bf10139703" (UID: "fe5ad29f-dbb7-4353-80ea-60bf10139703"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 02:59:44.395276 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:44.395249 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe5ad29f-dbb7-4353-80ea-60bf10139703-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "fe5ad29f-dbb7-4353-80ea-60bf10139703" (UID: "fe5ad29f-dbb7-4353-80ea-60bf10139703"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:59:44.401033 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:44.400990 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe5ad29f-dbb7-4353-80ea-60bf10139703-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "fe5ad29f-dbb7-4353-80ea-60bf10139703" (UID: "fe5ad29f-dbb7-4353-80ea-60bf10139703"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:59:44.450768 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:44.450736 2582 generic.go:356] "Generic (PLEG): container finished" podID="fe5ad29f-dbb7-4353-80ea-60bf10139703" containerID="867ed54c34375c1718567a2a46c44db03422263a33ff0ddcd81e259e84b9b4af" exitCode=0 Jul 22 02:59:44.450990 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:44.450794 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7dd8f5b7fd-2c4kr" Jul 22 02:59:44.450990 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:44.450829 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7dd8f5b7fd-2c4kr" event={"ID":"fe5ad29f-dbb7-4353-80ea-60bf10139703","Type":"ContainerDied","Data":"867ed54c34375c1718567a2a46c44db03422263a33ff0ddcd81e259e84b9b4af"} Jul 22 02:59:44.450990 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:44.450906 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7dd8f5b7fd-2c4kr" event={"ID":"fe5ad29f-dbb7-4353-80ea-60bf10139703","Type":"ContainerDied","Data":"efdcc6e09fda7b23aada5f9aeacba2e9718f3e17832059d893625c99746ef72d"} Jul 22 02:59:44.450990 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:44.450928 2582 scope.go:117] "RemoveContainer" containerID="867ed54c34375c1718567a2a46c44db03422263a33ff0ddcd81e259e84b9b4af" Jul 22 02:59:44.459170 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:44.459155 2582 scope.go:117] "RemoveContainer" containerID="867ed54c34375c1718567a2a46c44db03422263a33ff0ddcd81e259e84b9b4af" Jul 22 02:59:44.459437 ip-10-0-131-67 kubenswrapper[2582]: E0722 02:59:44.459418 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"867ed54c34375c1718567a2a46c44db03422263a33ff0ddcd81e259e84b9b4af\": container with ID starting with 867ed54c34375c1718567a2a46c44db03422263a33ff0ddcd81e259e84b9b4af not found: ID does not exist" containerID="867ed54c34375c1718567a2a46c44db03422263a33ff0ddcd81e259e84b9b4af" Jul 22 02:59:44.459491 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:44.459454 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"867ed54c34375c1718567a2a46c44db03422263a33ff0ddcd81e259e84b9b4af"} err="failed to get container status \"867ed54c34375c1718567a2a46c44db03422263a33ff0ddcd81e259e84b9b4af\": rpc error: code = NotFound desc = could not find container \"867ed54c34375c1718567a2a46c44db03422263a33ff0ddcd81e259e84b9b4af\": container with ID starting with 867ed54c34375c1718567a2a46c44db03422263a33ff0ddcd81e259e84b9b4af not found: ID does not exist" Jul 22 02:59:44.489408 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:44.489374 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fe5ad29f-dbb7-4353-80ea-60bf10139703-trusted-ca\") pod \"fe5ad29f-dbb7-4353-80ea-60bf10139703\" (UID: \"fe5ad29f-dbb7-4353-80ea-60bf10139703\") " Jul 22 02:59:44.489616 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:44.489596 2582 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fe5ad29f-dbb7-4353-80ea-60bf10139703-registry-tls\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 02:59:44.489663 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:44.489622 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2ttk\" (UniqueName: \"kubernetes.io/projected/fe5ad29f-dbb7-4353-80ea-60bf10139703-kube-api-access-w2ttk\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 02:59:44.489663 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:44.489637 2582 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fe5ad29f-dbb7-4353-80ea-60bf10139703-bound-sa-token\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 02:59:44.489663 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:44.489651 2582 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fe5ad29f-dbb7-4353-80ea-60bf10139703-installation-pull-secrets\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 02:59:44.489750 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:44.489665 2582 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fe5ad29f-dbb7-4353-80ea-60bf10139703-image-registry-private-configuration\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 02:59:44.489750 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:44.489680 2582 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fe5ad29f-dbb7-4353-80ea-60bf10139703-ca-trust-extracted\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 02:59:44.489820 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:44.489803 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe5ad29f-dbb7-4353-80ea-60bf10139703-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "fe5ad29f-dbb7-4353-80ea-60bf10139703" (UID: "fe5ad29f-dbb7-4353-80ea-60bf10139703"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 02:59:44.590335 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:44.590298 2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fe5ad29f-dbb7-4353-80ea-60bf10139703-trusted-ca\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 02:59:44.775031 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:44.774989 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7dd8f5b7fd-2c4kr"] Jul 22 02:59:44.779600 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:44.779575 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7dd8f5b7fd-2c4kr"] Jul 22 02:59:45.900028 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:45.899989 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe5ad29f-dbb7-4353-80ea-60bf10139703" path="/var/lib/kubelet/pods/fe5ad29f-dbb7-4353-80ea-60bf10139703/volumes" Jul 22 02:59:46.445423 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:46.445397 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5c648fddd6-v4m7c" Jul 22 02:59:47.303172 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:47.303139 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-5xv9c" Jul 22 02:59:48.187802 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:48.187771 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:48.381862 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:48.381821 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:49.894315 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:49.894285 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5494b569cc-m4xl8"] Jul 22 02:59:50.034447 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:50.034414 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:50.150233 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:50.150152 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:56.024493 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:56.024456 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 02:59:56.154944 ip-10-0-131-67 kubenswrapper[2582]: I0722 02:59:56.154909 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:00:03.756930 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:03.756895 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:00:03.837529 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:03.837500 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:00:05.259486 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:05.259446 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:00:05.345365 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:05.345337 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:00:11.051754 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:11.051721 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:00:11.167836 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:11.167804 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:00:14.921071 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:14.920989 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5494b569cc-m4xl8" podUID="bab72638-4717-4ea0-8a68-b4d9a1138154" containerName="console" containerID="cri-o://757893d35ddfaa48b8cf9aca3d9862e345a52426c6ed47a7741c3df9ff52bead" gracePeriod=15 Jul 22 03:00:15.167496 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:15.167466 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5494b569cc-m4xl8" Jul 22 03:00:15.221363 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:15.221283 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bab72638-4717-4ea0-8a68-b4d9a1138154-service-ca\") pod \"bab72638-4717-4ea0-8a68-b4d9a1138154\" (UID: \"bab72638-4717-4ea0-8a68-b4d9a1138154\") " Jul 22 03:00:15.221363 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:15.221320 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bab72638-4717-4ea0-8a68-b4d9a1138154-console-oauth-config\") pod \"bab72638-4717-4ea0-8a68-b4d9a1138154\" (UID: \"bab72638-4717-4ea0-8a68-b4d9a1138154\") " Jul 22 03:00:15.221576 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:15.221387 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bab72638-4717-4ea0-8a68-b4d9a1138154-console-serving-cert\") pod \"bab72638-4717-4ea0-8a68-b4d9a1138154\" (UID: \"bab72638-4717-4ea0-8a68-b4d9a1138154\") " Jul 22 03:00:15.221576 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:15.221406 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bab72638-4717-4ea0-8a68-b4d9a1138154-console-config\") pod \"bab72638-4717-4ea0-8a68-b4d9a1138154\" (UID: \"bab72638-4717-4ea0-8a68-b4d9a1138154\") " Jul 22 03:00:15.221576 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:15.221434 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bab72638-4717-4ea0-8a68-b4d9a1138154-trusted-ca-bundle\") pod \"bab72638-4717-4ea0-8a68-b4d9a1138154\" (UID: \"bab72638-4717-4ea0-8a68-b4d9a1138154\") " Jul 22 03:00:15.221576 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:15.221516 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tts94\" (UniqueName: \"kubernetes.io/projected/bab72638-4717-4ea0-8a68-b4d9a1138154-kube-api-access-tts94\") pod \"bab72638-4717-4ea0-8a68-b4d9a1138154\" (UID: \"bab72638-4717-4ea0-8a68-b4d9a1138154\") " Jul 22 03:00:15.221576 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:15.221548 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bab72638-4717-4ea0-8a68-b4d9a1138154-oauth-serving-cert\") pod \"bab72638-4717-4ea0-8a68-b4d9a1138154\" (UID: \"bab72638-4717-4ea0-8a68-b4d9a1138154\") " Jul 22 03:00:15.221823 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:15.221735 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bab72638-4717-4ea0-8a68-b4d9a1138154-service-ca" (OuterVolumeSpecName: "service-ca") pod "bab72638-4717-4ea0-8a68-b4d9a1138154" (UID: "bab72638-4717-4ea0-8a68-b4d9a1138154"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 03:00:15.221880 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:15.221828 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bab72638-4717-4ea0-8a68-b4d9a1138154-console-config" (OuterVolumeSpecName: "console-config") pod "bab72638-4717-4ea0-8a68-b4d9a1138154" (UID: "bab72638-4717-4ea0-8a68-b4d9a1138154"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 03:00:15.221880 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:15.221834 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bab72638-4717-4ea0-8a68-b4d9a1138154-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "bab72638-4717-4ea0-8a68-b4d9a1138154" (UID: "bab72638-4717-4ea0-8a68-b4d9a1138154"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 03:00:15.222090 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:15.222067 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bab72638-4717-4ea0-8a68-b4d9a1138154-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "bab72638-4717-4ea0-8a68-b4d9a1138154" (UID: "bab72638-4717-4ea0-8a68-b4d9a1138154"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 03:00:15.223722 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:15.223694 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bab72638-4717-4ea0-8a68-b4d9a1138154-kube-api-access-tts94" (OuterVolumeSpecName: "kube-api-access-tts94") pod "bab72638-4717-4ea0-8a68-b4d9a1138154" (UID: "bab72638-4717-4ea0-8a68-b4d9a1138154"). InnerVolumeSpecName "kube-api-access-tts94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 03:00:15.223820 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:15.223722 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bab72638-4717-4ea0-8a68-b4d9a1138154-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "bab72638-4717-4ea0-8a68-b4d9a1138154" (UID: "bab72638-4717-4ea0-8a68-b4d9a1138154"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 03:00:15.223820 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:15.223740 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bab72638-4717-4ea0-8a68-b4d9a1138154-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "bab72638-4717-4ea0-8a68-b4d9a1138154" (UID: "bab72638-4717-4ea0-8a68-b4d9a1138154"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 03:00:15.322796 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:15.322758 2582 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bab72638-4717-4ea0-8a68-b4d9a1138154-service-ca\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 03:00:15.322796 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:15.322788 2582 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bab72638-4717-4ea0-8a68-b4d9a1138154-console-oauth-config\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 03:00:15.322796 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:15.322800 2582 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bab72638-4717-4ea0-8a68-b4d9a1138154-console-serving-cert\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 03:00:15.322796 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:15.322809 2582 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bab72638-4717-4ea0-8a68-b4d9a1138154-console-config\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 03:00:15.323090 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:15.322818 2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bab72638-4717-4ea0-8a68-b4d9a1138154-trusted-ca-bundle\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 03:00:15.323090 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:15.322827 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tts94\" (UniqueName: \"kubernetes.io/projected/bab72638-4717-4ea0-8a68-b4d9a1138154-kube-api-access-tts94\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 03:00:15.323090 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:15.322836 2582 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bab72638-4717-4ea0-8a68-b4d9a1138154-oauth-serving-cert\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 03:00:15.572306 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:15.572219 2582 generic.go:356] "Generic (PLEG): container finished" podID="bab72638-4717-4ea0-8a68-b4d9a1138154" containerID="757893d35ddfaa48b8cf9aca3d9862e345a52426c6ed47a7741c3df9ff52bead" exitCode=0 Jul 22 03:00:15.572306 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:15.572273 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5494b569cc-m4xl8" Jul 22 03:00:15.572306 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:15.572298 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5494b569cc-m4xl8" event={"ID":"bab72638-4717-4ea0-8a68-b4d9a1138154","Type":"ContainerDied","Data":"757893d35ddfaa48b8cf9aca3d9862e345a52426c6ed47a7741c3df9ff52bead"} Jul 22 03:00:15.572505 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:15.572368 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5494b569cc-m4xl8" event={"ID":"bab72638-4717-4ea0-8a68-b4d9a1138154","Type":"ContainerDied","Data":"32d1deded5b25dfe93b798ffe2bdefe44f4d35cb9c0573086ba7210464ff4cb8"} Jul 22 03:00:15.572505 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:15.572389 2582 scope.go:117] "RemoveContainer" containerID="757893d35ddfaa48b8cf9aca3d9862e345a52426c6ed47a7741c3df9ff52bead" Jul 22 03:00:15.580900 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:15.580880 2582 scope.go:117] "RemoveContainer" containerID="757893d35ddfaa48b8cf9aca3d9862e345a52426c6ed47a7741c3df9ff52bead" Jul 22 03:00:15.581184 ip-10-0-131-67 kubenswrapper[2582]: E0722 03:00:15.581165 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"757893d35ddfaa48b8cf9aca3d9862e345a52426c6ed47a7741c3df9ff52bead\": container with ID starting with 757893d35ddfaa48b8cf9aca3d9862e345a52426c6ed47a7741c3df9ff52bead not found: ID does not exist" containerID="757893d35ddfaa48b8cf9aca3d9862e345a52426c6ed47a7741c3df9ff52bead" Jul 22 03:00:15.581256 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:15.581196 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"757893d35ddfaa48b8cf9aca3d9862e345a52426c6ed47a7741c3df9ff52bead"} err="failed to get container status \"757893d35ddfaa48b8cf9aca3d9862e345a52426c6ed47a7741c3df9ff52bead\": rpc error: code = NotFound desc = could not find container \"757893d35ddfaa48b8cf9aca3d9862e345a52426c6ed47a7741c3df9ff52bead\": container with ID starting with 757893d35ddfaa48b8cf9aca3d9862e345a52426c6ed47a7741c3df9ff52bead not found: ID does not exist" Jul 22 03:00:15.602000 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:15.601961 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5494b569cc-m4xl8"] Jul 22 03:00:15.604155 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:15.604130 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5494b569cc-m4xl8"] Jul 22 03:00:15.899951 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:15.899909 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bab72638-4717-4ea0-8a68-b4d9a1138154" path="/var/lib/kubelet/pods/bab72638-4717-4ea0-8a68-b4d9a1138154/volumes" Jul 22 03:00:24.608099 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:24.608063 2582 generic.go:356] "Generic (PLEG): container finished" podID="34cca172-635b-4437-a6e0-fa79ada1fd74" containerID="669e13f5da7b3bfc013ac3dbd4cd71a2e2e17bf7feb165b152f4447bd8cf2b54" exitCode=0 Jul 22 03:00:24.608580 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:24.608141 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-2nk8j" event={"ID":"34cca172-635b-4437-a6e0-fa79ada1fd74","Type":"ContainerDied","Data":"669e13f5da7b3bfc013ac3dbd4cd71a2e2e17bf7feb165b152f4447bd8cf2b54"} Jul 22 03:00:24.608580 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:24.608560 2582 scope.go:117] "RemoveContainer" containerID="669e13f5da7b3bfc013ac3dbd4cd71a2e2e17bf7feb165b152f4447bd8cf2b54" Jul 22 03:00:25.617663 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:25.617629 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-2nk8j" event={"ID":"34cca172-635b-4437-a6e0-fa79ada1fd74","Type":"ContainerStarted","Data":"98a4aa83536f7a9a2f0bed438c998ca6923ead5c2a6b33d10918a999ee0509b9"} Jul 22 03:00:52.599840 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:52.599796 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:00:52.873718 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:52.873683 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:00:57.687708 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:57.687676 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7787d7f5b8-5p8w2"] Jul 22 03:00:57.688198 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:57.687951 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe5ad29f-dbb7-4353-80ea-60bf10139703" containerName="registry" Jul 22 03:00:57.688198 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:57.687961 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe5ad29f-dbb7-4353-80ea-60bf10139703" containerName="registry" Jul 22 03:00:57.688198 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:57.687985 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bab72638-4717-4ea0-8a68-b4d9a1138154" containerName="console" Jul 22 03:00:57.688198 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:57.687990 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="bab72638-4717-4ea0-8a68-b4d9a1138154" containerName="console" Jul 22 03:00:57.688198 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:57.688048 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="bab72638-4717-4ea0-8a68-b4d9a1138154" containerName="console" Jul 22 03:00:57.688198 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:57.688055 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe5ad29f-dbb7-4353-80ea-60bf10139703" containerName="registry" Jul 22 03:00:57.692906 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:57.692868 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7787d7f5b8-5p8w2" Jul 22 03:00:57.695919 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:57.695896 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 22 03:00:57.695919 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:57.695899 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-wcrkn\"" Jul 22 03:00:57.696969 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:57.696806 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 22 03:00:57.698221 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:57.697663 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 22 03:00:57.698221 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:57.697855 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 22 03:00:57.698221 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:57.698054 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 22 03:00:57.702199 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:57.702181 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 22 03:00:57.705976 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:57.705955 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7787d7f5b8-5p8w2"] Jul 22 03:00:57.753457 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:57.753424 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/184bbaf0-8590-451b-9235-f9c9ab652e55-serving-certs-ca-bundle\") pod \"telemeter-client-7787d7f5b8-5p8w2\" (UID: \"184bbaf0-8590-451b-9235-f9c9ab652e55\") " pod="openshift-monitoring/telemeter-client-7787d7f5b8-5p8w2" Jul 22 03:00:57.753457 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:57.753455 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/184bbaf0-8590-451b-9235-f9c9ab652e55-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7787d7f5b8-5p8w2\" (UID: \"184bbaf0-8590-451b-9235-f9c9ab652e55\") " pod="openshift-monitoring/telemeter-client-7787d7f5b8-5p8w2" Jul 22 03:00:57.753658 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:57.753478 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/184bbaf0-8590-451b-9235-f9c9ab652e55-secret-telemeter-client\") pod \"telemeter-client-7787d7f5b8-5p8w2\" (UID: \"184bbaf0-8590-451b-9235-f9c9ab652e55\") " pod="openshift-monitoring/telemeter-client-7787d7f5b8-5p8w2" Jul 22 03:00:57.753658 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:57.753561 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/184bbaf0-8590-451b-9235-f9c9ab652e55-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7787d7f5b8-5p8w2\" (UID: \"184bbaf0-8590-451b-9235-f9c9ab652e55\") " pod="openshift-monitoring/telemeter-client-7787d7f5b8-5p8w2" Jul 22 03:00:57.753658 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:57.753587 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/184bbaf0-8590-451b-9235-f9c9ab652e55-telemeter-client-tls\") pod \"telemeter-client-7787d7f5b8-5p8w2\" (UID: \"184bbaf0-8590-451b-9235-f9c9ab652e55\") " pod="openshift-monitoring/telemeter-client-7787d7f5b8-5p8w2" Jul 22 03:00:57.753658 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:57.753616 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/184bbaf0-8590-451b-9235-f9c9ab652e55-federate-client-tls\") pod \"telemeter-client-7787d7f5b8-5p8w2\" (UID: \"184bbaf0-8590-451b-9235-f9c9ab652e55\") " pod="openshift-monitoring/telemeter-client-7787d7f5b8-5p8w2" Jul 22 03:00:57.753658 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:57.753636 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/184bbaf0-8590-451b-9235-f9c9ab652e55-metrics-client-ca\") pod \"telemeter-client-7787d7f5b8-5p8w2\" (UID: \"184bbaf0-8590-451b-9235-f9c9ab652e55\") " pod="openshift-monitoring/telemeter-client-7787d7f5b8-5p8w2" Jul 22 03:00:57.753811 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:57.753667 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ps2x\" (UniqueName: \"kubernetes.io/projected/184bbaf0-8590-451b-9235-f9c9ab652e55-kube-api-access-6ps2x\") pod \"telemeter-client-7787d7f5b8-5p8w2\" (UID: \"184bbaf0-8590-451b-9235-f9c9ab652e55\") " pod="openshift-monitoring/telemeter-client-7787d7f5b8-5p8w2" Jul 22 03:00:57.854150 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:57.854092 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/184bbaf0-8590-451b-9235-f9c9ab652e55-federate-client-tls\") pod \"telemeter-client-7787d7f5b8-5p8w2\" (UID: \"184bbaf0-8590-451b-9235-f9c9ab652e55\") " pod="openshift-monitoring/telemeter-client-7787d7f5b8-5p8w2" Jul 22 03:00:57.854150 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:57.854148 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/184bbaf0-8590-451b-9235-f9c9ab652e55-metrics-client-ca\") pod \"telemeter-client-7787d7f5b8-5p8w2\" (UID: \"184bbaf0-8590-451b-9235-f9c9ab652e55\") " pod="openshift-monitoring/telemeter-client-7787d7f5b8-5p8w2" Jul 22 03:00:57.854432 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:57.854184 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ps2x\" (UniqueName: \"kubernetes.io/projected/184bbaf0-8590-451b-9235-f9c9ab652e55-kube-api-access-6ps2x\") pod \"telemeter-client-7787d7f5b8-5p8w2\" (UID: \"184bbaf0-8590-451b-9235-f9c9ab652e55\") " pod="openshift-monitoring/telemeter-client-7787d7f5b8-5p8w2" Jul 22 03:00:57.854432 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:57.854205 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/184bbaf0-8590-451b-9235-f9c9ab652e55-serving-certs-ca-bundle\") pod \"telemeter-client-7787d7f5b8-5p8w2\" (UID: \"184bbaf0-8590-451b-9235-f9c9ab652e55\") " pod="openshift-monitoring/telemeter-client-7787d7f5b8-5p8w2" Jul 22 03:00:57.854432 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:57.854222 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/184bbaf0-8590-451b-9235-f9c9ab652e55-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7787d7f5b8-5p8w2\" (UID: \"184bbaf0-8590-451b-9235-f9c9ab652e55\") " pod="openshift-monitoring/telemeter-client-7787d7f5b8-5p8w2" Jul 22 03:00:57.854432 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:57.854254 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/184bbaf0-8590-451b-9235-f9c9ab652e55-secret-telemeter-client\") pod \"telemeter-client-7787d7f5b8-5p8w2\" (UID: \"184bbaf0-8590-451b-9235-f9c9ab652e55\") " pod="openshift-monitoring/telemeter-client-7787d7f5b8-5p8w2" Jul 22 03:00:57.854432 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:57.854297 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/184bbaf0-8590-451b-9235-f9c9ab652e55-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7787d7f5b8-5p8w2\" (UID: \"184bbaf0-8590-451b-9235-f9c9ab652e55\") " pod="openshift-monitoring/telemeter-client-7787d7f5b8-5p8w2" Jul 22 03:00:57.854432 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:57.854326 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/184bbaf0-8590-451b-9235-f9c9ab652e55-telemeter-client-tls\") pod \"telemeter-client-7787d7f5b8-5p8w2\" (UID: \"184bbaf0-8590-451b-9235-f9c9ab652e55\") " pod="openshift-monitoring/telemeter-client-7787d7f5b8-5p8w2" Jul 22 03:00:57.855166 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:57.855102 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/184bbaf0-8590-451b-9235-f9c9ab652e55-serving-certs-ca-bundle\") pod \"telemeter-client-7787d7f5b8-5p8w2\" (UID: \"184bbaf0-8590-451b-9235-f9c9ab652e55\") " pod="openshift-monitoring/telemeter-client-7787d7f5b8-5p8w2" Jul 22 03:00:57.855166 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:57.855142 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/184bbaf0-8590-451b-9235-f9c9ab652e55-metrics-client-ca\") pod \"telemeter-client-7787d7f5b8-5p8w2\" (UID: \"184bbaf0-8590-451b-9235-f9c9ab652e55\") " pod="openshift-monitoring/telemeter-client-7787d7f5b8-5p8w2" Jul 22 03:00:57.855395 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:57.855374 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/184bbaf0-8590-451b-9235-f9c9ab652e55-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7787d7f5b8-5p8w2\" (UID: \"184bbaf0-8590-451b-9235-f9c9ab652e55\") " pod="openshift-monitoring/telemeter-client-7787d7f5b8-5p8w2" Jul 22 03:00:57.856782 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:57.856756 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/184bbaf0-8590-451b-9235-f9c9ab652e55-secret-telemeter-client\") pod \"telemeter-client-7787d7f5b8-5p8w2\" (UID: \"184bbaf0-8590-451b-9235-f9c9ab652e55\") " pod="openshift-monitoring/telemeter-client-7787d7f5b8-5p8w2" Jul 22 03:00:57.856904 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:57.856889 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/184bbaf0-8590-451b-9235-f9c9ab652e55-telemeter-client-tls\") pod \"telemeter-client-7787d7f5b8-5p8w2\" (UID: \"184bbaf0-8590-451b-9235-f9c9ab652e55\") " pod="openshift-monitoring/telemeter-client-7787d7f5b8-5p8w2" Jul 22 03:00:57.857076 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:57.857059 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/184bbaf0-8590-451b-9235-f9c9ab652e55-federate-client-tls\") pod \"telemeter-client-7787d7f5b8-5p8w2\" (UID: \"184bbaf0-8590-451b-9235-f9c9ab652e55\") " pod="openshift-monitoring/telemeter-client-7787d7f5b8-5p8w2" Jul 22 03:00:57.857158 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:57.857136 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/184bbaf0-8590-451b-9235-f9c9ab652e55-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7787d7f5b8-5p8w2\" (UID: \"184bbaf0-8590-451b-9235-f9c9ab652e55\") " pod="openshift-monitoring/telemeter-client-7787d7f5b8-5p8w2" Jul 22 03:00:57.863657 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:57.863618 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ps2x\" (UniqueName: \"kubernetes.io/projected/184bbaf0-8590-451b-9235-f9c9ab652e55-kube-api-access-6ps2x\") pod \"telemeter-client-7787d7f5b8-5p8w2\" (UID: \"184bbaf0-8590-451b-9235-f9c9ab652e55\") " pod="openshift-monitoring/telemeter-client-7787d7f5b8-5p8w2" Jul 22 03:00:58.004785 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:58.004685 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7787d7f5b8-5p8w2" Jul 22 03:00:58.148925 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:58.148839 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7787d7f5b8-5p8w2"] Jul 22 03:00:58.151641 ip-10-0-131-67 kubenswrapper[2582]: W0722 03:00:58.151613 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod184bbaf0_8590_451b_9235_f9c9ab652e55.slice/crio-8191e8b15b07b4a882a950365ef6b892f9dd403d8687a8703d375050b406201f WatchSource:0}: Error finding container 8191e8b15b07b4a882a950365ef6b892f9dd403d8687a8703d375050b406201f: Status 404 returned error can't find the container with id 8191e8b15b07b4a882a950365ef6b892f9dd403d8687a8703d375050b406201f Jul 22 03:00:58.747186 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:00:58.747148 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7787d7f5b8-5p8w2" event={"ID":"184bbaf0-8590-451b-9235-f9c9ab652e55","Type":"ContainerStarted","Data":"8191e8b15b07b4a882a950365ef6b892f9dd403d8687a8703d375050b406201f"} Jul 22 03:01:00.756060 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:00.756000 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7787d7f5b8-5p8w2" event={"ID":"184bbaf0-8590-451b-9235-f9c9ab652e55","Type":"ContainerStarted","Data":"f95c306928e14eb1ec8640e1195de62b1f990e549e3102bcec3ee1ab9ad305ec"} Jul 22 03:01:01.761740 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:01.761650 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7787d7f5b8-5p8w2" event={"ID":"184bbaf0-8590-451b-9235-f9c9ab652e55","Type":"ContainerStarted","Data":"d904dc06a913f0b300f7dba81ea79a98c7c0a7d637a9d7805d2e8041dc77db83"} Jul 22 03:01:01.761740 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:01.761706 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7787d7f5b8-5p8w2" event={"ID":"184bbaf0-8590-451b-9235-f9c9ab652e55","Type":"ContainerStarted","Data":"b82fdeeecba925990532728a337f9317c8d394c32213035fab591996d3892ea3"} Jul 22 03:01:01.789878 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:01.789821 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7787d7f5b8-5p8w2" podStartSLOduration=1.5519701609999998 podStartE2EDuration="4.789803063s" totalImagesPullingTime="3.237832902s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:00:57 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:01:01.789313609 +0000 UTC m=+172.520934457" watchObservedRunningTime="2026-07-22 03:01:01.789803063 +0000 UTC m=+172.521423909" Jul 22 03:01:02.500486 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:02.500454 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-85b5466b7c-5nl8n"] Jul 22 03:01:02.503601 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:02.503583 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85b5466b7c-5nl8n" Jul 22 03:01:02.508510 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:02.508489 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 22 03:01:02.508954 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:02.508830 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 22 03:01:02.509704 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:02.509674 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 22 03:01:02.510058 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:02.510041 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 22 03:01:02.510184 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:02.510084 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 22 03:01:02.510184 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:02.510154 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 22 03:01:02.510584 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:02.510042 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-srtb2\"" Jul 22 03:01:02.510659 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:02.510638 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 22 03:01:02.515054 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:02.515028 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 22 03:01:02.524269 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:02.524233 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85b5466b7c-5nl8n"] Jul 22 03:01:02.580061 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:02.580005 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:01:02.589216 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:02.589181 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b6d25229-6309-410b-9a7b-8d9e8b6a963b-oauth-serving-cert\") pod \"console-85b5466b7c-5nl8n\" (UID: \"b6d25229-6309-410b-9a7b-8d9e8b6a963b\") " pod="openshift-console/console-85b5466b7c-5nl8n" Jul 22 03:01:02.589324 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:02.589247 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b6d25229-6309-410b-9a7b-8d9e8b6a963b-console-oauth-config\") pod \"console-85b5466b7c-5nl8n\" (UID: \"b6d25229-6309-410b-9a7b-8d9e8b6a963b\") " pod="openshift-console/console-85b5466b7c-5nl8n" Jul 22 03:01:02.589364 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:02.589323 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b6d25229-6309-410b-9a7b-8d9e8b6a963b-service-ca\") pod \"console-85b5466b7c-5nl8n\" (UID: \"b6d25229-6309-410b-9a7b-8d9e8b6a963b\") " pod="openshift-console/console-85b5466b7c-5nl8n" Jul 22 03:01:02.589426 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:02.589360 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6d25229-6309-410b-9a7b-8d9e8b6a963b-console-serving-cert\") pod \"console-85b5466b7c-5nl8n\" (UID: \"b6d25229-6309-410b-9a7b-8d9e8b6a963b\") " pod="openshift-console/console-85b5466b7c-5nl8n" Jul 22 03:01:02.589426 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:02.589378 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6d25229-6309-410b-9a7b-8d9e8b6a963b-trusted-ca-bundle\") pod \"console-85b5466b7c-5nl8n\" (UID: \"b6d25229-6309-410b-9a7b-8d9e8b6a963b\") " pod="openshift-console/console-85b5466b7c-5nl8n" Jul 22 03:01:02.589426 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:02.589399 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7x27\" (UniqueName: \"kubernetes.io/projected/b6d25229-6309-410b-9a7b-8d9e8b6a963b-kube-api-access-s7x27\") pod \"console-85b5466b7c-5nl8n\" (UID: \"b6d25229-6309-410b-9a7b-8d9e8b6a963b\") " pod="openshift-console/console-85b5466b7c-5nl8n" Jul 22 03:01:02.589517 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:02.589446 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b6d25229-6309-410b-9a7b-8d9e8b6a963b-console-config\") pod \"console-85b5466b7c-5nl8n\" (UID: \"b6d25229-6309-410b-9a7b-8d9e8b6a963b\") " pod="openshift-console/console-85b5466b7c-5nl8n" Jul 22 03:01:02.690197 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:02.690153 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b6d25229-6309-410b-9a7b-8d9e8b6a963b-console-config\") pod \"console-85b5466b7c-5nl8n\" (UID: \"b6d25229-6309-410b-9a7b-8d9e8b6a963b\") " pod="openshift-console/console-85b5466b7c-5nl8n" Jul 22 03:01:02.690197 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:02.690195 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b6d25229-6309-410b-9a7b-8d9e8b6a963b-oauth-serving-cert\") pod \"console-85b5466b7c-5nl8n\" (UID: \"b6d25229-6309-410b-9a7b-8d9e8b6a963b\") " pod="openshift-console/console-85b5466b7c-5nl8n" Jul 22 03:01:02.690453 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:02.690243 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b6d25229-6309-410b-9a7b-8d9e8b6a963b-console-oauth-config\") pod \"console-85b5466b7c-5nl8n\" (UID: \"b6d25229-6309-410b-9a7b-8d9e8b6a963b\") " pod="openshift-console/console-85b5466b7c-5nl8n" Jul 22 03:01:02.690453 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:02.690270 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b6d25229-6309-410b-9a7b-8d9e8b6a963b-service-ca\") pod \"console-85b5466b7c-5nl8n\" (UID: \"b6d25229-6309-410b-9a7b-8d9e8b6a963b\") " pod="openshift-console/console-85b5466b7c-5nl8n" Jul 22 03:01:02.690453 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:02.690301 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6d25229-6309-410b-9a7b-8d9e8b6a963b-console-serving-cert\") pod \"console-85b5466b7c-5nl8n\" (UID: \"b6d25229-6309-410b-9a7b-8d9e8b6a963b\") " pod="openshift-console/console-85b5466b7c-5nl8n" Jul 22 03:01:02.690453 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:02.690326 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6d25229-6309-410b-9a7b-8d9e8b6a963b-trusted-ca-bundle\") pod \"console-85b5466b7c-5nl8n\" (UID: \"b6d25229-6309-410b-9a7b-8d9e8b6a963b\") " pod="openshift-console/console-85b5466b7c-5nl8n" Jul 22 03:01:02.690453 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:02.690356 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7x27\" (UniqueName: \"kubernetes.io/projected/b6d25229-6309-410b-9a7b-8d9e8b6a963b-kube-api-access-s7x27\") pod \"console-85b5466b7c-5nl8n\" (UID: \"b6d25229-6309-410b-9a7b-8d9e8b6a963b\") " pod="openshift-console/console-85b5466b7c-5nl8n" Jul 22 03:01:02.691072 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:02.691038 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b6d25229-6309-410b-9a7b-8d9e8b6a963b-oauth-serving-cert\") pod \"console-85b5466b7c-5nl8n\" (UID: \"b6d25229-6309-410b-9a7b-8d9e8b6a963b\") " pod="openshift-console/console-85b5466b7c-5nl8n" Jul 22 03:01:02.691279 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:02.691254 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b6d25229-6309-410b-9a7b-8d9e8b6a963b-service-ca\") pod \"console-85b5466b7c-5nl8n\" (UID: \"b6d25229-6309-410b-9a7b-8d9e8b6a963b\") " pod="openshift-console/console-85b5466b7c-5nl8n" Jul 22 03:01:02.691366 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:02.691284 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6d25229-6309-410b-9a7b-8d9e8b6a963b-trusted-ca-bundle\") pod \"console-85b5466b7c-5nl8n\" (UID: \"b6d25229-6309-410b-9a7b-8d9e8b6a963b\") " pod="openshift-console/console-85b5466b7c-5nl8n" Jul 22 03:01:02.691438 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:02.691413 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b6d25229-6309-410b-9a7b-8d9e8b6a963b-console-config\") pod \"console-85b5466b7c-5nl8n\" (UID: \"b6d25229-6309-410b-9a7b-8d9e8b6a963b\") " pod="openshift-console/console-85b5466b7c-5nl8n" Jul 22 03:01:02.694411 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:02.694384 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b6d25229-6309-410b-9a7b-8d9e8b6a963b-console-oauth-config\") pod \"console-85b5466b7c-5nl8n\" (UID: \"b6d25229-6309-410b-9a7b-8d9e8b6a963b\") " pod="openshift-console/console-85b5466b7c-5nl8n" Jul 22 03:01:02.694712 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:02.694693 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6d25229-6309-410b-9a7b-8d9e8b6a963b-console-serving-cert\") pod \"console-85b5466b7c-5nl8n\" (UID: \"b6d25229-6309-410b-9a7b-8d9e8b6a963b\") " pod="openshift-console/console-85b5466b7c-5nl8n" Jul 22 03:01:02.701466 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:02.701434 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7x27\" (UniqueName: \"kubernetes.io/projected/b6d25229-6309-410b-9a7b-8d9e8b6a963b-kube-api-access-s7x27\") pod \"console-85b5466b7c-5nl8n\" (UID: \"b6d25229-6309-410b-9a7b-8d9e8b6a963b\") " pod="openshift-console/console-85b5466b7c-5nl8n" Jul 22 03:01:02.814077 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:02.813965 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85b5466b7c-5nl8n" Jul 22 03:01:02.975165 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:02.974596 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85b5466b7c-5nl8n"] Jul 22 03:01:02.978912 ip-10-0-131-67 kubenswrapper[2582]: W0722 03:01:02.978882 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6d25229_6309_410b_9a7b_8d9e8b6a963b.slice/crio-3e121c6a9035a25e1959072eb9f50d04f2c85972c80a28669bed58c413c4be39 WatchSource:0}: Error finding container 3e121c6a9035a25e1959072eb9f50d04f2c85972c80a28669bed58c413c4be39: Status 404 returned error can't find the container with id 3e121c6a9035a25e1959072eb9f50d04f2c85972c80a28669bed58c413c4be39 Jul 22 03:01:03.771772 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:03.771727 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85b5466b7c-5nl8n" event={"ID":"b6d25229-6309-410b-9a7b-8d9e8b6a963b","Type":"ContainerStarted","Data":"be8ecc59d55b39d2b9d71d273b4e6bb90dffa426f827e61100d20515bb8d8f19"} Jul 22 03:01:03.771963 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:03.771818 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85b5466b7c-5nl8n" event={"ID":"b6d25229-6309-410b-9a7b-8d9e8b6a963b","Type":"ContainerStarted","Data":"3e121c6a9035a25e1959072eb9f50d04f2c85972c80a28669bed58c413c4be39"} Jul 22 03:01:03.795420 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:03.795359 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-85b5466b7c-5nl8n" podStartSLOduration=1.795342871 podStartE2EDuration="1.795342871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:01:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:01:03.794059155 +0000 UTC m=+174.525680027" watchObservedRunningTime="2026-07-22 03:01:03.795342871 +0000 UTC m=+174.526963717" Jul 22 03:01:07.833552 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:07.833516 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:01:07.914727 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:07.914689 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:01:08.180728 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:08.180695 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-85b5466b7c-5nl8n"] Jul 22 03:01:12.815153 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:12.815113 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-85b5466b7c-5nl8n" Jul 22 03:01:19.364894 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:19.364854 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:01:19.468258 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:19.468225 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:01:19.531036 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:19.530983 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:01:19.882102 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:19.882069 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:01:19.959091 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:19.959054 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:01:26.017266 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:26.017219 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:01:26.121912 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:26.121883 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:01:33.200261 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:33.200199 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-85b5466b7c-5nl8n" podUID="b6d25229-6309-410b-9a7b-8d9e8b6a963b" containerName="console" containerID="cri-o://be8ecc59d55b39d2b9d71d273b4e6bb90dffa426f827e61100d20515bb8d8f19" gracePeriod=15 Jul 22 03:01:33.441233 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:33.441207 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85b5466b7c-5nl8n" Jul 22 03:01:33.532349 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:33.532257 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6d25229-6309-410b-9a7b-8d9e8b6a963b-trusted-ca-bundle\") pod \"b6d25229-6309-410b-9a7b-8d9e8b6a963b\" (UID: \"b6d25229-6309-410b-9a7b-8d9e8b6a963b\") " Jul 22 03:01:33.532349 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:33.532299 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6d25229-6309-410b-9a7b-8d9e8b6a963b-console-serving-cert\") pod \"b6d25229-6309-410b-9a7b-8d9e8b6a963b\" (UID: \"b6d25229-6309-410b-9a7b-8d9e8b6a963b\") " Jul 22 03:01:33.532349 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:33.532334 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s7x27\" (UniqueName: \"kubernetes.io/projected/b6d25229-6309-410b-9a7b-8d9e8b6a963b-kube-api-access-s7x27\") pod \"b6d25229-6309-410b-9a7b-8d9e8b6a963b\" (UID: \"b6d25229-6309-410b-9a7b-8d9e8b6a963b\") " Jul 22 03:01:33.532621 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:33.532363 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b6d25229-6309-410b-9a7b-8d9e8b6a963b-console-oauth-config\") pod \"b6d25229-6309-410b-9a7b-8d9e8b6a963b\" (UID: \"b6d25229-6309-410b-9a7b-8d9e8b6a963b\") " Jul 22 03:01:33.532621 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:33.532383 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b6d25229-6309-410b-9a7b-8d9e8b6a963b-service-ca\") pod \"b6d25229-6309-410b-9a7b-8d9e8b6a963b\" (UID: \"b6d25229-6309-410b-9a7b-8d9e8b6a963b\") " Jul 22 03:01:33.532621 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:33.532433 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b6d25229-6309-410b-9a7b-8d9e8b6a963b-oauth-serving-cert\") pod \"b6d25229-6309-410b-9a7b-8d9e8b6a963b\" (UID: \"b6d25229-6309-410b-9a7b-8d9e8b6a963b\") " Jul 22 03:01:33.532621 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:33.532457 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b6d25229-6309-410b-9a7b-8d9e8b6a963b-console-config\") pod \"b6d25229-6309-410b-9a7b-8d9e8b6a963b\" (UID: \"b6d25229-6309-410b-9a7b-8d9e8b6a963b\") " Jul 22 03:01:33.532924 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:33.532892 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6d25229-6309-410b-9a7b-8d9e8b6a963b-service-ca" (OuterVolumeSpecName: "service-ca") pod "b6d25229-6309-410b-9a7b-8d9e8b6a963b" (UID: "b6d25229-6309-410b-9a7b-8d9e8b6a963b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 03:01:33.532978 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:33.532914 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6d25229-6309-410b-9a7b-8d9e8b6a963b-console-config" (OuterVolumeSpecName: "console-config") pod "b6d25229-6309-410b-9a7b-8d9e8b6a963b" (UID: "b6d25229-6309-410b-9a7b-8d9e8b6a963b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 03:01:33.532978 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:33.532920 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6d25229-6309-410b-9a7b-8d9e8b6a963b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b6d25229-6309-410b-9a7b-8d9e8b6a963b" (UID: "b6d25229-6309-410b-9a7b-8d9e8b6a963b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 03:01:33.533099 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:33.533065 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6d25229-6309-410b-9a7b-8d9e8b6a963b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b6d25229-6309-410b-9a7b-8d9e8b6a963b" (UID: "b6d25229-6309-410b-9a7b-8d9e8b6a963b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 03:01:33.534480 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:33.534464 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6d25229-6309-410b-9a7b-8d9e8b6a963b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b6d25229-6309-410b-9a7b-8d9e8b6a963b" (UID: "b6d25229-6309-410b-9a7b-8d9e8b6a963b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 03:01:33.534892 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:33.534863 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6d25229-6309-410b-9a7b-8d9e8b6a963b-kube-api-access-s7x27" (OuterVolumeSpecName: "kube-api-access-s7x27") pod "b6d25229-6309-410b-9a7b-8d9e8b6a963b" (UID: "b6d25229-6309-410b-9a7b-8d9e8b6a963b"). InnerVolumeSpecName "kube-api-access-s7x27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 03:01:33.534892 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:33.534873 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6d25229-6309-410b-9a7b-8d9e8b6a963b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b6d25229-6309-410b-9a7b-8d9e8b6a963b" (UID: "b6d25229-6309-410b-9a7b-8d9e8b6a963b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 03:01:33.633930 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:33.633882 2582 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b6d25229-6309-410b-9a7b-8d9e8b6a963b-oauth-serving-cert\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 03:01:33.633930 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:33.633938 2582 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b6d25229-6309-410b-9a7b-8d9e8b6a963b-console-config\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 03:01:33.633930 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:33.633948 2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6d25229-6309-410b-9a7b-8d9e8b6a963b-trusted-ca-bundle\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 03:01:33.634185 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:33.633958 2582 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6d25229-6309-410b-9a7b-8d9e8b6a963b-console-serving-cert\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 03:01:33.634185 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:33.633966 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7x27\" (UniqueName: \"kubernetes.io/projected/b6d25229-6309-410b-9a7b-8d9e8b6a963b-kube-api-access-s7x27\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 03:01:33.634185 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:33.633976 2582 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b6d25229-6309-410b-9a7b-8d9e8b6a963b-console-oauth-config\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 03:01:33.634185 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:33.633984 2582 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b6d25229-6309-410b-9a7b-8d9e8b6a963b-service-ca\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 03:01:33.884606 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:33.884570 2582 generic.go:356] "Generic (PLEG): container finished" podID="b6d25229-6309-410b-9a7b-8d9e8b6a963b" containerID="be8ecc59d55b39d2b9d71d273b4e6bb90dffa426f827e61100d20515bb8d8f19" exitCode=0 Jul 22 03:01:33.884800 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:33.884623 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85b5466b7c-5nl8n" Jul 22 03:01:33.884800 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:33.884653 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85b5466b7c-5nl8n" event={"ID":"b6d25229-6309-410b-9a7b-8d9e8b6a963b","Type":"ContainerDied","Data":"be8ecc59d55b39d2b9d71d273b4e6bb90dffa426f827e61100d20515bb8d8f19"} Jul 22 03:01:33.884800 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:33.884703 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85b5466b7c-5nl8n" event={"ID":"b6d25229-6309-410b-9a7b-8d9e8b6a963b","Type":"ContainerDied","Data":"3e121c6a9035a25e1959072eb9f50d04f2c85972c80a28669bed58c413c4be39"} Jul 22 03:01:33.884800 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:33.884724 2582 scope.go:117] "RemoveContainer" containerID="be8ecc59d55b39d2b9d71d273b4e6bb90dffa426f827e61100d20515bb8d8f19" Jul 22 03:01:33.895519 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:33.895349 2582 scope.go:117] "RemoveContainer" containerID="be8ecc59d55b39d2b9d71d273b4e6bb90dffa426f827e61100d20515bb8d8f19" Jul 22 03:01:33.895767 ip-10-0-131-67 kubenswrapper[2582]: E0722 03:01:33.895739 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be8ecc59d55b39d2b9d71d273b4e6bb90dffa426f827e61100d20515bb8d8f19\": container with ID starting with be8ecc59d55b39d2b9d71d273b4e6bb90dffa426f827e61100d20515bb8d8f19 not found: ID does not exist" containerID="be8ecc59d55b39d2b9d71d273b4e6bb90dffa426f827e61100d20515bb8d8f19" Jul 22 03:01:33.895846 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:33.895779 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be8ecc59d55b39d2b9d71d273b4e6bb90dffa426f827e61100d20515bb8d8f19"} err="failed to get container status \"be8ecc59d55b39d2b9d71d273b4e6bb90dffa426f827e61100d20515bb8d8f19\": rpc error: code = NotFound desc = could not find container \"be8ecc59d55b39d2b9d71d273b4e6bb90dffa426f827e61100d20515bb8d8f19\": container with ID starting with be8ecc59d55b39d2b9d71d273b4e6bb90dffa426f827e61100d20515bb8d8f19 not found: ID does not exist" Jul 22 03:01:33.908535 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:33.908509 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-85b5466b7c-5nl8n"] Jul 22 03:01:33.914067 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:33.914044 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-85b5466b7c-5nl8n"] Jul 22 03:01:35.899435 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:35.899402 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6d25229-6309-410b-9a7b-8d9e8b6a963b" path="/var/lib/kubelet/pods/b6d25229-6309-410b-9a7b-8d9e8b6a963b/volumes" Jul 22 03:01:50.762650 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:50.762617 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:01:50.867176 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:50.867137 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:01:52.030858 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:52.030822 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:01:52.114182 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:52.114151 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:01:53.117148 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:53.117118 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q6m64"] Jul 22 03:01:53.117517 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:53.117400 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6d25229-6309-410b-9a7b-8d9e8b6a963b" containerName="console" Jul 22 03:01:53.117517 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:53.117411 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6d25229-6309-410b-9a7b-8d9e8b6a963b" containerName="console" Jul 22 03:01:53.117517 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:53.117467 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6d25229-6309-410b-9a7b-8d9e8b6a963b" containerName="console" Jul 22 03:01:53.121713 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:53.121695 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q6m64" Jul 22 03:01:53.125227 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:53.125206 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 22 03:01:53.125919 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:53.125903 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-44dcl\"" Jul 22 03:01:53.126033 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:53.125905 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 22 03:01:53.134983 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:53.134959 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q6m64"] Jul 22 03:01:53.298106 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:53.298049 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/454330bb-656d-4c4e-a278-b61ac06dd5e6-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q6m64\" (UID: \"454330bb-656d-4c4e-a278-b61ac06dd5e6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q6m64" Jul 22 03:01:53.298296 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:53.298129 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/454330bb-656d-4c4e-a278-b61ac06dd5e6-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q6m64\" (UID: \"454330bb-656d-4c4e-a278-b61ac06dd5e6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q6m64" Jul 22 03:01:53.298296 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:53.298190 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r9k5\" (UniqueName: \"kubernetes.io/projected/454330bb-656d-4c4e-a278-b61ac06dd5e6-kube-api-access-4r9k5\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q6m64\" (UID: \"454330bb-656d-4c4e-a278-b61ac06dd5e6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q6m64" Jul 22 03:01:53.398873 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:53.398829 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/454330bb-656d-4c4e-a278-b61ac06dd5e6-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q6m64\" (UID: \"454330bb-656d-4c4e-a278-b61ac06dd5e6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q6m64" Jul 22 03:01:53.399043 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:53.398897 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/454330bb-656d-4c4e-a278-b61ac06dd5e6-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q6m64\" (UID: \"454330bb-656d-4c4e-a278-b61ac06dd5e6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q6m64" Jul 22 03:01:53.399043 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:53.398928 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r9k5\" (UniqueName: \"kubernetes.io/projected/454330bb-656d-4c4e-a278-b61ac06dd5e6-kube-api-access-4r9k5\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q6m64\" (UID: \"454330bb-656d-4c4e-a278-b61ac06dd5e6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q6m64" Jul 22 03:01:53.399242 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:53.399221 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/454330bb-656d-4c4e-a278-b61ac06dd5e6-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q6m64\" (UID: \"454330bb-656d-4c4e-a278-b61ac06dd5e6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q6m64" Jul 22 03:01:53.399315 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:53.399299 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/454330bb-656d-4c4e-a278-b61ac06dd5e6-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q6m64\" (UID: \"454330bb-656d-4c4e-a278-b61ac06dd5e6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q6m64" Jul 22 03:01:53.409133 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:53.409112 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r9k5\" (UniqueName: \"kubernetes.io/projected/454330bb-656d-4c4e-a278-b61ac06dd5e6-kube-api-access-4r9k5\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q6m64\" (UID: \"454330bb-656d-4c4e-a278-b61ac06dd5e6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q6m64" Jul 22 03:01:53.430093 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:53.430053 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q6m64" Jul 22 03:01:53.571870 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:53.571837 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q6m64"] Jul 22 03:01:53.577097 ip-10-0-131-67 kubenswrapper[2582]: W0722 03:01:53.577071 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod454330bb_656d_4c4e_a278_b61ac06dd5e6.slice/crio-a07811aa9d27567538b2b694b96e4b83ccd2a06cad1dd73bec9199545795f779 WatchSource:0}: Error finding container a07811aa9d27567538b2b694b96e4b83ccd2a06cad1dd73bec9199545795f779: Status 404 returned error can't find the container with id a07811aa9d27567538b2b694b96e4b83ccd2a06cad1dd73bec9199545795f779 Jul 22 03:01:53.968087 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:01:53.968049 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q6m64" event={"ID":"454330bb-656d-4c4e-a278-b61ac06dd5e6","Type":"ContainerStarted","Data":"a07811aa9d27567538b2b694b96e4b83ccd2a06cad1dd73bec9199545795f779"} Jul 22 03:02:00.998892 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:00.998858 2582 generic.go:356] "Generic (PLEG): container finished" podID="454330bb-656d-4c4e-a278-b61ac06dd5e6" containerID="9180088cc221c0bb9d5331bf7bb6ba01ad1572ded72a25da0b086fbfd1774603" exitCode=0 Jul 22 03:02:00.999316 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:00.998908 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q6m64" event={"ID":"454330bb-656d-4c4e-a278-b61ac06dd5e6","Type":"ContainerDied","Data":"9180088cc221c0bb9d5331bf7bb6ba01ad1572ded72a25da0b086fbfd1774603"} Jul 22 03:02:04.013920 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:04.013885 2582 generic.go:356] "Generic (PLEG): container finished" podID="454330bb-656d-4c4e-a278-b61ac06dd5e6" containerID="4099c634b5316e0742ab812a30c2823c230e169c6b4d7d021e3f3494108f4041" exitCode=0 Jul 22 03:02:04.014317 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:04.013943 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q6m64" event={"ID":"454330bb-656d-4c4e-a278-b61ac06dd5e6","Type":"ContainerDied","Data":"4099c634b5316e0742ab812a30c2823c230e169c6b4d7d021e3f3494108f4041"} Jul 22 03:02:04.160371 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:04.160339 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:02:12.053938 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:12.053902 2582 generic.go:356] "Generic (PLEG): container finished" podID="454330bb-656d-4c4e-a278-b61ac06dd5e6" containerID="42803c8173b78a4eeb525dbd64a3fe21b2a88da34ede85ff76f6f9a46811bb1a" exitCode=0 Jul 22 03:02:12.054377 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:12.053988 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q6m64" event={"ID":"454330bb-656d-4c4e-a278-b61ac06dd5e6","Type":"ContainerDied","Data":"42803c8173b78a4eeb525dbd64a3fe21b2a88da34ede85ff76f6f9a46811bb1a"} Jul 22 03:02:13.177778 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:13.177752 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q6m64" Jul 22 03:02:13.253953 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:13.253923 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/454330bb-656d-4c4e-a278-b61ac06dd5e6-bundle\") pod \"454330bb-656d-4c4e-a278-b61ac06dd5e6\" (UID: \"454330bb-656d-4c4e-a278-b61ac06dd5e6\") " Jul 22 03:02:13.254132 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:13.253984 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/454330bb-656d-4c4e-a278-b61ac06dd5e6-util\") pod \"454330bb-656d-4c4e-a278-b61ac06dd5e6\" (UID: \"454330bb-656d-4c4e-a278-b61ac06dd5e6\") " Jul 22 03:02:13.254132 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:13.254036 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4r9k5\" (UniqueName: \"kubernetes.io/projected/454330bb-656d-4c4e-a278-b61ac06dd5e6-kube-api-access-4r9k5\") pod \"454330bb-656d-4c4e-a278-b61ac06dd5e6\" (UID: \"454330bb-656d-4c4e-a278-b61ac06dd5e6\") " Jul 22 03:02:13.254746 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:13.254719 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/454330bb-656d-4c4e-a278-b61ac06dd5e6-bundle" (OuterVolumeSpecName: "bundle") pod "454330bb-656d-4c4e-a278-b61ac06dd5e6" (UID: "454330bb-656d-4c4e-a278-b61ac06dd5e6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:02:13.256133 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:13.256109 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/454330bb-656d-4c4e-a278-b61ac06dd5e6-kube-api-access-4r9k5" (OuterVolumeSpecName: "kube-api-access-4r9k5") pod "454330bb-656d-4c4e-a278-b61ac06dd5e6" (UID: "454330bb-656d-4c4e-a278-b61ac06dd5e6"). InnerVolumeSpecName "kube-api-access-4r9k5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 03:02:13.258373 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:13.258348 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/454330bb-656d-4c4e-a278-b61ac06dd5e6-util" (OuterVolumeSpecName: "util") pod "454330bb-656d-4c4e-a278-b61ac06dd5e6" (UID: "454330bb-656d-4c4e-a278-b61ac06dd5e6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:02:13.355363 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:13.355275 2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/454330bb-656d-4c4e-a278-b61ac06dd5e6-util\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 03:02:13.355363 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:13.355311 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4r9k5\" (UniqueName: \"kubernetes.io/projected/454330bb-656d-4c4e-a278-b61ac06dd5e6-kube-api-access-4r9k5\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 03:02:13.355363 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:13.355322 2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/454330bb-656d-4c4e-a278-b61ac06dd5e6-bundle\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 03:02:14.062717 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:14.062691 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q6m64" Jul 22 03:02:14.062717 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:14.062693 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q6m64" event={"ID":"454330bb-656d-4c4e-a278-b61ac06dd5e6","Type":"ContainerDied","Data":"a07811aa9d27567538b2b694b96e4b83ccd2a06cad1dd73bec9199545795f779"} Jul 22 03:02:14.062910 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:14.062739 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a07811aa9d27567538b2b694b96e4b83ccd2a06cad1dd73bec9199545795f779" Jul 22 03:02:19.915903 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:19.915863 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:02:20.024120 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:20.024083 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:02:22.178649 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:22.178619 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-gl22z"] Jul 22 03:02:22.179102 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:22.178953 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="454330bb-656d-4c4e-a278-b61ac06dd5e6" containerName="extract" Jul 22 03:02:22.179102 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:22.178964 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="454330bb-656d-4c4e-a278-b61ac06dd5e6" containerName="extract" Jul 22 03:02:22.179102 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:22.178974 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="454330bb-656d-4c4e-a278-b61ac06dd5e6" containerName="util" Jul 22 03:02:22.179102 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:22.178981 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="454330bb-656d-4c4e-a278-b61ac06dd5e6" containerName="util" Jul 22 03:02:22.179102 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:22.178990 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="454330bb-656d-4c4e-a278-b61ac06dd5e6" containerName="pull" Jul 22 03:02:22.179102 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:22.178995 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="454330bb-656d-4c4e-a278-b61ac06dd5e6" containerName="pull" Jul 22 03:02:22.179102 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:22.179065 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="454330bb-656d-4c4e-a278-b61ac06dd5e6" containerName="extract" Jul 22 03:02:22.183148 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:22.183130 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-gl22z" Jul 22 03:02:22.186541 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:22.186517 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 22 03:02:22.186648 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:22.186517 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-96ssz\"" Jul 22 03:02:22.186736 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:22.186721 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 22 03:02:22.203342 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:22.203319 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-gl22z"] Jul 22 03:02:22.222795 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:22.222762 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea662d4d-4053-4ddf-b26f-69d9e47ac2a0-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-gl22z\" (UID: \"ea662d4d-4053-4ddf-b26f-69d9e47ac2a0\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-gl22z" Jul 22 03:02:22.222936 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:22.222828 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fprcl\" (UniqueName: \"kubernetes.io/projected/ea662d4d-4053-4ddf-b26f-69d9e47ac2a0-kube-api-access-fprcl\") pod \"cert-manager-operator-controller-manager-55c444bf86-gl22z\" (UID: \"ea662d4d-4053-4ddf-b26f-69d9e47ac2a0\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-gl22z" Jul 22 03:02:22.323849 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:22.323810 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fprcl\" (UniqueName: \"kubernetes.io/projected/ea662d4d-4053-4ddf-b26f-69d9e47ac2a0-kube-api-access-fprcl\") pod \"cert-manager-operator-controller-manager-55c444bf86-gl22z\" (UID: \"ea662d4d-4053-4ddf-b26f-69d9e47ac2a0\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-gl22z" Jul 22 03:02:22.323990 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:22.323875 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea662d4d-4053-4ddf-b26f-69d9e47ac2a0-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-gl22z\" (UID: \"ea662d4d-4053-4ddf-b26f-69d9e47ac2a0\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-gl22z" Jul 22 03:02:22.324261 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:22.324245 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea662d4d-4053-4ddf-b26f-69d9e47ac2a0-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-gl22z\" (UID: \"ea662d4d-4053-4ddf-b26f-69d9e47ac2a0\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-gl22z" Jul 22 03:02:22.333464 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:22.333436 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fprcl\" (UniqueName: \"kubernetes.io/projected/ea662d4d-4053-4ddf-b26f-69d9e47ac2a0-kube-api-access-fprcl\") pod \"cert-manager-operator-controller-manager-55c444bf86-gl22z\" (UID: \"ea662d4d-4053-4ddf-b26f-69d9e47ac2a0\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-gl22z" Jul 22 03:02:22.492207 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:22.492116 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-gl22z" Jul 22 03:02:22.631004 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:22.630979 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-gl22z"] Jul 22 03:02:22.633857 ip-10-0-131-67 kubenswrapper[2582]: W0722 03:02:22.633825 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea662d4d_4053_4ddf_b26f_69d9e47ac2a0.slice/crio-95e5591fb5acb80c687fb68ade420a267b73e0eacd3c74001ed06360198a9871 WatchSource:0}: Error finding container 95e5591fb5acb80c687fb68ade420a267b73e0eacd3c74001ed06360198a9871: Status 404 returned error can't find the container with id 95e5591fb5acb80c687fb68ade420a267b73e0eacd3c74001ed06360198a9871 Jul 22 03:02:23.094549 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:23.094514 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-gl22z" event={"ID":"ea662d4d-4053-4ddf-b26f-69d9e47ac2a0","Type":"ContainerStarted","Data":"95e5591fb5acb80c687fb68ade420a267b73e0eacd3c74001ed06360198a9871"} Jul 22 03:02:24.263088 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:24.263047 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:02:24.361247 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:24.361206 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:02:25.104755 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:25.104719 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-gl22z" event={"ID":"ea662d4d-4053-4ddf-b26f-69d9e47ac2a0","Type":"ContainerStarted","Data":"80e24d69a402df5ab9960de1714946147ff5ef4808582df34659b70eaaa33c3d"} Jul 22 03:02:25.169492 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:25.169435 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-gl22z" podStartSLOduration=1.283855334 podStartE2EDuration="3.169418969s" totalImagesPullingTime="1.885563635s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:02:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:02:25.168461689 +0000 UTC m=+255.900082554" watchObservedRunningTime="2026-07-22 03:02:25.169418969 +0000 UTC m=+255.901039815" Jul 22 03:02:25.667056 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:25.667005 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:02:25.765400 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:25.765364 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:02:25.885945 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:25.885916 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsg958"] Jul 22 03:02:25.888277 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:25.888256 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsg958" Jul 22 03:02:25.890739 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:25.890719 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 22 03:02:25.891039 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:25.891003 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-44dcl\"" Jul 22 03:02:25.891733 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:25.891716 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 22 03:02:25.908740 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:25.908714 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsg958"] Jul 22 03:02:25.953403 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:25.953314 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76rf6\" (UniqueName: \"kubernetes.io/projected/b5d1f610-59af-40b8-ad1f-41916bfa321b-kube-api-access-76rf6\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsg958\" (UID: \"b5d1f610-59af-40b8-ad1f-41916bfa321b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsg958" Jul 22 03:02:25.953403 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:25.953368 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b5d1f610-59af-40b8-ad1f-41916bfa321b-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsg958\" (UID: \"b5d1f610-59af-40b8-ad1f-41916bfa321b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsg958" Jul 22 03:02:25.953403 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:25.953390 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b5d1f610-59af-40b8-ad1f-41916bfa321b-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsg958\" (UID: \"b5d1f610-59af-40b8-ad1f-41916bfa321b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsg958" Jul 22 03:02:26.054388 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:26.054361 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b5d1f610-59af-40b8-ad1f-41916bfa321b-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsg958\" (UID: \"b5d1f610-59af-40b8-ad1f-41916bfa321b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsg958" Jul 22 03:02:26.054542 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:26.054427 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76rf6\" (UniqueName: \"kubernetes.io/projected/b5d1f610-59af-40b8-ad1f-41916bfa321b-kube-api-access-76rf6\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsg958\" (UID: \"b5d1f610-59af-40b8-ad1f-41916bfa321b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsg958" Jul 22 03:02:26.054542 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:26.054472 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b5d1f610-59af-40b8-ad1f-41916bfa321b-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsg958\" (UID: \"b5d1f610-59af-40b8-ad1f-41916bfa321b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsg958" Jul 22 03:02:26.054832 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:26.054813 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b5d1f610-59af-40b8-ad1f-41916bfa321b-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsg958\" (UID: \"b5d1f610-59af-40b8-ad1f-41916bfa321b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsg958" Jul 22 03:02:26.054870 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:26.054820 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b5d1f610-59af-40b8-ad1f-41916bfa321b-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsg958\" (UID: \"b5d1f610-59af-40b8-ad1f-41916bfa321b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsg958" Jul 22 03:02:26.071923 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:26.071888 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76rf6\" (UniqueName: \"kubernetes.io/projected/b5d1f610-59af-40b8-ad1f-41916bfa321b-kube-api-access-76rf6\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsg958\" (UID: \"b5d1f610-59af-40b8-ad1f-41916bfa321b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsg958" Jul 22 03:02:26.197731 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:26.197700 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsg958" Jul 22 03:02:26.372289 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:26.372264 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsg958"] Jul 22 03:02:26.374421 ip-10-0-131-67 kubenswrapper[2582]: W0722 03:02:26.374394 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5d1f610_59af_40b8_ad1f_41916bfa321b.slice/crio-4d59b2862b99acc096e0a7a35e8f1684566126599c86d7ce3ccd8b69aa502f39 WatchSource:0}: Error finding container 4d59b2862b99acc096e0a7a35e8f1684566126599c86d7ce3ccd8b69aa502f39: Status 404 returned error can't find the container with id 4d59b2862b99acc096e0a7a35e8f1684566126599c86d7ce3ccd8b69aa502f39 Jul 22 03:02:26.416932 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:26.416905 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:02:26.503983 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:26.503951 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:02:27.115422 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:27.115390 2582 generic.go:356] "Generic (PLEG): container finished" podID="b5d1f610-59af-40b8-ad1f-41916bfa321b" containerID="8f9358807af6361b0cb88f0583c24e11eba60a3ab9569d8d33318180f55011d8" exitCode=0 Jul 22 03:02:27.115826 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:27.115453 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsg958" event={"ID":"b5d1f610-59af-40b8-ad1f-41916bfa321b","Type":"ContainerDied","Data":"8f9358807af6361b0cb88f0583c24e11eba60a3ab9569d8d33318180f55011d8"} Jul 22 03:02:27.115826 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:27.115492 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsg958" event={"ID":"b5d1f610-59af-40b8-ad1f-41916bfa321b","Type":"ContainerStarted","Data":"4d59b2862b99acc096e0a7a35e8f1684566126599c86d7ce3ccd8b69aa502f39"} Jul 22 03:02:27.678351 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:27.678317 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-zs7nk"] Jul 22 03:02:27.680704 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:27.680686 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-zs7nk" Jul 22 03:02:27.682949 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:27.682920 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 22 03:02:27.683128 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:27.683109 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-bbtj6\"" Jul 22 03:02:27.683761 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:27.683746 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 22 03:02:27.691739 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:27.691719 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-zs7nk"] Jul 22 03:02:27.769919 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:27.769879 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cc6ebdf0-9f08-4cbf-8325-04afa6e45fd5-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-zs7nk\" (UID: \"cc6ebdf0-9f08-4cbf-8325-04afa6e45fd5\") " pod="cert-manager/cert-manager-webhook-6486ffd997-zs7nk" Jul 22 03:02:27.769919 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:27.769919 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rmx2\" (UniqueName: \"kubernetes.io/projected/cc6ebdf0-9f08-4cbf-8325-04afa6e45fd5-kube-api-access-4rmx2\") pod \"cert-manager-webhook-6486ffd997-zs7nk\" (UID: \"cc6ebdf0-9f08-4cbf-8325-04afa6e45fd5\") " pod="cert-manager/cert-manager-webhook-6486ffd997-zs7nk" Jul 22 03:02:27.871223 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:27.871184 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cc6ebdf0-9f08-4cbf-8325-04afa6e45fd5-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-zs7nk\" (UID: \"cc6ebdf0-9f08-4cbf-8325-04afa6e45fd5\") " pod="cert-manager/cert-manager-webhook-6486ffd997-zs7nk" Jul 22 03:02:27.871223 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:27.871222 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rmx2\" (UniqueName: \"kubernetes.io/projected/cc6ebdf0-9f08-4cbf-8325-04afa6e45fd5-kube-api-access-4rmx2\") pod \"cert-manager-webhook-6486ffd997-zs7nk\" (UID: \"cc6ebdf0-9f08-4cbf-8325-04afa6e45fd5\") " pod="cert-manager/cert-manager-webhook-6486ffd997-zs7nk" Jul 22 03:02:27.884200 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:27.884173 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cc6ebdf0-9f08-4cbf-8325-04afa6e45fd5-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-zs7nk\" (UID: \"cc6ebdf0-9f08-4cbf-8325-04afa6e45fd5\") " pod="cert-manager/cert-manager-webhook-6486ffd997-zs7nk" Jul 22 03:02:27.884332 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:27.884225 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rmx2\" (UniqueName: \"kubernetes.io/projected/cc6ebdf0-9f08-4cbf-8325-04afa6e45fd5-kube-api-access-4rmx2\") pod \"cert-manager-webhook-6486ffd997-zs7nk\" (UID: \"cc6ebdf0-9f08-4cbf-8325-04afa6e45fd5\") " pod="cert-manager/cert-manager-webhook-6486ffd997-zs7nk" Jul 22 03:02:27.989847 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:27.989751 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-zs7nk" Jul 22 03:02:28.129356 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:28.129327 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-zs7nk"] Jul 22 03:02:28.131730 ip-10-0-131-67 kubenswrapper[2582]: W0722 03:02:28.131706 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcc6ebdf0_9f08_4cbf_8325_04afa6e45fd5.slice/crio-1a9d189123bba32a3f78a54d0b5aa4df5cf8a59a64e8651ee9f69099ff2d0aae WatchSource:0}: Error finding container 1a9d189123bba32a3f78a54d0b5aa4df5cf8a59a64e8651ee9f69099ff2d0aae: Status 404 returned error can't find the container with id 1a9d189123bba32a3f78a54d0b5aa4df5cf8a59a64e8651ee9f69099ff2d0aae Jul 22 03:02:28.777619 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:28.777580 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:02:28.957236 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:28.957202 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:02:29.127612 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:29.127560 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-zs7nk" event={"ID":"cc6ebdf0-9f08-4cbf-8325-04afa6e45fd5","Type":"ContainerStarted","Data":"1a9d189123bba32a3f78a54d0b5aa4df5cf8a59a64e8651ee9f69099ff2d0aae"} Jul 22 03:02:30.135460 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:30.135392 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsg958" event={"ID":"b5d1f610-59af-40b8-ad1f-41916bfa321b","Type":"ContainerStarted","Data":"c924c84e88ecc19c3fdb1a21536fc3e96a7d2ba09095a9f4ad46929aba81aab3"} Jul 22 03:02:30.661824 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:30.661771 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:02:30.815872 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:30.815835 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:02:31.141586 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:31.141506 2582 generic.go:356] "Generic (PLEG): container finished" podID="b5d1f610-59af-40b8-ad1f-41916bfa321b" containerID="c924c84e88ecc19c3fdb1a21536fc3e96a7d2ba09095a9f4ad46929aba81aab3" exitCode=0 Jul 22 03:02:31.141586 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:31.141577 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsg958" event={"ID":"b5d1f610-59af-40b8-ad1f-41916bfa321b","Type":"ContainerDied","Data":"c924c84e88ecc19c3fdb1a21536fc3e96a7d2ba09095a9f4ad46929aba81aab3"} Jul 22 03:02:31.143060 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:31.143039 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-zs7nk" event={"ID":"cc6ebdf0-9f08-4cbf-8325-04afa6e45fd5","Type":"ContainerStarted","Data":"9f2b3c5f20bc4e58b34a9292bd4e989b2bf6ffd8106bcd5390d1d24f26023fd3"} Jul 22 03:02:31.143170 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:31.143151 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-zs7nk" Jul 22 03:02:31.213632 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:31.213584 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-zs7nk" podStartSLOduration=1.424129746 podStartE2EDuration="4.213566227s" totalImagesPullingTime="2.789436481s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:02:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:02:31.209540955 +0000 UTC m=+261.941161800" watchObservedRunningTime="2026-07-22 03:02:31.213566227 +0000 UTC m=+261.945187072" Jul 22 03:02:32.149298 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:32.149264 2582 generic.go:356] "Generic (PLEG): container finished" podID="b5d1f610-59af-40b8-ad1f-41916bfa321b" containerID="a95ed832c8eaf64ba521abbfeee400c535248bc49494e807737bca0acc9c99d1" exitCode=0 Jul 22 03:02:32.149676 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:32.149341 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsg958" event={"ID":"b5d1f610-59af-40b8-ad1f-41916bfa321b","Type":"ContainerDied","Data":"a95ed832c8eaf64ba521abbfeee400c535248bc49494e807737bca0acc9c99d1"} Jul 22 03:02:33.107281 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:33.107248 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-7tgtl"] Jul 22 03:02:33.109423 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:33.109407 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-7tgtl" Jul 22 03:02:33.122480 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:33.122457 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-zphjw\"" Jul 22 03:02:33.131022 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:33.130977 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-7tgtl"] Jul 22 03:02:33.216459 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:33.216423 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fde5dd81-43cc-4eb3-9fa6-eb252e37d32a-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-7tgtl\" (UID: \"fde5dd81-43cc-4eb3-9fa6-eb252e37d32a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-7tgtl" Jul 22 03:02:33.216844 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:33.216547 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz9zp\" (UniqueName: \"kubernetes.io/projected/fde5dd81-43cc-4eb3-9fa6-eb252e37d32a-kube-api-access-lz9zp\") pod \"cert-manager-cainjector-559cdd8c87-7tgtl\" (UID: \"fde5dd81-43cc-4eb3-9fa6-eb252e37d32a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-7tgtl" Jul 22 03:02:33.275696 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:33.275673 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsg958" Jul 22 03:02:33.317663 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:33.317633 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-76rf6\" (UniqueName: \"kubernetes.io/projected/b5d1f610-59af-40b8-ad1f-41916bfa321b-kube-api-access-76rf6\") pod \"b5d1f610-59af-40b8-ad1f-41916bfa321b\" (UID: \"b5d1f610-59af-40b8-ad1f-41916bfa321b\") " Jul 22 03:02:33.317663 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:33.317666 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b5d1f610-59af-40b8-ad1f-41916bfa321b-bundle\") pod \"b5d1f610-59af-40b8-ad1f-41916bfa321b\" (UID: \"b5d1f610-59af-40b8-ad1f-41916bfa321b\") " Jul 22 03:02:33.317901 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:33.317701 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b5d1f610-59af-40b8-ad1f-41916bfa321b-util\") pod \"b5d1f610-59af-40b8-ad1f-41916bfa321b\" (UID: \"b5d1f610-59af-40b8-ad1f-41916bfa321b\") " Jul 22 03:02:33.317962 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:33.317909 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fde5dd81-43cc-4eb3-9fa6-eb252e37d32a-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-7tgtl\" (UID: \"fde5dd81-43cc-4eb3-9fa6-eb252e37d32a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-7tgtl" Jul 22 03:02:33.318047 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:33.317994 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz9zp\" (UniqueName: \"kubernetes.io/projected/fde5dd81-43cc-4eb3-9fa6-eb252e37d32a-kube-api-access-lz9zp\") pod \"cert-manager-cainjector-559cdd8c87-7tgtl\" (UID: \"fde5dd81-43cc-4eb3-9fa6-eb252e37d32a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-7tgtl" Jul 22 03:02:33.318133 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:33.318109 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5d1f610-59af-40b8-ad1f-41916bfa321b-bundle" (OuterVolumeSpecName: "bundle") pod "b5d1f610-59af-40b8-ad1f-41916bfa321b" (UID: "b5d1f610-59af-40b8-ad1f-41916bfa321b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:02:33.319831 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:33.319803 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5d1f610-59af-40b8-ad1f-41916bfa321b-kube-api-access-76rf6" (OuterVolumeSpecName: "kube-api-access-76rf6") pod "b5d1f610-59af-40b8-ad1f-41916bfa321b" (UID: "b5d1f610-59af-40b8-ad1f-41916bfa321b"). InnerVolumeSpecName "kube-api-access-76rf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 03:02:33.324977 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:33.324950 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5d1f610-59af-40b8-ad1f-41916bfa321b-util" (OuterVolumeSpecName: "util") pod "b5d1f610-59af-40b8-ad1f-41916bfa321b" (UID: "b5d1f610-59af-40b8-ad1f-41916bfa321b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:02:33.334566 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:33.334538 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz9zp\" (UniqueName: \"kubernetes.io/projected/fde5dd81-43cc-4eb3-9fa6-eb252e37d32a-kube-api-access-lz9zp\") pod \"cert-manager-cainjector-559cdd8c87-7tgtl\" (UID: \"fde5dd81-43cc-4eb3-9fa6-eb252e37d32a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-7tgtl" Jul 22 03:02:33.334900 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:33.334880 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fde5dd81-43cc-4eb3-9fa6-eb252e37d32a-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-7tgtl\" (UID: \"fde5dd81-43cc-4eb3-9fa6-eb252e37d32a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-7tgtl" Jul 22 03:02:33.418342 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:33.418313 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-7tgtl" Jul 22 03:02:33.418584 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:33.418559 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76rf6\" (UniqueName: \"kubernetes.io/projected/b5d1f610-59af-40b8-ad1f-41916bfa321b-kube-api-access-76rf6\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 03:02:33.418650 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:33.418591 2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b5d1f610-59af-40b8-ad1f-41916bfa321b-bundle\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 03:02:33.418650 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:33.418605 2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b5d1f610-59af-40b8-ad1f-41916bfa321b-util\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 03:02:33.552040 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:33.552004 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-7tgtl"] Jul 22 03:02:33.553958 ip-10-0-131-67 kubenswrapper[2582]: W0722 03:02:33.553931 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfde5dd81_43cc_4eb3_9fa6_eb252e37d32a.slice/crio-7082374b83a660e5dc018667ba7425c3b86a779285faeb89ec97be774cd8984e WatchSource:0}: Error finding container 7082374b83a660e5dc018667ba7425c3b86a779285faeb89ec97be774cd8984e: Status 404 returned error can't find the container with id 7082374b83a660e5dc018667ba7425c3b86a779285faeb89ec97be774cd8984e Jul 22 03:02:34.161080 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:34.160997 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsg958" Jul 22 03:02:34.161245 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:34.160994 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsg958" event={"ID":"b5d1f610-59af-40b8-ad1f-41916bfa321b","Type":"ContainerDied","Data":"4d59b2862b99acc096e0a7a35e8f1684566126599c86d7ce3ccd8b69aa502f39"} Jul 22 03:02:34.161245 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:34.161134 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d59b2862b99acc096e0a7a35e8f1684566126599c86d7ce3ccd8b69aa502f39" Jul 22 03:02:34.162611 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:34.162590 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-7tgtl" event={"ID":"fde5dd81-43cc-4eb3-9fa6-eb252e37d32a","Type":"ContainerStarted","Data":"327dd95fff6bccbe3736ddf8dd1e8f83a67b6cec0952faf16cbbcc575a4d88ad"} Jul 22 03:02:34.162611 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:34.162618 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-7tgtl" event={"ID":"fde5dd81-43cc-4eb3-9fa6-eb252e37d32a","Type":"ContainerStarted","Data":"7082374b83a660e5dc018667ba7425c3b86a779285faeb89ec97be774cd8984e"} Jul 22 03:02:34.181444 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:34.181394 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-7tgtl" podStartSLOduration=1.181378478 podStartE2EDuration="1.181378478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:02:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:02:34.179751251 +0000 UTC m=+264.911372108" watchObservedRunningTime="2026-07-22 03:02:34.181378478 +0000 UTC m=+264.912999324" Jul 22 03:02:34.338635 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:34.338601 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:02:34.434758 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:34.434676 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:02:36.369277 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:36.369241 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:02:36.957732 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:36.957701 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:02:37.049817 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:37.049783 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:02:37.152296 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:37.152268 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-zs7nk" Jul 22 03:02:43.486544 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:43.486507 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:02:43.588249 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:43.588216 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:02:45.254818 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:45.254787 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgv4hp"] Jul 22 03:02:45.255221 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:45.255100 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5d1f610-59af-40b8-ad1f-41916bfa321b" containerName="util" Jul 22 03:02:45.255221 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:45.255111 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5d1f610-59af-40b8-ad1f-41916bfa321b" containerName="util" Jul 22 03:02:45.255221 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:45.255122 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5d1f610-59af-40b8-ad1f-41916bfa321b" containerName="extract" Jul 22 03:02:45.255221 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:45.255128 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5d1f610-59af-40b8-ad1f-41916bfa321b" containerName="extract" Jul 22 03:02:45.255221 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:45.255139 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5d1f610-59af-40b8-ad1f-41916bfa321b" containerName="pull" Jul 22 03:02:45.255221 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:45.255145 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5d1f610-59af-40b8-ad1f-41916bfa321b" containerName="pull" Jul 22 03:02:45.255221 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:45.255195 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5d1f610-59af-40b8-ad1f-41916bfa321b" containerName="extract" Jul 22 03:02:45.292358 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:45.292320 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgv4hp"] Jul 22 03:02:45.292518 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:45.292427 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgv4hp" Jul 22 03:02:45.295146 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:45.295121 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-44dcl\"" Jul 22 03:02:45.295270 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:45.295185 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 22 03:02:45.296297 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:45.296271 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 22 03:02:45.424619 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:45.424590 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcdw5\" (UniqueName: \"kubernetes.io/projected/b6019ec7-a567-4d87-9904-de4b20ae6dea-kube-api-access-qcdw5\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgv4hp\" (UID: \"b6019ec7-a567-4d87-9904-de4b20ae6dea\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgv4hp" Jul 22 03:02:45.424794 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:45.424628 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b6019ec7-a567-4d87-9904-de4b20ae6dea-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgv4hp\" (UID: \"b6019ec7-a567-4d87-9904-de4b20ae6dea\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgv4hp" Jul 22 03:02:45.424794 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:45.424665 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b6019ec7-a567-4d87-9904-de4b20ae6dea-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgv4hp\" (UID: \"b6019ec7-a567-4d87-9904-de4b20ae6dea\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgv4hp" Jul 22 03:02:45.525509 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:45.525420 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b6019ec7-a567-4d87-9904-de4b20ae6dea-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgv4hp\" (UID: \"b6019ec7-a567-4d87-9904-de4b20ae6dea\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgv4hp" Jul 22 03:02:45.525509 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:45.525472 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b6019ec7-a567-4d87-9904-de4b20ae6dea-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgv4hp\" (UID: \"b6019ec7-a567-4d87-9904-de4b20ae6dea\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgv4hp" Jul 22 03:02:45.525735 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:45.525564 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcdw5\" (UniqueName: \"kubernetes.io/projected/b6019ec7-a567-4d87-9904-de4b20ae6dea-kube-api-access-qcdw5\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgv4hp\" (UID: \"b6019ec7-a567-4d87-9904-de4b20ae6dea\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgv4hp" Jul 22 03:02:45.525813 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:45.525794 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b6019ec7-a567-4d87-9904-de4b20ae6dea-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgv4hp\" (UID: \"b6019ec7-a567-4d87-9904-de4b20ae6dea\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgv4hp" Jul 22 03:02:45.525870 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:45.525829 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b6019ec7-a567-4d87-9904-de4b20ae6dea-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgv4hp\" (UID: \"b6019ec7-a567-4d87-9904-de4b20ae6dea\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgv4hp" Jul 22 03:02:45.536912 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:45.536884 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcdw5\" (UniqueName: \"kubernetes.io/projected/b6019ec7-a567-4d87-9904-de4b20ae6dea-kube-api-access-qcdw5\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgv4hp\" (UID: \"b6019ec7-a567-4d87-9904-de4b20ae6dea\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgv4hp" Jul 22 03:02:45.580942 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:45.580910 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:02:45.601605 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:45.601576 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgv4hp" Jul 22 03:02:45.749474 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:45.749451 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgv4hp"] Jul 22 03:02:45.752300 ip-10-0-131-67 kubenswrapper[2582]: W0722 03:02:45.752274 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6019ec7_a567_4d87_9904_de4b20ae6dea.slice/crio-5aa6a1a2b21dc3907e8ad5d2fbab5c61281e770fd16628ba46e3e660c33b2bcd WatchSource:0}: Error finding container 5aa6a1a2b21dc3907e8ad5d2fbab5c61281e770fd16628ba46e3e660c33b2bcd: Status 404 returned error can't find the container with id 5aa6a1a2b21dc3907e8ad5d2fbab5c61281e770fd16628ba46e3e660c33b2bcd Jul 22 03:02:46.216758 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:46.216718 2582 generic.go:356] "Generic (PLEG): container finished" podID="b6019ec7-a567-4d87-9904-de4b20ae6dea" containerID="9257bf4ea761ffe95d45bb5147b6cccea3db81ae1dfe14ae0620753a11e7786a" exitCode=0 Jul 22 03:02:46.216949 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:46.216790 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgv4hp" event={"ID":"b6019ec7-a567-4d87-9904-de4b20ae6dea","Type":"ContainerDied","Data":"9257bf4ea761ffe95d45bb5147b6cccea3db81ae1dfe14ae0620753a11e7786a"} Jul 22 03:02:46.216949 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:46.216847 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgv4hp" event={"ID":"b6019ec7-a567-4d87-9904-de4b20ae6dea","Type":"ContainerStarted","Data":"5aa6a1a2b21dc3907e8ad5d2fbab5c61281e770fd16628ba46e3e660c33b2bcd"} Jul 22 03:02:47.222906 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:47.222871 2582 generic.go:356] "Generic (PLEG): container finished" podID="b6019ec7-a567-4d87-9904-de4b20ae6dea" containerID="171509179b281378653feac0365ba2fe80542a28e23faf4ac55afc6dd208ac49" exitCode=0 Jul 22 03:02:47.222906 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:47.222908 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgv4hp" event={"ID":"b6019ec7-a567-4d87-9904-de4b20ae6dea","Type":"ContainerDied","Data":"171509179b281378653feac0365ba2fe80542a28e23faf4ac55afc6dd208ac49"} Jul 22 03:02:48.229841 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:48.229808 2582 generic.go:356] "Generic (PLEG): container finished" podID="b6019ec7-a567-4d87-9904-de4b20ae6dea" containerID="3cec89bf7b203b72da52d2b3c457b80a6c1b3903eb39fe7e01cf59ff54fdfb4c" exitCode=0 Jul 22 03:02:48.230247 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:48.229860 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgv4hp" event={"ID":"b6019ec7-a567-4d87-9904-de4b20ae6dea","Type":"ContainerDied","Data":"3cec89bf7b203b72da52d2b3c457b80a6c1b3903eb39fe7e01cf59ff54fdfb4c"} Jul 22 03:02:49.360087 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:49.360066 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgv4hp" Jul 22 03:02:49.453175 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:49.453144 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b6019ec7-a567-4d87-9904-de4b20ae6dea-bundle\") pod \"b6019ec7-a567-4d87-9904-de4b20ae6dea\" (UID: \"b6019ec7-a567-4d87-9904-de4b20ae6dea\") " Jul 22 03:02:49.453330 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:49.453202 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b6019ec7-a567-4d87-9904-de4b20ae6dea-util\") pod \"b6019ec7-a567-4d87-9904-de4b20ae6dea\" (UID: \"b6019ec7-a567-4d87-9904-de4b20ae6dea\") " Jul 22 03:02:49.453330 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:49.453227 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qcdw5\" (UniqueName: \"kubernetes.io/projected/b6019ec7-a567-4d87-9904-de4b20ae6dea-kube-api-access-qcdw5\") pod \"b6019ec7-a567-4d87-9904-de4b20ae6dea\" (UID: \"b6019ec7-a567-4d87-9904-de4b20ae6dea\") " Jul 22 03:02:49.453879 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:49.453848 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6019ec7-a567-4d87-9904-de4b20ae6dea-bundle" (OuterVolumeSpecName: "bundle") pod "b6019ec7-a567-4d87-9904-de4b20ae6dea" (UID: "b6019ec7-a567-4d87-9904-de4b20ae6dea"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:02:49.455176 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:49.455154 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6019ec7-a567-4d87-9904-de4b20ae6dea-kube-api-access-qcdw5" (OuterVolumeSpecName: "kube-api-access-qcdw5") pod "b6019ec7-a567-4d87-9904-de4b20ae6dea" (UID: "b6019ec7-a567-4d87-9904-de4b20ae6dea"). InnerVolumeSpecName "kube-api-access-qcdw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 03:02:49.458862 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:49.458830 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6019ec7-a567-4d87-9904-de4b20ae6dea-util" (OuterVolumeSpecName: "util") pod "b6019ec7-a567-4d87-9904-de4b20ae6dea" (UID: "b6019ec7-a567-4d87-9904-de4b20ae6dea"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:02:49.554591 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:49.554482 2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b6019ec7-a567-4d87-9904-de4b20ae6dea-util\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 03:02:49.554591 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:49.554527 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcdw5\" (UniqueName: \"kubernetes.io/projected/b6019ec7-a567-4d87-9904-de4b20ae6dea-kube-api-access-qcdw5\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 03:02:49.554591 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:49.554541 2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b6019ec7-a567-4d87-9904-de4b20ae6dea-bundle\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 03:02:50.240298 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:50.240222 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgv4hp" Jul 22 03:02:50.240440 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:50.240220 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgv4hp" event={"ID":"b6019ec7-a567-4d87-9904-de4b20ae6dea","Type":"ContainerDied","Data":"5aa6a1a2b21dc3907e8ad5d2fbab5c61281e770fd16628ba46e3e660c33b2bcd"} Jul 22 03:02:50.240440 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:02:50.240337 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5aa6a1a2b21dc3907e8ad5d2fbab5c61281e770fd16628ba46e3e660c33b2bcd" Jul 22 03:03:01.570888 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:01.570840 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:03:01.664037 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:01.663962 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:03:02.141595 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:02.141548 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:03:02.463913 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:02.463828 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:03:02.580913 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:02.580871 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:03:02.665452 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:02.665420 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-94fc48448-gsdv4"] Jul 22 03:03:02.665719 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:02.665709 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6019ec7-a567-4d87-9904-de4b20ae6dea" containerName="pull" Jul 22 03:03:02.665767 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:02.665721 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6019ec7-a567-4d87-9904-de4b20ae6dea" containerName="pull" Jul 22 03:03:02.665767 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:02.665731 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6019ec7-a567-4d87-9904-de4b20ae6dea" containerName="extract" Jul 22 03:03:02.665767 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:02.665736 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6019ec7-a567-4d87-9904-de4b20ae6dea" containerName="extract" Jul 22 03:03:02.665767 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:02.665751 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6019ec7-a567-4d87-9904-de4b20ae6dea" containerName="util" Jul 22 03:03:02.665767 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:02.665756 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6019ec7-a567-4d87-9904-de4b20ae6dea" containerName="util" Jul 22 03:03:02.665908 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:02.665806 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6019ec7-a567-4d87-9904-de4b20ae6dea" containerName="extract" Jul 22 03:03:02.667750 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:02.667734 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-94fc48448-gsdv4" Jul 22 03:03:02.674917 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:02.674901 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 22 03:03:02.675035 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:02.674907 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 22 03:03:02.675313 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:02.675292 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 22 03:03:02.675539 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:02.675521 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 22 03:03:02.675653 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:02.675620 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-f4xjc\"" Jul 22 03:03:02.694353 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:02.694320 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-94fc48448-gsdv4"] Jul 22 03:03:02.761271 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:02.761187 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/79701dcf-7b01-484f-8cc0-d6ce840733d1-tmp\") pod \"opendatahub-operator-controller-manager-94fc48448-gsdv4\" (UID: \"79701dcf-7b01-484f-8cc0-d6ce840733d1\") " pod="opendatahub/opendatahub-operator-controller-manager-94fc48448-gsdv4" Jul 22 03:03:02.761271 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:02.761225 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22wk9\" (UniqueName: \"kubernetes.io/projected/79701dcf-7b01-484f-8cc0-d6ce840733d1-kube-api-access-22wk9\") pod \"opendatahub-operator-controller-manager-94fc48448-gsdv4\" (UID: \"79701dcf-7b01-484f-8cc0-d6ce840733d1\") " pod="opendatahub/opendatahub-operator-controller-manager-94fc48448-gsdv4" Jul 22 03:03:02.761271 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:02.761262 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/79701dcf-7b01-484f-8cc0-d6ce840733d1-manifests\") pod \"opendatahub-operator-controller-manager-94fc48448-gsdv4\" (UID: \"79701dcf-7b01-484f-8cc0-d6ce840733d1\") " pod="opendatahub/opendatahub-operator-controller-manager-94fc48448-gsdv4" Jul 22 03:03:02.761504 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:02.761329 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/79701dcf-7b01-484f-8cc0-d6ce840733d1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-94fc48448-gsdv4\" (UID: \"79701dcf-7b01-484f-8cc0-d6ce840733d1\") " pod="opendatahub/opendatahub-operator-controller-manager-94fc48448-gsdv4" Jul 22 03:03:02.761504 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:02.761369 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/79701dcf-7b01-484f-8cc0-d6ce840733d1-webhook-cert\") pod \"opendatahub-operator-controller-manager-94fc48448-gsdv4\" (UID: \"79701dcf-7b01-484f-8cc0-d6ce840733d1\") " pod="opendatahub/opendatahub-operator-controller-manager-94fc48448-gsdv4" Jul 22 03:03:02.862246 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:02.862195 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22wk9\" (UniqueName: \"kubernetes.io/projected/79701dcf-7b01-484f-8cc0-d6ce840733d1-kube-api-access-22wk9\") pod \"opendatahub-operator-controller-manager-94fc48448-gsdv4\" (UID: \"79701dcf-7b01-484f-8cc0-d6ce840733d1\") " pod="opendatahub/opendatahub-operator-controller-manager-94fc48448-gsdv4" Jul 22 03:03:02.862436 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:02.862272 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/79701dcf-7b01-484f-8cc0-d6ce840733d1-manifests\") pod \"opendatahub-operator-controller-manager-94fc48448-gsdv4\" (UID: \"79701dcf-7b01-484f-8cc0-d6ce840733d1\") " pod="opendatahub/opendatahub-operator-controller-manager-94fc48448-gsdv4" Jul 22 03:03:02.862436 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:02.862321 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/79701dcf-7b01-484f-8cc0-d6ce840733d1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-94fc48448-gsdv4\" (UID: \"79701dcf-7b01-484f-8cc0-d6ce840733d1\") " pod="opendatahub/opendatahub-operator-controller-manager-94fc48448-gsdv4" Jul 22 03:03:02.862436 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:02.862352 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/79701dcf-7b01-484f-8cc0-d6ce840733d1-webhook-cert\") pod \"opendatahub-operator-controller-manager-94fc48448-gsdv4\" (UID: \"79701dcf-7b01-484f-8cc0-d6ce840733d1\") " pod="opendatahub/opendatahub-operator-controller-manager-94fc48448-gsdv4" Jul 22 03:03:02.862436 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:02.862406 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/79701dcf-7b01-484f-8cc0-d6ce840733d1-tmp\") pod \"opendatahub-operator-controller-manager-94fc48448-gsdv4\" (UID: \"79701dcf-7b01-484f-8cc0-d6ce840733d1\") " pod="opendatahub/opendatahub-operator-controller-manager-94fc48448-gsdv4" Jul 22 03:03:02.862671 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:02.862649 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/79701dcf-7b01-484f-8cc0-d6ce840733d1-manifests\") pod \"opendatahub-operator-controller-manager-94fc48448-gsdv4\" (UID: \"79701dcf-7b01-484f-8cc0-d6ce840733d1\") " pod="opendatahub/opendatahub-operator-controller-manager-94fc48448-gsdv4" Jul 22 03:03:02.864568 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:02.864545 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/79701dcf-7b01-484f-8cc0-d6ce840733d1-tmp\") pod \"opendatahub-operator-controller-manager-94fc48448-gsdv4\" (UID: \"79701dcf-7b01-484f-8cc0-d6ce840733d1\") " pod="opendatahub/opendatahub-operator-controller-manager-94fc48448-gsdv4" Jul 22 03:03:02.864828 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:02.864809 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/79701dcf-7b01-484f-8cc0-d6ce840733d1-webhook-cert\") pod \"opendatahub-operator-controller-manager-94fc48448-gsdv4\" (UID: \"79701dcf-7b01-484f-8cc0-d6ce840733d1\") " pod="opendatahub/opendatahub-operator-controller-manager-94fc48448-gsdv4" Jul 22 03:03:02.864907 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:02.864868 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/79701dcf-7b01-484f-8cc0-d6ce840733d1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-94fc48448-gsdv4\" (UID: \"79701dcf-7b01-484f-8cc0-d6ce840733d1\") " pod="opendatahub/opendatahub-operator-controller-manager-94fc48448-gsdv4" Jul 22 03:03:02.884609 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:02.884575 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:03:02.898712 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:02.898681 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22wk9\" (UniqueName: \"kubernetes.io/projected/79701dcf-7b01-484f-8cc0-d6ce840733d1-kube-api-access-22wk9\") pod \"opendatahub-operator-controller-manager-94fc48448-gsdv4\" (UID: \"79701dcf-7b01-484f-8cc0-d6ce840733d1\") " pod="opendatahub/opendatahub-operator-controller-manager-94fc48448-gsdv4" Jul 22 03:03:02.978164 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:02.978109 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-94fc48448-gsdv4" Jul 22 03:03:02.984923 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:02.984893 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:03:03.091945 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:03.091917 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:03:03.127327 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:03.127301 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-94fc48448-gsdv4"] Jul 22 03:03:03.129415 ip-10-0-131-67 kubenswrapper[2582]: W0722 03:03:03.129392 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79701dcf_7b01_484f_8cc0_d6ce840733d1.slice/crio-b6b61a7b1c03ee8cdcce447f80d75f3085f606c476e14bb47dcacefe972f8fda WatchSource:0}: Error finding container b6b61a7b1c03ee8cdcce447f80d75f3085f606c476e14bb47dcacefe972f8fda: Status 404 returned error can't find the container with id b6b61a7b1c03ee8cdcce447f80d75f3085f606c476e14bb47dcacefe972f8fda Jul 22 03:03:03.217238 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:03.217205 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:03:03.298715 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:03.298638 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-94fc48448-gsdv4" event={"ID":"79701dcf-7b01-484f-8cc0-d6ce840733d1","Type":"ContainerStarted","Data":"b6b61a7b1c03ee8cdcce447f80d75f3085f606c476e14bb47dcacefe972f8fda"} Jul 22 03:03:03.354123 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:03.354085 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:03:03.459500 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:03.459472 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:03:06.314087 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:06.314050 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-94fc48448-gsdv4" event={"ID":"79701dcf-7b01-484f-8cc0-d6ce840733d1","Type":"ContainerStarted","Data":"bbc05a39089c03e1e4f0e15119a1e84782cc19a490383ab1cbabc04d8c06f420"} Jul 22 03:03:08.324186 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:08.324152 2582 generic.go:356] "Generic (PLEG): container finished" podID="79701dcf-7b01-484f-8cc0-d6ce840733d1" containerID="bbc05a39089c03e1e4f0e15119a1e84782cc19a490383ab1cbabc04d8c06f420" exitCode=0 Jul 22 03:03:08.324561 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:08.324197 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-94fc48448-gsdv4" event={"ID":"79701dcf-7b01-484f-8cc0-d6ce840733d1","Type":"ContainerDied","Data":"bbc05a39089c03e1e4f0e15119a1e84782cc19a490383ab1cbabc04d8c06f420"} Jul 22 03:03:09.330787 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:09.330753 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-94fc48448-gsdv4" event={"ID":"79701dcf-7b01-484f-8cc0-d6ce840733d1","Type":"ContainerStarted","Data":"b046de4de988c480fffd3ca7e803c228f9540a41a43ed57e3986916c20e5f40d"} Jul 22 03:03:09.331175 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:09.330961 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-94fc48448-gsdv4" Jul 22 03:03:09.362587 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:09.362536 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-94fc48448-gsdv4" podStartSLOduration=2.858904692 podStartE2EDuration="7.362519599s" totalImagesPullingTime="3.456191597s" totalInitContainerRuntime="1.04742331s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:03:02 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:03:09.357103823 +0000 UTC m=+300.088724667" watchObservedRunningTime="2026-07-22 03:03:09.362519599 +0000 UTC m=+300.094140444" Jul 22 03:03:09.717142 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:09.717116 2582 kubelet.go:1648] "Image garbage collection succeeded" Jul 22 03:03:15.357401 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:15.357368 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:03:16.870499 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:16.870464 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:03:20.337615 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:20.337580 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-94fc48448-gsdv4" Jul 22 03:03:22.042878 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:22.042843 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:03:22.156976 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:22.156938 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:03:22.440838 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:22.440797 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:03:22.540885 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:22.540859 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:03:22.584578 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:22.584546 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-684578bfd-2jd6x"] Jul 22 03:03:22.600250 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:22.600222 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-684578bfd-2jd6x"] Jul 22 03:03:22.600439 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:22.600310 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-684578bfd-2jd6x" Jul 22 03:03:22.603235 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:22.603209 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 22 03:03:22.603235 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:22.603232 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 22 03:03:22.603456 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:22.603209 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-w5zj4\"" Jul 22 03:03:22.603971 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:22.603950 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 22 03:03:22.603971 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:22.603966 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 22 03:03:22.620410 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:22.620388 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d10d14d5-4415-431c-81c3-623c8050e7c8-tmp\") pod \"kube-auth-proxy-684578bfd-2jd6x\" (UID: \"d10d14d5-4415-431c-81c3-623c8050e7c8\") " pod="openshift-ingress/kube-auth-proxy-684578bfd-2jd6x" Jul 22 03:03:22.620543 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:22.620416 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g65rc\" (UniqueName: \"kubernetes.io/projected/d10d14d5-4415-431c-81c3-623c8050e7c8-kube-api-access-g65rc\") pod \"kube-auth-proxy-684578bfd-2jd6x\" (UID: \"d10d14d5-4415-431c-81c3-623c8050e7c8\") " pod="openshift-ingress/kube-auth-proxy-684578bfd-2jd6x" Jul 22 03:03:22.620543 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:22.620518 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d10d14d5-4415-431c-81c3-623c8050e7c8-tls-certs\") pod \"kube-auth-proxy-684578bfd-2jd6x\" (UID: \"d10d14d5-4415-431c-81c3-623c8050e7c8\") " pod="openshift-ingress/kube-auth-proxy-684578bfd-2jd6x" Jul 22 03:03:22.720963 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:22.720879 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d10d14d5-4415-431c-81c3-623c8050e7c8-tls-certs\") pod \"kube-auth-proxy-684578bfd-2jd6x\" (UID: \"d10d14d5-4415-431c-81c3-623c8050e7c8\") " pod="openshift-ingress/kube-auth-proxy-684578bfd-2jd6x" Jul 22 03:03:22.720963 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:22.720922 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d10d14d5-4415-431c-81c3-623c8050e7c8-tmp\") pod \"kube-auth-proxy-684578bfd-2jd6x\" (UID: \"d10d14d5-4415-431c-81c3-623c8050e7c8\") " pod="openshift-ingress/kube-auth-proxy-684578bfd-2jd6x" Jul 22 03:03:22.720963 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:22.720939 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g65rc\" (UniqueName: \"kubernetes.io/projected/d10d14d5-4415-431c-81c3-623c8050e7c8-kube-api-access-g65rc\") pod \"kube-auth-proxy-684578bfd-2jd6x\" (UID: \"d10d14d5-4415-431c-81c3-623c8050e7c8\") " pod="openshift-ingress/kube-auth-proxy-684578bfd-2jd6x" Jul 22 03:03:22.723310 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:22.723274 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d10d14d5-4415-431c-81c3-623c8050e7c8-tmp\") pod \"kube-auth-proxy-684578bfd-2jd6x\" (UID: \"d10d14d5-4415-431c-81c3-623c8050e7c8\") " pod="openshift-ingress/kube-auth-proxy-684578bfd-2jd6x" Jul 22 03:03:22.723490 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:22.723470 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d10d14d5-4415-431c-81c3-623c8050e7c8-tls-certs\") pod \"kube-auth-proxy-684578bfd-2jd6x\" (UID: \"d10d14d5-4415-431c-81c3-623c8050e7c8\") " pod="openshift-ingress/kube-auth-proxy-684578bfd-2jd6x" Jul 22 03:03:22.733711 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:22.733684 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g65rc\" (UniqueName: \"kubernetes.io/projected/d10d14d5-4415-431c-81c3-623c8050e7c8-kube-api-access-g65rc\") pod \"kube-auth-proxy-684578bfd-2jd6x\" (UID: \"d10d14d5-4415-431c-81c3-623c8050e7c8\") " pod="openshift-ingress/kube-auth-proxy-684578bfd-2jd6x" Jul 22 03:03:22.794063 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:22.794025 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-nbpjd"] Jul 22 03:03:22.796347 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:22.796330 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-nbpjd" Jul 22 03:03:22.798922 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:22.798906 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 22 03:03:22.799003 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:22.798945 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-wdnc6\"" Jul 22 03:03:22.808231 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:22.808205 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-nbpjd"] Jul 22 03:03:22.821835 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:22.821809 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/39934543-08e4-4456-b200-27ce181b98e8-cert\") pod \"odh-model-controller-7998c67dc4-nbpjd\" (UID: \"39934543-08e4-4456-b200-27ce181b98e8\") " pod="opendatahub/odh-model-controller-7998c67dc4-nbpjd" Jul 22 03:03:22.821978 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:22.821876 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6k8c\" (UniqueName: \"kubernetes.io/projected/39934543-08e4-4456-b200-27ce181b98e8-kube-api-access-s6k8c\") pod \"odh-model-controller-7998c67dc4-nbpjd\" (UID: \"39934543-08e4-4456-b200-27ce181b98e8\") " pod="opendatahub/odh-model-controller-7998c67dc4-nbpjd" Jul 22 03:03:22.834798 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:22.834771 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:03:22.910496 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:22.910459 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-684578bfd-2jd6x" Jul 22 03:03:22.922829 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:22.922798 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/39934543-08e4-4456-b200-27ce181b98e8-cert\") pod \"odh-model-controller-7998c67dc4-nbpjd\" (UID: \"39934543-08e4-4456-b200-27ce181b98e8\") " pod="opendatahub/odh-model-controller-7998c67dc4-nbpjd" Jul 22 03:03:22.922966 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:22.922897 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6k8c\" (UniqueName: \"kubernetes.io/projected/39934543-08e4-4456-b200-27ce181b98e8-kube-api-access-s6k8c\") pod \"odh-model-controller-7998c67dc4-nbpjd\" (UID: \"39934543-08e4-4456-b200-27ce181b98e8\") " pod="opendatahub/odh-model-controller-7998c67dc4-nbpjd" Jul 22 03:03:22.923249 ip-10-0-131-67 kubenswrapper[2582]: E0722 03:03:22.923229 2582 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 22 03:03:22.923419 ip-10-0-131-67 kubenswrapper[2582]: E0722 03:03:22.923402 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39934543-08e4-4456-b200-27ce181b98e8-cert podName:39934543-08e4-4456-b200-27ce181b98e8 nodeName:}" failed. No retries permitted until 2026-07-22 03:03:23.423380037 +0000 UTC m=+314.155000865 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/39934543-08e4-4456-b200-27ce181b98e8-cert") pod "odh-model-controller-7998c67dc4-nbpjd" (UID: "39934543-08e4-4456-b200-27ce181b98e8") : secret "odh-model-controller-webhook-cert" not found Jul 22 03:03:22.933459 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:22.933432 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:03:22.937796 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:22.937767 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6k8c\" (UniqueName: \"kubernetes.io/projected/39934543-08e4-4456-b200-27ce181b98e8-kube-api-access-s6k8c\") pod \"odh-model-controller-7998c67dc4-nbpjd\" (UID: \"39934543-08e4-4456-b200-27ce181b98e8\") " pod="opendatahub/odh-model-controller-7998c67dc4-nbpjd" Jul 22 03:03:23.026611 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:23.026578 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:03:23.045896 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:23.045871 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-684578bfd-2jd6x"] Jul 22 03:03:23.048206 ip-10-0-131-67 kubenswrapper[2582]: W0722 03:03:23.048172 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd10d14d5_4415_431c_81c3_623c8050e7c8.slice/crio-a7299fe210bd7a584d2103c43208ab79589859e71d2122e33683b46d87e9d196 WatchSource:0}: Error finding container a7299fe210bd7a584d2103c43208ab79589859e71d2122e33683b46d87e9d196: Status 404 returned error can't find the container with id a7299fe210bd7a584d2103c43208ab79589859e71d2122e33683b46d87e9d196 Jul 22 03:03:23.050431 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:23.050412 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 03:03:23.127130 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:23.127099 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:03:23.394025 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:23.393972 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-684578bfd-2jd6x" event={"ID":"d10d14d5-4415-431c-81c3-623c8050e7c8","Type":"ContainerStarted","Data":"a7299fe210bd7a584d2103c43208ab79589859e71d2122e33683b46d87e9d196"} Jul 22 03:03:23.427142 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:23.427102 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/39934543-08e4-4456-b200-27ce181b98e8-cert\") pod \"odh-model-controller-7998c67dc4-nbpjd\" (UID: \"39934543-08e4-4456-b200-27ce181b98e8\") " pod="opendatahub/odh-model-controller-7998c67dc4-nbpjd" Jul 22 03:03:23.429640 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:23.429614 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/39934543-08e4-4456-b200-27ce181b98e8-cert\") pod \"odh-model-controller-7998c67dc4-nbpjd\" (UID: \"39934543-08e4-4456-b200-27ce181b98e8\") " pod="opendatahub/odh-model-controller-7998c67dc4-nbpjd" Jul 22 03:03:23.524750 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:23.524713 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-pnl96"] Jul 22 03:03:23.527808 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:23.527787 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-pnl96" Jul 22 03:03:23.530921 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:23.530896 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 22 03:03:23.541600 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:23.541505 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-pnl96"] Jul 22 03:03:23.542343 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:23.541899 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:03:23.628440 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:23.628402 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfmvh\" (UniqueName: \"kubernetes.io/projected/20b8fa3f-a5f9-425b-8fd2-e52ee34adf8a-kube-api-access-gfmvh\") pod \"model-serving-api-54d89bd5fc-pnl96\" (UID: \"20b8fa3f-a5f9-425b-8fd2-e52ee34adf8a\") " pod="opendatahub/model-serving-api-54d89bd5fc-pnl96" Jul 22 03:03:23.628632 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:23.628461 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20b8fa3f-a5f9-425b-8fd2-e52ee34adf8a-tls-certs\") pod \"model-serving-api-54d89bd5fc-pnl96\" (UID: \"20b8fa3f-a5f9-425b-8fd2-e52ee34adf8a\") " pod="opendatahub/model-serving-api-54d89bd5fc-pnl96" Jul 22 03:03:23.647620 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:23.647511 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:03:23.707275 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:23.707238 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-nbpjd" Jul 22 03:03:23.729814 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:23.729781 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfmvh\" (UniqueName: \"kubernetes.io/projected/20b8fa3f-a5f9-425b-8fd2-e52ee34adf8a-kube-api-access-gfmvh\") pod \"model-serving-api-54d89bd5fc-pnl96\" (UID: \"20b8fa3f-a5f9-425b-8fd2-e52ee34adf8a\") " pod="opendatahub/model-serving-api-54d89bd5fc-pnl96" Jul 22 03:03:23.730052 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:23.729846 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20b8fa3f-a5f9-425b-8fd2-e52ee34adf8a-tls-certs\") pod \"model-serving-api-54d89bd5fc-pnl96\" (UID: \"20b8fa3f-a5f9-425b-8fd2-e52ee34adf8a\") " pod="opendatahub/model-serving-api-54d89bd5fc-pnl96" Jul 22 03:03:23.733634 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:23.733607 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20b8fa3f-a5f9-425b-8fd2-e52ee34adf8a-tls-certs\") pod \"model-serving-api-54d89bd5fc-pnl96\" (UID: \"20b8fa3f-a5f9-425b-8fd2-e52ee34adf8a\") " pod="opendatahub/model-serving-api-54d89bd5fc-pnl96" Jul 22 03:03:23.738893 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:23.738863 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfmvh\" (UniqueName: \"kubernetes.io/projected/20b8fa3f-a5f9-425b-8fd2-e52ee34adf8a-kube-api-access-gfmvh\") pod \"model-serving-api-54d89bd5fc-pnl96\" (UID: \"20b8fa3f-a5f9-425b-8fd2-e52ee34adf8a\") " pod="opendatahub/model-serving-api-54d89bd5fc-pnl96" Jul 22 03:03:23.838751 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:23.838705 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-pnl96" Jul 22 03:03:23.866556 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:23.866523 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-nbpjd"] Jul 22 03:03:23.868804 ip-10-0-131-67 kubenswrapper[2582]: W0722 03:03:23.868757 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39934543_08e4_4456_b200_27ce181b98e8.slice/crio-df14d81eabb02cf78015d321dc663bb6fc2b01bd32667aa6f838805482c3444e WatchSource:0}: Error finding container df14d81eabb02cf78015d321dc663bb6fc2b01bd32667aa6f838805482c3444e: Status 404 returned error can't find the container with id df14d81eabb02cf78015d321dc663bb6fc2b01bd32667aa6f838805482c3444e Jul 22 03:03:23.919917 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:23.919886 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:03:23.936419 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:23.936392 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-wbctj"] Jul 22 03:03:23.940380 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:23.940355 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-wbctj" Jul 22 03:03:23.943423 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:23.943182 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 22 03:03:23.943423 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:23.943265 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-s9t4w\"" Jul 22 03:03:23.943635 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:23.943598 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 22 03:03:23.943866 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:23.943804 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 22 03:03:23.943994 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:23.943977 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 22 03:03:23.959791 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:23.959733 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-wbctj"] Jul 22 03:03:24.006392 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:24.006359 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-pnl96"] Jul 22 03:03:24.009481 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:24.009452 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:03:24.009671 ip-10-0-131-67 kubenswrapper[2582]: W0722 03:03:24.009653 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20b8fa3f_a5f9_425b_8fd2_e52ee34adf8a.slice/crio-f740f33e5390b455b2332a24474b8489c3d19fb5fcd78ed244c4c202ace1f43a WatchSource:0}: Error finding container f740f33e5390b455b2332a24474b8489c3d19fb5fcd78ed244c4c202ace1f43a: Status 404 returned error can't find the container with id f740f33e5390b455b2332a24474b8489c3d19fb5fcd78ed244c4c202ace1f43a Jul 22 03:03:24.032431 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:24.032398 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8fb1f87-8ad1-4476-8117-c7b5d129a5d0-local-certs\") pod \"istiod-openshift-gateway-7574459656-wbctj\" (UID: \"a8fb1f87-8ad1-4476-8117-c7b5d129a5d0\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-wbctj" Jul 22 03:03:24.032628 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:24.032440 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a8fb1f87-8ad1-4476-8117-c7b5d129a5d0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-wbctj\" (UID: \"a8fb1f87-8ad1-4476-8117-c7b5d129a5d0\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-wbctj" Jul 22 03:03:24.032628 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:24.032469 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a8fb1f87-8ad1-4476-8117-c7b5d129a5d0-cacerts\") pod \"istiod-openshift-gateway-7574459656-wbctj\" (UID: \"a8fb1f87-8ad1-4476-8117-c7b5d129a5d0\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-wbctj" Jul 22 03:03:24.032628 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:24.032495 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a8fb1f87-8ad1-4476-8117-c7b5d129a5d0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-wbctj\" (UID: \"a8fb1f87-8ad1-4476-8117-c7b5d129a5d0\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-wbctj" Jul 22 03:03:24.032628 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:24.032554 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8fb1f87-8ad1-4476-8117-c7b5d129a5d0-istio-token\") pod \"istiod-openshift-gateway-7574459656-wbctj\" (UID: \"a8fb1f87-8ad1-4476-8117-c7b5d129a5d0\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-wbctj" Jul 22 03:03:24.032628 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:24.032610 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a8fb1f87-8ad1-4476-8117-c7b5d129a5d0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-wbctj\" (UID: \"a8fb1f87-8ad1-4476-8117-c7b5d129a5d0\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-wbctj" Jul 22 03:03:24.032815 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:24.032634 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crh8d\" (UniqueName: \"kubernetes.io/projected/a8fb1f87-8ad1-4476-8117-c7b5d129a5d0-kube-api-access-crh8d\") pod \"istiod-openshift-gateway-7574459656-wbctj\" (UID: \"a8fb1f87-8ad1-4476-8117-c7b5d129a5d0\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-wbctj" Jul 22 03:03:24.133768 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:24.133730 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8fb1f87-8ad1-4476-8117-c7b5d129a5d0-local-certs\") pod \"istiod-openshift-gateway-7574459656-wbctj\" (UID: \"a8fb1f87-8ad1-4476-8117-c7b5d129a5d0\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-wbctj" Jul 22 03:03:24.134438 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:24.133779 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a8fb1f87-8ad1-4476-8117-c7b5d129a5d0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-wbctj\" (UID: \"a8fb1f87-8ad1-4476-8117-c7b5d129a5d0\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-wbctj" Jul 22 03:03:24.134438 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:24.133914 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a8fb1f87-8ad1-4476-8117-c7b5d129a5d0-cacerts\") pod \"istiod-openshift-gateway-7574459656-wbctj\" (UID: \"a8fb1f87-8ad1-4476-8117-c7b5d129a5d0\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-wbctj" Jul 22 03:03:24.134438 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:24.133959 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a8fb1f87-8ad1-4476-8117-c7b5d129a5d0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-wbctj\" (UID: \"a8fb1f87-8ad1-4476-8117-c7b5d129a5d0\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-wbctj" Jul 22 03:03:24.134438 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:24.134048 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8fb1f87-8ad1-4476-8117-c7b5d129a5d0-istio-token\") pod \"istiod-openshift-gateway-7574459656-wbctj\" (UID: \"a8fb1f87-8ad1-4476-8117-c7b5d129a5d0\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-wbctj" Jul 22 03:03:24.134438 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:24.134120 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a8fb1f87-8ad1-4476-8117-c7b5d129a5d0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-wbctj\" (UID: \"a8fb1f87-8ad1-4476-8117-c7b5d129a5d0\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-wbctj" Jul 22 03:03:24.134438 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:24.134147 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crh8d\" (UniqueName: \"kubernetes.io/projected/a8fb1f87-8ad1-4476-8117-c7b5d129a5d0-kube-api-access-crh8d\") pod \"istiod-openshift-gateway-7574459656-wbctj\" (UID: \"a8fb1f87-8ad1-4476-8117-c7b5d129a5d0\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-wbctj" Jul 22 03:03:24.135088 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:24.134927 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a8fb1f87-8ad1-4476-8117-c7b5d129a5d0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-wbctj\" (UID: \"a8fb1f87-8ad1-4476-8117-c7b5d129a5d0\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-wbctj" Jul 22 03:03:24.136517 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:24.136492 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8fb1f87-8ad1-4476-8117-c7b5d129a5d0-local-certs\") pod \"istiod-openshift-gateway-7574459656-wbctj\" (UID: \"a8fb1f87-8ad1-4476-8117-c7b5d129a5d0\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-wbctj" Jul 22 03:03:24.136961 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:24.136931 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a8fb1f87-8ad1-4476-8117-c7b5d129a5d0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-wbctj\" (UID: \"a8fb1f87-8ad1-4476-8117-c7b5d129a5d0\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-wbctj" Jul 22 03:03:24.137427 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:24.137407 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a8fb1f87-8ad1-4476-8117-c7b5d129a5d0-cacerts\") pod \"istiod-openshift-gateway-7574459656-wbctj\" (UID: \"a8fb1f87-8ad1-4476-8117-c7b5d129a5d0\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-wbctj" Jul 22 03:03:24.137505 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:24.137447 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a8fb1f87-8ad1-4476-8117-c7b5d129a5d0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-wbctj\" (UID: \"a8fb1f87-8ad1-4476-8117-c7b5d129a5d0\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-wbctj" Jul 22 03:03:24.142450 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:24.142426 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crh8d\" (UniqueName: \"kubernetes.io/projected/a8fb1f87-8ad1-4476-8117-c7b5d129a5d0-kube-api-access-crh8d\") pod \"istiod-openshift-gateway-7574459656-wbctj\" (UID: \"a8fb1f87-8ad1-4476-8117-c7b5d129a5d0\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-wbctj" Jul 22 03:03:24.143155 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:24.143137 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8fb1f87-8ad1-4476-8117-c7b5d129a5d0-istio-token\") pod \"istiod-openshift-gateway-7574459656-wbctj\" (UID: \"a8fb1f87-8ad1-4476-8117-c7b5d129a5d0\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-wbctj" Jul 22 03:03:24.253128 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:24.253037 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-wbctj" Jul 22 03:03:24.399445 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:24.399419 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-wbctj"] Jul 22 03:03:24.401856 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:24.401819 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-pnl96" event={"ID":"20b8fa3f-a5f9-425b-8fd2-e52ee34adf8a","Type":"ContainerStarted","Data":"f740f33e5390b455b2332a24474b8489c3d19fb5fcd78ed244c4c202ace1f43a"} Jul 22 03:03:24.401991 ip-10-0-131-67 kubenswrapper[2582]: W0722 03:03:24.401878 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8fb1f87_8ad1_4476_8117_c7b5d129a5d0.slice/crio-2822a237cefcf70143fe1facdd95c8de0d0b0251653b7d7f47bcd855e74f5910 WatchSource:0}: Error finding container 2822a237cefcf70143fe1facdd95c8de0d0b0251653b7d7f47bcd855e74f5910: Status 404 returned error can't find the container with id 2822a237cefcf70143fe1facdd95c8de0d0b0251653b7d7f47bcd855e74f5910 Jul 22 03:03:24.403729 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:24.403700 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-nbpjd" event={"ID":"39934543-08e4-4456-b200-27ce181b98e8","Type":"ContainerStarted","Data":"df14d81eabb02cf78015d321dc663bb6fc2b01bd32667aa6f838805482c3444e"} Jul 22 03:03:25.148477 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:25.148395 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:03:25.416561 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:25.416424 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-wbctj" event={"ID":"a8fb1f87-8ad1-4476-8117-c7b5d129a5d0","Type":"ContainerStarted","Data":"2822a237cefcf70143fe1facdd95c8de0d0b0251653b7d7f47bcd855e74f5910"} Jul 22 03:03:28.659027 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:28.658973 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 03:03:28.659396 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:28.659083 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 03:03:29.222693 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:29.222638 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:03:29.369653 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:29.369621 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:03:29.440672 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:29.440633 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-nbpjd" event={"ID":"39934543-08e4-4456-b200-27ce181b98e8","Type":"ContainerStarted","Data":"205f40b8f786b82541eb3a19580d139714a67d2c8a6d215dabf6c79a40539fa5"} Jul 22 03:03:29.440858 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:29.440814 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-nbpjd" Jul 22 03:03:29.442209 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:29.442185 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-684578bfd-2jd6x" event={"ID":"d10d14d5-4415-431c-81c3-623c8050e7c8","Type":"ContainerStarted","Data":"52c0928592434ddcf4e4b8c237d7e1f2abcec9fc20607dfdd1973e903c667604"} Jul 22 03:03:29.443701 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:29.443679 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-wbctj" event={"ID":"a8fb1f87-8ad1-4476-8117-c7b5d129a5d0","Type":"ContainerStarted","Data":"a7b6a8d0e371bbf0eac60a84e90c9dc2780d15d587bde4e6f5bc605367658b83"} Jul 22 03:03:29.443800 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:29.443722 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-wbctj" Jul 22 03:03:29.445425 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:29.445396 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-pnl96" event={"ID":"20b8fa3f-a5f9-425b-8fd2-e52ee34adf8a","Type":"ContainerStarted","Data":"90bb409c4ad4d7cbdd14d4b2ecad3b0d9f54b1dbd6b1f1b82e140ef21d9f94b1"} Jul 22 03:03:29.445545 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:29.445529 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-pnl96" Jul 22 03:03:29.445744 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:29.445723 2582 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-wbctj container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 22 03:03:29.445802 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:29.445767 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-wbctj" podUID="a8fb1f87-8ad1-4476-8117-c7b5d129a5d0" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 22 03:03:29.502587 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:29.502486 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-nbpjd" podStartSLOduration=2.784375946 podStartE2EDuration="7.502467943s" totalImagesPullingTime="4.718091997s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:03:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:03:29.474774505 +0000 UTC m=+320.206395351" watchObservedRunningTime="2026-07-22 03:03:29.502467943 +0000 UTC m=+320.234088821" Jul 22 03:03:29.504025 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:29.503981 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-wbctj" podStartSLOduration=2.24961448 podStartE2EDuration="6.503971651s" totalImagesPullingTime="4.254357171s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:03:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:03:29.501129078 +0000 UTC m=+320.232749942" watchObservedRunningTime="2026-07-22 03:03:29.503971651 +0000 UTC m=+320.235592501" Jul 22 03:03:29.531315 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:29.531263 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-684578bfd-2jd6x" podStartSLOduration=1.993181405 podStartE2EDuration="7.531248656s" totalImagesPullingTime="5.538067251s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:03:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:03:29.529430047 +0000 UTC m=+320.261050942" watchObservedRunningTime="2026-07-22 03:03:29.531248656 +0000 UTC m=+320.262869528" Jul 22 03:03:29.551666 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:29.551614 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-pnl96" podStartSLOduration=1.973746959 podStartE2EDuration="6.551593922s" totalImagesPullingTime="4.577846963s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:03:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:03:29.547929646 +0000 UTC m=+320.279550494" watchObservedRunningTime="2026-07-22 03:03:29.551593922 +0000 UTC m=+320.283214769" Jul 22 03:03:30.145600 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:30.145576 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:03:30.316404 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:30.316370 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:03:30.417938 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:30.417859 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:03:30.450548 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:30.450524 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-wbctj" Jul 22 03:03:33.977055 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:33.976996 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:03:34.180850 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:34.180801 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:03:34.237914 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:34.237842 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-lzsb5"] Jul 22 03:03:34.241249 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:34.241231 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-lzsb5" Jul 22 03:03:34.246091 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:34.246072 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 22 03:03:34.246209 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:34.246139 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-25cl9\"" Jul 22 03:03:34.271793 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:34.271588 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-lzsb5"] Jul 22 03:03:34.273559 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:34.273535 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-grbpl"] Jul 22 03:03:34.276863 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:34.276843 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-grbpl" Jul 22 03:03:34.279645 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:34.279625 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 22 03:03:34.279776 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:34.279627 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-fdmwz\"" Jul 22 03:03:34.295247 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:34.295216 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:03:34.296722 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:34.296698 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-grbpl"] Jul 22 03:03:34.332789 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:34.332754 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvc2d\" (UniqueName: \"kubernetes.io/projected/6edeec1e-1428-4fe4-b99b-f4348dc39d74-kube-api-access-zvc2d\") pod \"kserve-controller-manager-d6f4b65b8-lzsb5\" (UID: \"6edeec1e-1428-4fe4-b99b-f4348dc39d74\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-lzsb5" Jul 22 03:03:34.332789 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:34.332793 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdtbp\" (UniqueName: \"kubernetes.io/projected/c6393858-4cb7-4730-82ad-eac79608c284-kube-api-access-pdtbp\") pod \"llmisvc-controller-manager-777c6bd7d5-grbpl\" (UID: \"c6393858-4cb7-4730-82ad-eac79608c284\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-grbpl" Jul 22 03:03:34.333006 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:34.332875 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6edeec1e-1428-4fe4-b99b-f4348dc39d74-cert\") pod \"kserve-controller-manager-d6f4b65b8-lzsb5\" (UID: \"6edeec1e-1428-4fe4-b99b-f4348dc39d74\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-lzsb5" Jul 22 03:03:34.333006 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:34.332900 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c6393858-4cb7-4730-82ad-eac79608c284-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-grbpl\" (UID: \"c6393858-4cb7-4730-82ad-eac79608c284\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-grbpl" Jul 22 03:03:34.433960 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:34.433924 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6edeec1e-1428-4fe4-b99b-f4348dc39d74-cert\") pod \"kserve-controller-manager-d6f4b65b8-lzsb5\" (UID: \"6edeec1e-1428-4fe4-b99b-f4348dc39d74\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-lzsb5" Jul 22 03:03:34.433960 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:34.433963 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c6393858-4cb7-4730-82ad-eac79608c284-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-grbpl\" (UID: \"c6393858-4cb7-4730-82ad-eac79608c284\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-grbpl" Jul 22 03:03:34.434183 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:34.434042 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvc2d\" (UniqueName: \"kubernetes.io/projected/6edeec1e-1428-4fe4-b99b-f4348dc39d74-kube-api-access-zvc2d\") pod \"kserve-controller-manager-d6f4b65b8-lzsb5\" (UID: \"6edeec1e-1428-4fe4-b99b-f4348dc39d74\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-lzsb5" Jul 22 03:03:34.434183 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:34.434069 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdtbp\" (UniqueName: \"kubernetes.io/projected/c6393858-4cb7-4730-82ad-eac79608c284-kube-api-access-pdtbp\") pod \"llmisvc-controller-manager-777c6bd7d5-grbpl\" (UID: \"c6393858-4cb7-4730-82ad-eac79608c284\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-grbpl" Jul 22 03:03:34.434183 ip-10-0-131-67 kubenswrapper[2582]: E0722 03:03:34.434109 2582 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 22 03:03:34.434183 ip-10-0-131-67 kubenswrapper[2582]: E0722 03:03:34.434173 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6edeec1e-1428-4fe4-b99b-f4348dc39d74-cert podName:6edeec1e-1428-4fe4-b99b-f4348dc39d74 nodeName:}" failed. No retries permitted until 2026-07-22 03:03:34.934153141 +0000 UTC m=+325.665773975 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/6edeec1e-1428-4fe4-b99b-f4348dc39d74-cert") pod "kserve-controller-manager-d6f4b65b8-lzsb5" (UID: "6edeec1e-1428-4fe4-b99b-f4348dc39d74") : secret "kserve-webhook-server-cert" not found Jul 22 03:03:34.434336 ip-10-0-131-67 kubenswrapper[2582]: E0722 03:03:34.434257 2582 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 22 03:03:34.434336 ip-10-0-131-67 kubenswrapper[2582]: E0722 03:03:34.434303 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6393858-4cb7-4730-82ad-eac79608c284-cert podName:c6393858-4cb7-4730-82ad-eac79608c284 nodeName:}" failed. No retries permitted until 2026-07-22 03:03:34.934290622 +0000 UTC m=+325.665911447 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/c6393858-4cb7-4730-82ad-eac79608c284-cert") pod "llmisvc-controller-manager-777c6bd7d5-grbpl" (UID: "c6393858-4cb7-4730-82ad-eac79608c284") : secret "llmisvc-webhook-server-cert" not found Jul 22 03:03:34.487660 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:34.487623 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvc2d\" (UniqueName: \"kubernetes.io/projected/6edeec1e-1428-4fe4-b99b-f4348dc39d74-kube-api-access-zvc2d\") pod \"kserve-controller-manager-d6f4b65b8-lzsb5\" (UID: \"6edeec1e-1428-4fe4-b99b-f4348dc39d74\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-lzsb5" Jul 22 03:03:34.501835 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:34.501767 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdtbp\" (UniqueName: \"kubernetes.io/projected/c6393858-4cb7-4730-82ad-eac79608c284-kube-api-access-pdtbp\") pod \"llmisvc-controller-manager-777c6bd7d5-grbpl\" (UID: \"c6393858-4cb7-4730-82ad-eac79608c284\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-grbpl" Jul 22 03:03:34.554497 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:34.554462 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:03:34.626845 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:34.626814 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:03:34.938095 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:34.938047 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6edeec1e-1428-4fe4-b99b-f4348dc39d74-cert\") pod \"kserve-controller-manager-d6f4b65b8-lzsb5\" (UID: \"6edeec1e-1428-4fe4-b99b-f4348dc39d74\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-lzsb5" Jul 22 03:03:34.938095 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:34.938102 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c6393858-4cb7-4730-82ad-eac79608c284-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-grbpl\" (UID: \"c6393858-4cb7-4730-82ad-eac79608c284\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-grbpl" Jul 22 03:03:34.940545 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:34.940518 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c6393858-4cb7-4730-82ad-eac79608c284-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-grbpl\" (UID: \"c6393858-4cb7-4730-82ad-eac79608c284\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-grbpl" Jul 22 03:03:34.940664 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:34.940547 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6edeec1e-1428-4fe4-b99b-f4348dc39d74-cert\") pod \"kserve-controller-manager-d6f4b65b8-lzsb5\" (UID: \"6edeec1e-1428-4fe4-b99b-f4348dc39d74\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-lzsb5" Jul 22 03:03:35.040751 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:35.040718 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-wbctj"] Jul 22 03:03:35.041180 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:35.040884 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-wbctj" podUID="a8fb1f87-8ad1-4476-8117-c7b5d129a5d0" containerName="discovery" containerID="cri-o://a7b6a8d0e371bbf0eac60a84e90c9dc2780d15d587bde4e6f5bc605367658b83" gracePeriod=30 Jul 22 03:03:35.151465 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:35.151434 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-lzsb5" Jul 22 03:03:35.186704 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:35.186666 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-grbpl" Jul 22 03:03:35.330612 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:35.330584 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-lzsb5"] Jul 22 03:03:35.334359 ip-10-0-131-67 kubenswrapper[2582]: W0722 03:03:35.334333 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6edeec1e_1428_4fe4_b99b_f4348dc39d74.slice/crio-df0aa47fb5a88eb6d4677452357626a212722ed54730ae807b4ab99b6b904766 WatchSource:0}: Error finding container df0aa47fb5a88eb6d4677452357626a212722ed54730ae807b4ab99b6b904766: Status 404 returned error can't find the container with id df0aa47fb5a88eb6d4677452357626a212722ed54730ae807b4ab99b6b904766 Jul 22 03:03:35.335527 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:35.335509 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-wbctj" Jul 22 03:03:35.371310 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:35.365796 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-grbpl"] Jul 22 03:03:35.445736 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:35.445642 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8fb1f87-8ad1-4476-8117-c7b5d129a5d0-istio-token\") pod \"a8fb1f87-8ad1-4476-8117-c7b5d129a5d0\" (UID: \"a8fb1f87-8ad1-4476-8117-c7b5d129a5d0\") " Jul 22 03:03:35.445736 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:35.445689 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a8fb1f87-8ad1-4476-8117-c7b5d129a5d0-istio-csr-ca-configmap\") pod \"a8fb1f87-8ad1-4476-8117-c7b5d129a5d0\" (UID: \"a8fb1f87-8ad1-4476-8117-c7b5d129a5d0\") " Jul 22 03:03:35.445736 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:35.445711 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a8fb1f87-8ad1-4476-8117-c7b5d129a5d0-istio-kubeconfig\") pod \"a8fb1f87-8ad1-4476-8117-c7b5d129a5d0\" (UID: \"a8fb1f87-8ad1-4476-8117-c7b5d129a5d0\") " Jul 22 03:03:35.445736 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:35.445739 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a8fb1f87-8ad1-4476-8117-c7b5d129a5d0-cacerts\") pod \"a8fb1f87-8ad1-4476-8117-c7b5d129a5d0\" (UID: \"a8fb1f87-8ad1-4476-8117-c7b5d129a5d0\") " Jul 22 03:03:35.446115 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:35.445919 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-crh8d\" (UniqueName: \"kubernetes.io/projected/a8fb1f87-8ad1-4476-8117-c7b5d129a5d0-kube-api-access-crh8d\") pod \"a8fb1f87-8ad1-4476-8117-c7b5d129a5d0\" (UID: \"a8fb1f87-8ad1-4476-8117-c7b5d129a5d0\") " Jul 22 03:03:35.446115 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:35.445988 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8fb1f87-8ad1-4476-8117-c7b5d129a5d0-local-certs\") pod \"a8fb1f87-8ad1-4476-8117-c7b5d129a5d0\" (UID: \"a8fb1f87-8ad1-4476-8117-c7b5d129a5d0\") " Jul 22 03:03:35.446115 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:35.446056 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a8fb1f87-8ad1-4476-8117-c7b5d129a5d0-istio-csr-dns-cert\") pod \"a8fb1f87-8ad1-4476-8117-c7b5d129a5d0\" (UID: \"a8fb1f87-8ad1-4476-8117-c7b5d129a5d0\") " Jul 22 03:03:35.446257 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:35.446157 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8fb1f87-8ad1-4476-8117-c7b5d129a5d0-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "a8fb1f87-8ad1-4476-8117-c7b5d129a5d0" (UID: "a8fb1f87-8ad1-4476-8117-c7b5d129a5d0"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 03:03:35.446371 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:35.446352 2582 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a8fb1f87-8ad1-4476-8117-c7b5d129a5d0-istio-csr-ca-configmap\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 03:03:35.448189 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:35.448162 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8fb1f87-8ad1-4476-8117-c7b5d129a5d0-cacerts" (OuterVolumeSpecName: "cacerts") pod "a8fb1f87-8ad1-4476-8117-c7b5d129a5d0" (UID: "a8fb1f87-8ad1-4476-8117-c7b5d129a5d0"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 03:03:35.448314 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:35.448167 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8fb1f87-8ad1-4476-8117-c7b5d129a5d0-local-certs" (OuterVolumeSpecName: "local-certs") pod "a8fb1f87-8ad1-4476-8117-c7b5d129a5d0" (UID: "a8fb1f87-8ad1-4476-8117-c7b5d129a5d0"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:03:35.448314 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:35.448265 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8fb1f87-8ad1-4476-8117-c7b5d129a5d0-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "a8fb1f87-8ad1-4476-8117-c7b5d129a5d0" (UID: "a8fb1f87-8ad1-4476-8117-c7b5d129a5d0"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 03:03:35.448390 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:35.448309 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8fb1f87-8ad1-4476-8117-c7b5d129a5d0-istio-token" (OuterVolumeSpecName: "istio-token") pod "a8fb1f87-8ad1-4476-8117-c7b5d129a5d0" (UID: "a8fb1f87-8ad1-4476-8117-c7b5d129a5d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 03:03:35.448579 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:35.448554 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8fb1f87-8ad1-4476-8117-c7b5d129a5d0-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "a8fb1f87-8ad1-4476-8117-c7b5d129a5d0" (UID: "a8fb1f87-8ad1-4476-8117-c7b5d129a5d0"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 03:03:35.448579 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:35.448566 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8fb1f87-8ad1-4476-8117-c7b5d129a5d0-kube-api-access-crh8d" (OuterVolumeSpecName: "kube-api-access-crh8d") pod "a8fb1f87-8ad1-4476-8117-c7b5d129a5d0" (UID: "a8fb1f87-8ad1-4476-8117-c7b5d129a5d0"). InnerVolumeSpecName "kube-api-access-crh8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 03:03:35.473677 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:35.473642 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-grbpl" event={"ID":"c6393858-4cb7-4730-82ad-eac79608c284","Type":"ContainerStarted","Data":"fbf5137aaf3c4a5df7e7d717bf5943362b8e4a1f74bb13647539d3c1319865ae"} Jul 22 03:03:35.474851 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:35.474819 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-lzsb5" event={"ID":"6edeec1e-1428-4fe4-b99b-f4348dc39d74","Type":"ContainerStarted","Data":"df0aa47fb5a88eb6d4677452357626a212722ed54730ae807b4ab99b6b904766"} Jul 22 03:03:35.476092 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:35.476060 2582 generic.go:356] "Generic (PLEG): container finished" podID="a8fb1f87-8ad1-4476-8117-c7b5d129a5d0" containerID="a7b6a8d0e371bbf0eac60a84e90c9dc2780d15d587bde4e6f5bc605367658b83" exitCode=0 Jul 22 03:03:35.476214 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:35.476133 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-wbctj" Jul 22 03:03:35.476214 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:35.476148 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-wbctj" event={"ID":"a8fb1f87-8ad1-4476-8117-c7b5d129a5d0","Type":"ContainerDied","Data":"a7b6a8d0e371bbf0eac60a84e90c9dc2780d15d587bde4e6f5bc605367658b83"} Jul 22 03:03:35.476214 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:35.476196 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-wbctj" event={"ID":"a8fb1f87-8ad1-4476-8117-c7b5d129a5d0","Type":"ContainerDied","Data":"2822a237cefcf70143fe1facdd95c8de0d0b0251653b7d7f47bcd855e74f5910"} Jul 22 03:03:35.476357 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:35.476218 2582 scope.go:117] "RemoveContainer" containerID="a7b6a8d0e371bbf0eac60a84e90c9dc2780d15d587bde4e6f5bc605367658b83" Jul 22 03:03:35.484953 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:35.484931 2582 scope.go:117] "RemoveContainer" containerID="a7b6a8d0e371bbf0eac60a84e90c9dc2780d15d587bde4e6f5bc605367658b83" Jul 22 03:03:35.485242 ip-10-0-131-67 kubenswrapper[2582]: E0722 03:03:35.485222 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7b6a8d0e371bbf0eac60a84e90c9dc2780d15d587bde4e6f5bc605367658b83\": container with ID starting with a7b6a8d0e371bbf0eac60a84e90c9dc2780d15d587bde4e6f5bc605367658b83 not found: ID does not exist" containerID="a7b6a8d0e371bbf0eac60a84e90c9dc2780d15d587bde4e6f5bc605367658b83" Jul 22 03:03:35.485329 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:35.485253 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7b6a8d0e371bbf0eac60a84e90c9dc2780d15d587bde4e6f5bc605367658b83"} err="failed to get container status \"a7b6a8d0e371bbf0eac60a84e90c9dc2780d15d587bde4e6f5bc605367658b83\": rpc error: code = NotFound desc = could not find container \"a7b6a8d0e371bbf0eac60a84e90c9dc2780d15d587bde4e6f5bc605367658b83\": container with ID starting with a7b6a8d0e371bbf0eac60a84e90c9dc2780d15d587bde4e6f5bc605367658b83 not found: ID does not exist" Jul 22 03:03:35.508197 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:35.508165 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-wbctj"] Jul 22 03:03:35.522183 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:35.522150 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-wbctj"] Jul 22 03:03:35.547683 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:35.547652 2582 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8fb1f87-8ad1-4476-8117-c7b5d129a5d0-local-certs\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 03:03:35.547683 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:35.547682 2582 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a8fb1f87-8ad1-4476-8117-c7b5d129a5d0-istio-csr-dns-cert\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 03:03:35.547869 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:35.547693 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8fb1f87-8ad1-4476-8117-c7b5d129a5d0-istio-token\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 03:03:35.547869 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:35.547703 2582 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a8fb1f87-8ad1-4476-8117-c7b5d129a5d0-istio-kubeconfig\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 03:03:35.547869 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:35.547711 2582 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a8fb1f87-8ad1-4476-8117-c7b5d129a5d0-cacerts\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 03:03:35.547869 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:35.547718 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crh8d\" (UniqueName: \"kubernetes.io/projected/a8fb1f87-8ad1-4476-8117-c7b5d129a5d0-kube-api-access-crh8d\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 03:03:35.710614 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:35.710522 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:03:35.816220 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:35.816190 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:03:35.901950 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:35.901919 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8fb1f87-8ad1-4476-8117-c7b5d129a5d0" path="/var/lib/kubelet/pods/a8fb1f87-8ad1-4476-8117-c7b5d129a5d0/volumes" Jul 22 03:03:35.913688 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:35.913646 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:03:36.484548 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:36.484515 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:03:36.581647 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:36.581596 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:03:38.496003 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:38.495904 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-lzsb5" event={"ID":"6edeec1e-1428-4fe4-b99b-f4348dc39d74","Type":"ContainerStarted","Data":"b15274a7335f383fd380c22d82f5e8c466324ef68d271b431c3f320e3ad4e1bd"} Jul 22 03:03:38.496427 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:38.496027 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-lzsb5" Jul 22 03:03:38.515214 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:38.515163 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-lzsb5" podStartSLOduration=1.65431703 podStartE2EDuration="4.515142983s" totalImagesPullingTime="2.860825953s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:03:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:03:38.51467119 +0000 UTC m=+329.246292081" watchObservedRunningTime="2026-07-22 03:03:38.515142983 +0000 UTC m=+329.246763831" Jul 22 03:03:39.502812 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:39.502733 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-grbpl" event={"ID":"c6393858-4cb7-4730-82ad-eac79608c284","Type":"ContainerStarted","Data":"1d7ba1820cde8525cd498811a01527e16de19a8810674e5d17969fdae341790d"} Jul 22 03:03:39.527961 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:39.527903 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-grbpl" podStartSLOduration=1.703138074 podStartE2EDuration="5.527884686s" totalImagesPullingTime="3.824746612s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:03:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:03:39.525915079 +0000 UTC m=+330.257535945" watchObservedRunningTime="2026-07-22 03:03:39.527884686 +0000 UTC m=+330.259505533" Jul 22 03:03:40.452177 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:40.452139 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-nbpjd" Jul 22 03:03:40.454368 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:40.454346 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-pnl96" Jul 22 03:03:40.508558 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:03:40.508528 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-grbpl" Jul 22 03:04:09.508336 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:09.508257 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-lzsb5" Jul 22 03:04:11.515638 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:11.515603 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-grbpl" Jul 22 03:04:23.745353 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:23.745311 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:04:23.885175 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:23.885141 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:04:23.892745 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:23.892686 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-pm2pb"] Jul 22 03:04:23.893146 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:23.893132 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8fb1f87-8ad1-4476-8117-c7b5d129a5d0" containerName="discovery" Jul 22 03:04:23.893205 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:23.893148 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8fb1f87-8ad1-4476-8117-c7b5d129a5d0" containerName="discovery" Jul 22 03:04:23.893241 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:23.893212 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8fb1f87-8ad1-4476-8117-c7b5d129a5d0" containerName="discovery" Jul 22 03:04:23.899138 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:23.899112 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-pm2pb" Jul 22 03:04:23.902392 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:23.902367 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 22 03:04:23.902641 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:23.902627 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-8fs9l\"" Jul 22 03:04:23.902789 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:23.902621 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 22 03:04:23.908158 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:23.908134 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-pm2pb"] Jul 22 03:04:23.955899 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:23.955870 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:04:23.976226 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:23.976190 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqgb4\" (UniqueName: \"kubernetes.io/projected/d0c2013c-a7e9-4155-89b0-77d7a8cf9863-kube-api-access-dqgb4\") pod \"kuadrant-operator-catalog-pm2pb\" (UID: \"d0c2013c-a7e9-4155-89b0-77d7a8cf9863\") " pod="kuadrant-system/kuadrant-operator-catalog-pm2pb" Jul 22 03:04:24.077641 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:24.077551 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqgb4\" (UniqueName: \"kubernetes.io/projected/d0c2013c-a7e9-4155-89b0-77d7a8cf9863-kube-api-access-dqgb4\") pod \"kuadrant-operator-catalog-pm2pb\" (UID: \"d0c2013c-a7e9-4155-89b0-77d7a8cf9863\") " pod="kuadrant-system/kuadrant-operator-catalog-pm2pb" Jul 22 03:04:24.086037 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:24.085985 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqgb4\" (UniqueName: \"kubernetes.io/projected/d0c2013c-a7e9-4155-89b0-77d7a8cf9863-kube-api-access-dqgb4\") pod \"kuadrant-operator-catalog-pm2pb\" (UID: \"d0c2013c-a7e9-4155-89b0-77d7a8cf9863\") " pod="kuadrant-system/kuadrant-operator-catalog-pm2pb" Jul 22 03:04:24.212130 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:24.212079 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-pm2pb" Jul 22 03:04:24.237072 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:24.237042 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-pm2pb"] Jul 22 03:04:24.348662 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:24.348636 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-pm2pb"] Jul 22 03:04:24.351317 ip-10-0-131-67 kubenswrapper[2582]: W0722 03:04:24.351282 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0c2013c_a7e9_4155_89b0_77d7a8cf9863.slice/crio-ff7345695936ce886745cb5c36683afef976ef54390b8b63fadf2cc9cae43b97 WatchSource:0}: Error finding container ff7345695936ce886745cb5c36683afef976ef54390b8b63fadf2cc9cae43b97: Status 404 returned error can't find the container with id ff7345695936ce886745cb5c36683afef976ef54390b8b63fadf2cc9cae43b97 Jul 22 03:04:24.451219 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:24.451182 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-sp78b"] Jul 22 03:04:24.454339 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:24.454315 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-sp78b" Jul 22 03:04:24.464067 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:24.464042 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-sp78b"] Jul 22 03:04:24.582715 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:24.582679 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrtb5\" (UniqueName: \"kubernetes.io/projected/dc453083-f6e0-4809-9b8e-92034ab30c43-kube-api-access-mrtb5\") pod \"kuadrant-operator-catalog-sp78b\" (UID: \"dc453083-f6e0-4809-9b8e-92034ab30c43\") " pod="kuadrant-system/kuadrant-operator-catalog-sp78b" Jul 22 03:04:24.683124 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:24.683085 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrtb5\" (UniqueName: \"kubernetes.io/projected/dc453083-f6e0-4809-9b8e-92034ab30c43-kube-api-access-mrtb5\") pod \"kuadrant-operator-catalog-sp78b\" (UID: \"dc453083-f6e0-4809-9b8e-92034ab30c43\") " pod="kuadrant-system/kuadrant-operator-catalog-sp78b" Jul 22 03:04:24.691657 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:24.691629 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrtb5\" (UniqueName: \"kubernetes.io/projected/dc453083-f6e0-4809-9b8e-92034ab30c43-kube-api-access-mrtb5\") pod \"kuadrant-operator-catalog-sp78b\" (UID: \"dc453083-f6e0-4809-9b8e-92034ab30c43\") " pod="kuadrant-system/kuadrant-operator-catalog-sp78b" Jul 22 03:04:24.718504 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:24.718464 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-pm2pb" event={"ID":"d0c2013c-a7e9-4155-89b0-77d7a8cf9863","Type":"ContainerStarted","Data":"ff7345695936ce886745cb5c36683afef976ef54390b8b63fadf2cc9cae43b97"} Jul 22 03:04:24.763860 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:24.763820 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-sp78b" Jul 22 03:04:24.910900 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:24.910870 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-sp78b"] Jul 22 03:04:24.914070 ip-10-0-131-67 kubenswrapper[2582]: W0722 03:04:24.914042 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc453083_f6e0_4809_9b8e_92034ab30c43.slice/crio-b1f58aef6ccab102409d2780910c8e8387337f8d59700df66881291e6f2e0f70 WatchSource:0}: Error finding container b1f58aef6ccab102409d2780910c8e8387337f8d59700df66881291e6f2e0f70: Status 404 returned error can't find the container with id b1f58aef6ccab102409d2780910c8e8387337f8d59700df66881291e6f2e0f70 Jul 22 03:04:25.728409 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:25.728324 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-sp78b" event={"ID":"dc453083-f6e0-4809-9b8e-92034ab30c43","Type":"ContainerStarted","Data":"b1f58aef6ccab102409d2780910c8e8387337f8d59700df66881291e6f2e0f70"} Jul 22 03:04:26.735332 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:26.735229 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-sp78b" event={"ID":"dc453083-f6e0-4809-9b8e-92034ab30c43","Type":"ContainerStarted","Data":"587476c07af0c9200453dd6a99bf8881049e952d4b36989e885df192c2237879"} Jul 22 03:04:26.736759 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:26.736738 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-pm2pb" event={"ID":"d0c2013c-a7e9-4155-89b0-77d7a8cf9863","Type":"ContainerStarted","Data":"2698f0ddeb2a58db0e3404a096938d66a21aaf1836785580ed9b2c1adde1229b"} Jul 22 03:04:26.736845 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:26.736798 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-pm2pb" podUID="d0c2013c-a7e9-4155-89b0-77d7a8cf9863" containerName="registry-server" containerID="cri-o://2698f0ddeb2a58db0e3404a096938d66a21aaf1836785580ed9b2c1adde1229b" gracePeriod=2 Jul 22 03:04:26.756022 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:26.755946 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-sp78b" podStartSLOduration=1.222099517 podStartE2EDuration="2.755928293s" totalImagesPullingTime="1.533828776s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:04:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:04:26.754624071 +0000 UTC m=+377.486244948" watchObservedRunningTime="2026-07-22 03:04:26.755928293 +0000 UTC m=+377.487549140" Jul 22 03:04:26.772127 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:26.772065 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-pm2pb" podStartSLOduration=1.680549552 podStartE2EDuration="3.772046057s" totalImagesPullingTime="2.091496505s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:04:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:04:26.770349913 +0000 UTC m=+377.501970760" watchObservedRunningTime="2026-07-22 03:04:26.772046057 +0000 UTC m=+377.503667261" Jul 22 03:04:26.985858 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:26.985791 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-pm2pb" Jul 22 03:04:27.105462 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:27.105429 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dqgb4\" (UniqueName: \"kubernetes.io/projected/d0c2013c-a7e9-4155-89b0-77d7a8cf9863-kube-api-access-dqgb4\") pod \"d0c2013c-a7e9-4155-89b0-77d7a8cf9863\" (UID: \"d0c2013c-a7e9-4155-89b0-77d7a8cf9863\") " Jul 22 03:04:27.107614 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:27.107585 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0c2013c-a7e9-4155-89b0-77d7a8cf9863-kube-api-access-dqgb4" (OuterVolumeSpecName: "kube-api-access-dqgb4") pod "d0c2013c-a7e9-4155-89b0-77d7a8cf9863" (UID: "d0c2013c-a7e9-4155-89b0-77d7a8cf9863"). InnerVolumeSpecName "kube-api-access-dqgb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 03:04:27.207044 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:27.206967 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqgb4\" (UniqueName: \"kubernetes.io/projected/d0c2013c-a7e9-4155-89b0-77d7a8cf9863-kube-api-access-dqgb4\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 03:04:27.742993 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:27.742906 2582 generic.go:356] "Generic (PLEG): container finished" podID="d0c2013c-a7e9-4155-89b0-77d7a8cf9863" containerID="2698f0ddeb2a58db0e3404a096938d66a21aaf1836785580ed9b2c1adde1229b" exitCode=0 Jul 22 03:04:27.742993 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:27.742956 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-pm2pb" Jul 22 03:04:27.743451 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:27.742996 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-pm2pb" event={"ID":"d0c2013c-a7e9-4155-89b0-77d7a8cf9863","Type":"ContainerDied","Data":"2698f0ddeb2a58db0e3404a096938d66a21aaf1836785580ed9b2c1adde1229b"} Jul 22 03:04:27.743451 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:27.743099 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-pm2pb" event={"ID":"d0c2013c-a7e9-4155-89b0-77d7a8cf9863","Type":"ContainerDied","Data":"ff7345695936ce886745cb5c36683afef976ef54390b8b63fadf2cc9cae43b97"} Jul 22 03:04:27.743451 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:27.743120 2582 scope.go:117] "RemoveContainer" containerID="2698f0ddeb2a58db0e3404a096938d66a21aaf1836785580ed9b2c1adde1229b" Jul 22 03:04:27.753396 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:27.753377 2582 scope.go:117] "RemoveContainer" containerID="2698f0ddeb2a58db0e3404a096938d66a21aaf1836785580ed9b2c1adde1229b" Jul 22 03:04:27.753722 ip-10-0-131-67 kubenswrapper[2582]: E0722 03:04:27.753703 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2698f0ddeb2a58db0e3404a096938d66a21aaf1836785580ed9b2c1adde1229b\": container with ID starting with 2698f0ddeb2a58db0e3404a096938d66a21aaf1836785580ed9b2c1adde1229b not found: ID does not exist" containerID="2698f0ddeb2a58db0e3404a096938d66a21aaf1836785580ed9b2c1adde1229b" Jul 22 03:04:27.753784 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:27.753732 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2698f0ddeb2a58db0e3404a096938d66a21aaf1836785580ed9b2c1adde1229b"} err="failed to get container status \"2698f0ddeb2a58db0e3404a096938d66a21aaf1836785580ed9b2c1adde1229b\": rpc error: code = NotFound desc = could not find container \"2698f0ddeb2a58db0e3404a096938d66a21aaf1836785580ed9b2c1adde1229b\": container with ID starting with 2698f0ddeb2a58db0e3404a096938d66a21aaf1836785580ed9b2c1adde1229b not found: ID does not exist" Jul 22 03:04:27.765837 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:27.765805 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-pm2pb"] Jul 22 03:04:27.771505 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:27.771468 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-pm2pb"] Jul 22 03:04:27.901800 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:27.901751 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0c2013c-a7e9-4155-89b0-77d7a8cf9863" path="/var/lib/kubelet/pods/d0c2013c-a7e9-4155-89b0-77d7a8cf9863/volumes" Jul 22 03:04:34.764212 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:34.764171 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-sp78b" Jul 22 03:04:34.764652 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:34.764256 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-sp78b" Jul 22 03:04:34.787334 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:34.787308 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-sp78b" Jul 22 03:04:35.805157 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:35.805128 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-sp78b" Jul 22 03:04:39.505426 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:39.505384 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kvc54"] Jul 22 03:04:39.505885 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:39.505732 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0c2013c-a7e9-4155-89b0-77d7a8cf9863" containerName="registry-server" Jul 22 03:04:39.505885 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:39.505743 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0c2013c-a7e9-4155-89b0-77d7a8cf9863" containerName="registry-server" Jul 22 03:04:39.505885 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:39.505826 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0c2013c-a7e9-4155-89b0-77d7a8cf9863" containerName="registry-server" Jul 22 03:04:39.508297 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:39.508278 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kvc54" Jul 22 03:04:39.510860 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:39.510843 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-mp52x\"" Jul 22 03:04:39.518468 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:39.518443 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kvc54"] Jul 22 03:04:39.608876 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:39.608845 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k96zn\" (UniqueName: \"kubernetes.io/projected/3f608664-0cb4-47f7-9e41-ca87b2abe2cf-kube-api-access-k96zn\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kvc54\" (UID: \"3f608664-0cb4-47f7-9e41-ca87b2abe2cf\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kvc54" Jul 22 03:04:39.609078 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:39.608885 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3f608664-0cb4-47f7-9e41-ca87b2abe2cf-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kvc54\" (UID: \"3f608664-0cb4-47f7-9e41-ca87b2abe2cf\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kvc54" Jul 22 03:04:39.609078 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:39.608919 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3f608664-0cb4-47f7-9e41-ca87b2abe2cf-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kvc54\" (UID: \"3f608664-0cb4-47f7-9e41-ca87b2abe2cf\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kvc54" Jul 22 03:04:39.709714 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:39.709675 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k96zn\" (UniqueName: \"kubernetes.io/projected/3f608664-0cb4-47f7-9e41-ca87b2abe2cf-kube-api-access-k96zn\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kvc54\" (UID: \"3f608664-0cb4-47f7-9e41-ca87b2abe2cf\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kvc54" Jul 22 03:04:39.709884 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:39.709724 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3f608664-0cb4-47f7-9e41-ca87b2abe2cf-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kvc54\" (UID: \"3f608664-0cb4-47f7-9e41-ca87b2abe2cf\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kvc54" Jul 22 03:04:39.709884 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:39.709833 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3f608664-0cb4-47f7-9e41-ca87b2abe2cf-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kvc54\" (UID: \"3f608664-0cb4-47f7-9e41-ca87b2abe2cf\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kvc54" Jul 22 03:04:39.710121 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:39.710101 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3f608664-0cb4-47f7-9e41-ca87b2abe2cf-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kvc54\" (UID: \"3f608664-0cb4-47f7-9e41-ca87b2abe2cf\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kvc54" Jul 22 03:04:39.710167 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:39.710146 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3f608664-0cb4-47f7-9e41-ca87b2abe2cf-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kvc54\" (UID: \"3f608664-0cb4-47f7-9e41-ca87b2abe2cf\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kvc54" Jul 22 03:04:39.718675 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:39.718654 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k96zn\" (UniqueName: \"kubernetes.io/projected/3f608664-0cb4-47f7-9e41-ca87b2abe2cf-kube-api-access-k96zn\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kvc54\" (UID: \"3f608664-0cb4-47f7-9e41-ca87b2abe2cf\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kvc54" Jul 22 03:04:39.816963 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:39.816881 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kvc54" Jul 22 03:04:39.958518 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:39.958495 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kvc54"] Jul 22 03:04:39.960456 ip-10-0-131-67 kubenswrapper[2582]: W0722 03:04:39.960419 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f608664_0cb4_47f7_9e41_ca87b2abe2cf.slice/crio-643563aa536b9e247a28728e30ba2fbed717cd247e1dfe6811a18dab89ae2a9a WatchSource:0}: Error finding container 643563aa536b9e247a28728e30ba2fbed717cd247e1dfe6811a18dab89ae2a9a: Status 404 returned error can't find the container with id 643563aa536b9e247a28728e30ba2fbed717cd247e1dfe6811a18dab89ae2a9a Jul 22 03:04:40.095449 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:40.095381 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d6k5j"] Jul 22 03:04:40.097925 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:40.097910 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d6k5j" Jul 22 03:04:40.111065 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:40.111041 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d6k5j"] Jul 22 03:04:40.215173 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:40.215135 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f7e9741-e996-4fd5-8363-c54e86d3dd05-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d6k5j\" (UID: \"6f7e9741-e996-4fd5-8363-c54e86d3dd05\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d6k5j" Jul 22 03:04:40.215380 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:40.215200 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mrtl\" (UniqueName: \"kubernetes.io/projected/6f7e9741-e996-4fd5-8363-c54e86d3dd05-kube-api-access-8mrtl\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d6k5j\" (UID: \"6f7e9741-e996-4fd5-8363-c54e86d3dd05\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d6k5j" Jul 22 03:04:40.215380 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:40.215272 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f7e9741-e996-4fd5-8363-c54e86d3dd05-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d6k5j\" (UID: \"6f7e9741-e996-4fd5-8363-c54e86d3dd05\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d6k5j" Jul 22 03:04:40.316060 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:40.315997 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f7e9741-e996-4fd5-8363-c54e86d3dd05-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d6k5j\" (UID: \"6f7e9741-e996-4fd5-8363-c54e86d3dd05\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d6k5j" Jul 22 03:04:40.316247 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:40.316092 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mrtl\" (UniqueName: \"kubernetes.io/projected/6f7e9741-e996-4fd5-8363-c54e86d3dd05-kube-api-access-8mrtl\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d6k5j\" (UID: \"6f7e9741-e996-4fd5-8363-c54e86d3dd05\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d6k5j" Jul 22 03:04:40.316247 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:40.316122 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f7e9741-e996-4fd5-8363-c54e86d3dd05-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d6k5j\" (UID: \"6f7e9741-e996-4fd5-8363-c54e86d3dd05\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d6k5j" Jul 22 03:04:40.316458 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:40.316433 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f7e9741-e996-4fd5-8363-c54e86d3dd05-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d6k5j\" (UID: \"6f7e9741-e996-4fd5-8363-c54e86d3dd05\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d6k5j" Jul 22 03:04:40.316529 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:40.316477 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f7e9741-e996-4fd5-8363-c54e86d3dd05-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d6k5j\" (UID: \"6f7e9741-e996-4fd5-8363-c54e86d3dd05\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d6k5j" Jul 22 03:04:40.325353 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:40.325334 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mrtl\" (UniqueName: \"kubernetes.io/projected/6f7e9741-e996-4fd5-8363-c54e86d3dd05-kube-api-access-8mrtl\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d6k5j\" (UID: \"6f7e9741-e996-4fd5-8363-c54e86d3dd05\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d6k5j" Jul 22 03:04:40.414698 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:40.414664 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d6k5j" Jul 22 03:04:40.483804 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:40.483761 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef187sg6"] Jul 22 03:04:40.487602 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:40.487576 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef187sg6" Jul 22 03:04:40.507074 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:40.507047 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef187sg6"] Jul 22 03:04:40.558382 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:40.558355 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d6k5j"] Jul 22 03:04:40.561272 ip-10-0-131-67 kubenswrapper[2582]: W0722 03:04:40.561247 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f7e9741_e996_4fd5_8363_c54e86d3dd05.slice/crio-f2883bd6c7156b60cce6fb81c3b27491015f8f57fd85a94f539074a2b78a4376 WatchSource:0}: Error finding container f2883bd6c7156b60cce6fb81c3b27491015f8f57fd85a94f539074a2b78a4376: Status 404 returned error can't find the container with id f2883bd6c7156b60cce6fb81c3b27491015f8f57fd85a94f539074a2b78a4376 Jul 22 03:04:40.618438 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:40.618395 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/90fe6071-e5e0-4797-8252-17607fbc5def-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef187sg6\" (UID: \"90fe6071-e5e0-4797-8252-17607fbc5def\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef187sg6" Jul 22 03:04:40.618438 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:40.618451 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/90fe6071-e5e0-4797-8252-17607fbc5def-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef187sg6\" (UID: \"90fe6071-e5e0-4797-8252-17607fbc5def\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef187sg6" Jul 22 03:04:40.618613 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:40.618499 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfhgc\" (UniqueName: \"kubernetes.io/projected/90fe6071-e5e0-4797-8252-17607fbc5def-kube-api-access-dfhgc\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef187sg6\" (UID: \"90fe6071-e5e0-4797-8252-17607fbc5def\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef187sg6" Jul 22 03:04:40.719118 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:40.719048 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/90fe6071-e5e0-4797-8252-17607fbc5def-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef187sg6\" (UID: \"90fe6071-e5e0-4797-8252-17607fbc5def\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef187sg6" Jul 22 03:04:40.719118 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:40.719094 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfhgc\" (UniqueName: \"kubernetes.io/projected/90fe6071-e5e0-4797-8252-17607fbc5def-kube-api-access-dfhgc\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef187sg6\" (UID: \"90fe6071-e5e0-4797-8252-17607fbc5def\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef187sg6" Jul 22 03:04:40.719301 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:40.719210 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/90fe6071-e5e0-4797-8252-17607fbc5def-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef187sg6\" (UID: \"90fe6071-e5e0-4797-8252-17607fbc5def\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef187sg6" Jul 22 03:04:40.719375 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:40.719356 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/90fe6071-e5e0-4797-8252-17607fbc5def-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef187sg6\" (UID: \"90fe6071-e5e0-4797-8252-17607fbc5def\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef187sg6" Jul 22 03:04:40.719551 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:40.719531 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/90fe6071-e5e0-4797-8252-17607fbc5def-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef187sg6\" (UID: \"90fe6071-e5e0-4797-8252-17607fbc5def\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef187sg6" Jul 22 03:04:40.729816 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:40.729791 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfhgc\" (UniqueName: \"kubernetes.io/projected/90fe6071-e5e0-4797-8252-17607fbc5def-kube-api-access-dfhgc\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef187sg6\" (UID: \"90fe6071-e5e0-4797-8252-17607fbc5def\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef187sg6" Jul 22 03:04:40.797309 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:40.797276 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef187sg6" Jul 22 03:04:40.814800 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:40.814768 2582 generic.go:356] "Generic (PLEG): container finished" podID="3f608664-0cb4-47f7-9e41-ca87b2abe2cf" containerID="7df73fd09f0183a6fa02226c2bbb120c89ad93e9fcdb4e422a3fc0554b6c94d0" exitCode=0 Jul 22 03:04:40.814922 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:40.814857 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kvc54" event={"ID":"3f608664-0cb4-47f7-9e41-ca87b2abe2cf","Type":"ContainerDied","Data":"7df73fd09f0183a6fa02226c2bbb120c89ad93e9fcdb4e422a3fc0554b6c94d0"} Jul 22 03:04:40.815039 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:40.814999 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kvc54" event={"ID":"3f608664-0cb4-47f7-9e41-ca87b2abe2cf","Type":"ContainerStarted","Data":"643563aa536b9e247a28728e30ba2fbed717cd247e1dfe6811a18dab89ae2a9a"} Jul 22 03:04:40.816673 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:40.816644 2582 generic.go:356] "Generic (PLEG): container finished" podID="6f7e9741-e996-4fd5-8363-c54e86d3dd05" containerID="042a87fdfc35ef7d83ca29728caec93fe7896f802a22d6df632484b0ef7e070d" exitCode=0 Jul 22 03:04:40.816776 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:40.816700 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d6k5j" event={"ID":"6f7e9741-e996-4fd5-8363-c54e86d3dd05","Type":"ContainerDied","Data":"042a87fdfc35ef7d83ca29728caec93fe7896f802a22d6df632484b0ef7e070d"} Jul 22 03:04:40.816776 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:40.816749 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d6k5j" event={"ID":"6f7e9741-e996-4fd5-8363-c54e86d3dd05","Type":"ContainerStarted","Data":"f2883bd6c7156b60cce6fb81c3b27491015f8f57fd85a94f539074a2b78a4376"} Jul 22 03:04:40.899134 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:40.899096 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lv5sb"] Jul 22 03:04:40.904990 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:40.904967 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lv5sb" Jul 22 03:04:40.917154 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:40.917127 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lv5sb"] Jul 22 03:04:40.944509 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:40.944484 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef187sg6"] Jul 22 03:04:40.946740 ip-10-0-131-67 kubenswrapper[2582]: W0722 03:04:40.946717 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90fe6071_e5e0_4797_8252_17607fbc5def.slice/crio-0530cb7e00574d54eb0cd968778cf7ac318d49bf5a53f2626233f8e850144df5 WatchSource:0}: Error finding container 0530cb7e00574d54eb0cd968778cf7ac318d49bf5a53f2626233f8e850144df5: Status 404 returned error can't find the container with id 0530cb7e00574d54eb0cd968778cf7ac318d49bf5a53f2626233f8e850144df5 Jul 22 03:04:41.021690 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:41.021660 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bce77674-28ac-4fcf-8d29-73c7178ed338-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lv5sb\" (UID: \"bce77674-28ac-4fcf-8d29-73c7178ed338\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lv5sb" Jul 22 03:04:41.021836 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:41.021766 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bce77674-28ac-4fcf-8d29-73c7178ed338-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lv5sb\" (UID: \"bce77674-28ac-4fcf-8d29-73c7178ed338\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lv5sb" Jul 22 03:04:41.021836 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:41.021809 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x9rv\" (UniqueName: \"kubernetes.io/projected/bce77674-28ac-4fcf-8d29-73c7178ed338-kube-api-access-4x9rv\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lv5sb\" (UID: \"bce77674-28ac-4fcf-8d29-73c7178ed338\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lv5sb" Jul 22 03:04:41.122891 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:41.122862 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bce77674-28ac-4fcf-8d29-73c7178ed338-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lv5sb\" (UID: \"bce77674-28ac-4fcf-8d29-73c7178ed338\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lv5sb" Jul 22 03:04:41.123056 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:41.122904 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x9rv\" (UniqueName: \"kubernetes.io/projected/bce77674-28ac-4fcf-8d29-73c7178ed338-kube-api-access-4x9rv\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lv5sb\" (UID: \"bce77674-28ac-4fcf-8d29-73c7178ed338\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lv5sb" Jul 22 03:04:41.123056 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:41.122938 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bce77674-28ac-4fcf-8d29-73c7178ed338-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lv5sb\" (UID: \"bce77674-28ac-4fcf-8d29-73c7178ed338\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lv5sb" Jul 22 03:04:41.123351 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:41.123335 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bce77674-28ac-4fcf-8d29-73c7178ed338-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lv5sb\" (UID: \"bce77674-28ac-4fcf-8d29-73c7178ed338\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lv5sb" Jul 22 03:04:41.123396 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:41.123374 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bce77674-28ac-4fcf-8d29-73c7178ed338-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lv5sb\" (UID: \"bce77674-28ac-4fcf-8d29-73c7178ed338\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lv5sb" Jul 22 03:04:41.132073 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:41.132050 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x9rv\" (UniqueName: \"kubernetes.io/projected/bce77674-28ac-4fcf-8d29-73c7178ed338-kube-api-access-4x9rv\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lv5sb\" (UID: \"bce77674-28ac-4fcf-8d29-73c7178ed338\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lv5sb" Jul 22 03:04:41.218393 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:41.218362 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lv5sb" Jul 22 03:04:41.351421 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:41.351392 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lv5sb"] Jul 22 03:04:41.375213 ip-10-0-131-67 kubenswrapper[2582]: W0722 03:04:41.375184 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbce77674_28ac_4fcf_8d29_73c7178ed338.slice/crio-de133c676f0fb2cf609010a476c84c650b4756a00da078466c5f59f281c753eb WatchSource:0}: Error finding container de133c676f0fb2cf609010a476c84c650b4756a00da078466c5f59f281c753eb: Status 404 returned error can't find the container with id de133c676f0fb2cf609010a476c84c650b4756a00da078466c5f59f281c753eb Jul 22 03:04:41.824802 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:41.824776 2582 generic.go:356] "Generic (PLEG): container finished" podID="bce77674-28ac-4fcf-8d29-73c7178ed338" containerID="13d4e8dff0b181d31bf4eb0146fdceb6a8124b809892b2539b0bf61ad4775845" exitCode=0 Jul 22 03:04:41.825165 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:41.824859 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lv5sb" event={"ID":"bce77674-28ac-4fcf-8d29-73c7178ed338","Type":"ContainerDied","Data":"13d4e8dff0b181d31bf4eb0146fdceb6a8124b809892b2539b0bf61ad4775845"} Jul 22 03:04:41.825165 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:41.824915 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lv5sb" event={"ID":"bce77674-28ac-4fcf-8d29-73c7178ed338","Type":"ContainerStarted","Data":"de133c676f0fb2cf609010a476c84c650b4756a00da078466c5f59f281c753eb"} Jul 22 03:04:41.826515 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:41.826490 2582 generic.go:356] "Generic (PLEG): container finished" podID="90fe6071-e5e0-4797-8252-17607fbc5def" containerID="85d2884f236defe23d3f251381a3d14dfbacad4564dfa4a2a45e3dad9c608a3f" exitCode=0 Jul 22 03:04:41.826639 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:41.826618 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef187sg6" event={"ID":"90fe6071-e5e0-4797-8252-17607fbc5def","Type":"ContainerDied","Data":"85d2884f236defe23d3f251381a3d14dfbacad4564dfa4a2a45e3dad9c608a3f"} Jul 22 03:04:41.826704 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:41.826668 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef187sg6" event={"ID":"90fe6071-e5e0-4797-8252-17607fbc5def","Type":"ContainerStarted","Data":"0530cb7e00574d54eb0cd968778cf7ac318d49bf5a53f2626233f8e850144df5"} Jul 22 03:04:41.828593 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:41.828571 2582 generic.go:356] "Generic (PLEG): container finished" podID="3f608664-0cb4-47f7-9e41-ca87b2abe2cf" containerID="0cd1c08013757ea597fd0e845825c18c0d986df0b87a4c563742e66fcdef6094" exitCode=0 Jul 22 03:04:41.828840 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:41.828599 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kvc54" event={"ID":"3f608664-0cb4-47f7-9e41-ca87b2abe2cf","Type":"ContainerDied","Data":"0cd1c08013757ea597fd0e845825c18c0d986df0b87a4c563742e66fcdef6094"} Jul 22 03:04:41.830698 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:41.830675 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d6k5j" event={"ID":"6f7e9741-e996-4fd5-8363-c54e86d3dd05","Type":"ContainerStarted","Data":"d0f61e92b9c23e2ecc66630a63d5c8e4a74d4ef17dda4b12c355e1e042ddd22a"} Jul 22 03:04:42.838755 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:42.838721 2582 generic.go:356] "Generic (PLEG): container finished" podID="3f608664-0cb4-47f7-9e41-ca87b2abe2cf" containerID="f44889de502bac69e661b9dc97f3f16a0df1a397f2643125cd4d71c19816fcfe" exitCode=0 Jul 22 03:04:42.839152 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:42.838798 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kvc54" event={"ID":"3f608664-0cb4-47f7-9e41-ca87b2abe2cf","Type":"ContainerDied","Data":"f44889de502bac69e661b9dc97f3f16a0df1a397f2643125cd4d71c19816fcfe"} Jul 22 03:04:42.841083 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:42.841054 2582 generic.go:356] "Generic (PLEG): container finished" podID="6f7e9741-e996-4fd5-8363-c54e86d3dd05" containerID="d0f61e92b9c23e2ecc66630a63d5c8e4a74d4ef17dda4b12c355e1e042ddd22a" exitCode=0 Jul 22 03:04:42.841083 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:42.841077 2582 generic.go:356] "Generic (PLEG): container finished" podID="6f7e9741-e996-4fd5-8363-c54e86d3dd05" containerID="70afe35654f61d6ace366313b012e98a9daed432fbd8fc2979a163f2794b9b12" exitCode=0 Jul 22 03:04:42.841270 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:42.841125 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d6k5j" event={"ID":"6f7e9741-e996-4fd5-8363-c54e86d3dd05","Type":"ContainerDied","Data":"d0f61e92b9c23e2ecc66630a63d5c8e4a74d4ef17dda4b12c355e1e042ddd22a"} Jul 22 03:04:42.841270 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:42.841173 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d6k5j" event={"ID":"6f7e9741-e996-4fd5-8363-c54e86d3dd05","Type":"ContainerDied","Data":"70afe35654f61d6ace366313b012e98a9daed432fbd8fc2979a163f2794b9b12"} Jul 22 03:04:42.843158 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:42.843133 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lv5sb" event={"ID":"bce77674-28ac-4fcf-8d29-73c7178ed338","Type":"ContainerStarted","Data":"fb24f729d381cd51c086dfccb3a38b84c59704dd8c1621b139dffe4f426b5268"} Jul 22 03:04:42.845488 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:42.845465 2582 generic.go:356] "Generic (PLEG): container finished" podID="90fe6071-e5e0-4797-8252-17607fbc5def" containerID="04527ebe290150f94a98e8da9d5aef95a51fcb06eaa3a74aa65e06a3f22d19df" exitCode=0 Jul 22 03:04:42.845567 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:42.845529 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef187sg6" event={"ID":"90fe6071-e5e0-4797-8252-17607fbc5def","Type":"ContainerDied","Data":"04527ebe290150f94a98e8da9d5aef95a51fcb06eaa3a74aa65e06a3f22d19df"} Jul 22 03:04:43.856664 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:43.856631 2582 generic.go:356] "Generic (PLEG): container finished" podID="bce77674-28ac-4fcf-8d29-73c7178ed338" containerID="fb24f729d381cd51c086dfccb3a38b84c59704dd8c1621b139dffe4f426b5268" exitCode=0 Jul 22 03:04:43.857088 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:43.856707 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lv5sb" event={"ID":"bce77674-28ac-4fcf-8d29-73c7178ed338","Type":"ContainerDied","Data":"fb24f729d381cd51c086dfccb3a38b84c59704dd8c1621b139dffe4f426b5268"} Jul 22 03:04:43.858997 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:43.858974 2582 generic.go:356] "Generic (PLEG): container finished" podID="90fe6071-e5e0-4797-8252-17607fbc5def" containerID="cfa75bfdec94d538573f5cdf90d08c410d599f1aef9591cf4c1342de1747f1da" exitCode=0 Jul 22 03:04:43.859103 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:43.859039 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef187sg6" event={"ID":"90fe6071-e5e0-4797-8252-17607fbc5def","Type":"ContainerDied","Data":"cfa75bfdec94d538573f5cdf90d08c410d599f1aef9591cf4c1342de1747f1da"} Jul 22 03:04:44.004773 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:44.004747 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d6k5j" Jul 22 03:04:44.040141 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:44.040116 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kvc54" Jul 22 03:04:44.146282 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:44.146250 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k96zn\" (UniqueName: \"kubernetes.io/projected/3f608664-0cb4-47f7-9e41-ca87b2abe2cf-kube-api-access-k96zn\") pod \"3f608664-0cb4-47f7-9e41-ca87b2abe2cf\" (UID: \"3f608664-0cb4-47f7-9e41-ca87b2abe2cf\") " Jul 22 03:04:44.146282 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:44.146283 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8mrtl\" (UniqueName: \"kubernetes.io/projected/6f7e9741-e996-4fd5-8363-c54e86d3dd05-kube-api-access-8mrtl\") pod \"6f7e9741-e996-4fd5-8363-c54e86d3dd05\" (UID: \"6f7e9741-e996-4fd5-8363-c54e86d3dd05\") " Jul 22 03:04:44.146522 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:44.146327 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3f608664-0cb4-47f7-9e41-ca87b2abe2cf-util\") pod \"3f608664-0cb4-47f7-9e41-ca87b2abe2cf\" (UID: \"3f608664-0cb4-47f7-9e41-ca87b2abe2cf\") " Jul 22 03:04:44.146522 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:44.146370 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f7e9741-e996-4fd5-8363-c54e86d3dd05-bundle\") pod \"6f7e9741-e996-4fd5-8363-c54e86d3dd05\" (UID: \"6f7e9741-e996-4fd5-8363-c54e86d3dd05\") " Jul 22 03:04:44.146522 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:44.146393 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3f608664-0cb4-47f7-9e41-ca87b2abe2cf-bundle\") pod \"3f608664-0cb4-47f7-9e41-ca87b2abe2cf\" (UID: \"3f608664-0cb4-47f7-9e41-ca87b2abe2cf\") " Jul 22 03:04:44.146522 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:44.146425 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f7e9741-e996-4fd5-8363-c54e86d3dd05-util\") pod \"6f7e9741-e996-4fd5-8363-c54e86d3dd05\" (UID: \"6f7e9741-e996-4fd5-8363-c54e86d3dd05\") " Jul 22 03:04:44.146870 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:44.146843 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f7e9741-e996-4fd5-8363-c54e86d3dd05-bundle" (OuterVolumeSpecName: "bundle") pod "6f7e9741-e996-4fd5-8363-c54e86d3dd05" (UID: "6f7e9741-e996-4fd5-8363-c54e86d3dd05"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:04:44.146984 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:44.146952 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f608664-0cb4-47f7-9e41-ca87b2abe2cf-bundle" (OuterVolumeSpecName: "bundle") pod "3f608664-0cb4-47f7-9e41-ca87b2abe2cf" (UID: "3f608664-0cb4-47f7-9e41-ca87b2abe2cf"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:04:44.148516 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:44.148491 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f7e9741-e996-4fd5-8363-c54e86d3dd05-kube-api-access-8mrtl" (OuterVolumeSpecName: "kube-api-access-8mrtl") pod "6f7e9741-e996-4fd5-8363-c54e86d3dd05" (UID: "6f7e9741-e996-4fd5-8363-c54e86d3dd05"). InnerVolumeSpecName "kube-api-access-8mrtl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 03:04:44.148623 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:44.148520 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f608664-0cb4-47f7-9e41-ca87b2abe2cf-kube-api-access-k96zn" (OuterVolumeSpecName: "kube-api-access-k96zn") pod "3f608664-0cb4-47f7-9e41-ca87b2abe2cf" (UID: "3f608664-0cb4-47f7-9e41-ca87b2abe2cf"). InnerVolumeSpecName "kube-api-access-k96zn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 03:04:44.155034 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:44.154994 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f608664-0cb4-47f7-9e41-ca87b2abe2cf-util" (OuterVolumeSpecName: "util") pod "3f608664-0cb4-47f7-9e41-ca87b2abe2cf" (UID: "3f608664-0cb4-47f7-9e41-ca87b2abe2cf"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:04:44.155215 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:44.155194 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f7e9741-e996-4fd5-8363-c54e86d3dd05-util" (OuterVolumeSpecName: "util") pod "6f7e9741-e996-4fd5-8363-c54e86d3dd05" (UID: "6f7e9741-e996-4fd5-8363-c54e86d3dd05"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:04:44.247713 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:44.247680 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k96zn\" (UniqueName: \"kubernetes.io/projected/3f608664-0cb4-47f7-9e41-ca87b2abe2cf-kube-api-access-k96zn\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 03:04:44.247713 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:44.247709 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mrtl\" (UniqueName: \"kubernetes.io/projected/6f7e9741-e996-4fd5-8363-c54e86d3dd05-kube-api-access-8mrtl\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 03:04:44.247713 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:44.247719 2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3f608664-0cb4-47f7-9e41-ca87b2abe2cf-util\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 03:04:44.247929 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:44.247729 2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f7e9741-e996-4fd5-8363-c54e86d3dd05-bundle\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 03:04:44.247929 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:44.247738 2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3f608664-0cb4-47f7-9e41-ca87b2abe2cf-bundle\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 03:04:44.247929 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:44.247745 2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f7e9741-e996-4fd5-8363-c54e86d3dd05-util\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 03:04:44.867169 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:44.867139 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d6k5j" Jul 22 03:04:44.867169 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:44.867163 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d6k5j" event={"ID":"6f7e9741-e996-4fd5-8363-c54e86d3dd05","Type":"ContainerDied","Data":"f2883bd6c7156b60cce6fb81c3b27491015f8f57fd85a94f539074a2b78a4376"} Jul 22 03:04:44.867659 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:44.867211 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2883bd6c7156b60cce6fb81c3b27491015f8f57fd85a94f539074a2b78a4376" Jul 22 03:04:44.869351 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:44.869325 2582 generic.go:356] "Generic (PLEG): container finished" podID="bce77674-28ac-4fcf-8d29-73c7178ed338" containerID="673e08a7f22fcdcf7df89d9b2d709767a7e520500a98f5984d5d45643fb57e6e" exitCode=0 Jul 22 03:04:44.869474 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:44.869397 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lv5sb" event={"ID":"bce77674-28ac-4fcf-8d29-73c7178ed338","Type":"ContainerDied","Data":"673e08a7f22fcdcf7df89d9b2d709767a7e520500a98f5984d5d45643fb57e6e"} Jul 22 03:04:44.871266 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:44.871243 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kvc54" event={"ID":"3f608664-0cb4-47f7-9e41-ca87b2abe2cf","Type":"ContainerDied","Data":"643563aa536b9e247a28728e30ba2fbed717cd247e1dfe6811a18dab89ae2a9a"} Jul 22 03:04:44.871367 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:44.871273 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kvc54" Jul 22 03:04:44.871367 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:44.871286 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="643563aa536b9e247a28728e30ba2fbed717cd247e1dfe6811a18dab89ae2a9a" Jul 22 03:04:45.003713 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:45.003686 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef187sg6" Jul 22 03:04:45.155750 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:45.155721 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/90fe6071-e5e0-4797-8252-17607fbc5def-util\") pod \"90fe6071-e5e0-4797-8252-17607fbc5def\" (UID: \"90fe6071-e5e0-4797-8252-17607fbc5def\") " Jul 22 03:04:45.155917 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:45.155768 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/90fe6071-e5e0-4797-8252-17607fbc5def-bundle\") pod \"90fe6071-e5e0-4797-8252-17607fbc5def\" (UID: \"90fe6071-e5e0-4797-8252-17607fbc5def\") " Jul 22 03:04:45.155917 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:45.155820 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dfhgc\" (UniqueName: \"kubernetes.io/projected/90fe6071-e5e0-4797-8252-17607fbc5def-kube-api-access-dfhgc\") pod \"90fe6071-e5e0-4797-8252-17607fbc5def\" (UID: \"90fe6071-e5e0-4797-8252-17607fbc5def\") " Jul 22 03:04:45.156276 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:45.156253 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90fe6071-e5e0-4797-8252-17607fbc5def-bundle" (OuterVolumeSpecName: "bundle") pod "90fe6071-e5e0-4797-8252-17607fbc5def" (UID: "90fe6071-e5e0-4797-8252-17607fbc5def"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:04:45.157845 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:45.157824 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90fe6071-e5e0-4797-8252-17607fbc5def-kube-api-access-dfhgc" (OuterVolumeSpecName: "kube-api-access-dfhgc") pod "90fe6071-e5e0-4797-8252-17607fbc5def" (UID: "90fe6071-e5e0-4797-8252-17607fbc5def"). InnerVolumeSpecName "kube-api-access-dfhgc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 03:04:45.161332 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:45.161307 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90fe6071-e5e0-4797-8252-17607fbc5def-util" (OuterVolumeSpecName: "util") pod "90fe6071-e5e0-4797-8252-17607fbc5def" (UID: "90fe6071-e5e0-4797-8252-17607fbc5def"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:04:45.256743 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:45.256710 2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/90fe6071-e5e0-4797-8252-17607fbc5def-util\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 03:04:45.256743 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:45.256736 2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/90fe6071-e5e0-4797-8252-17607fbc5def-bundle\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 03:04:45.256743 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:45.256746 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfhgc\" (UniqueName: \"kubernetes.io/projected/90fe6071-e5e0-4797-8252-17607fbc5def-kube-api-access-dfhgc\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 03:04:45.878406 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:45.878374 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef187sg6" Jul 22 03:04:45.878826 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:45.878409 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef187sg6" event={"ID":"90fe6071-e5e0-4797-8252-17607fbc5def","Type":"ContainerDied","Data":"0530cb7e00574d54eb0cd968778cf7ac318d49bf5a53f2626233f8e850144df5"} Jul 22 03:04:45.878826 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:45.878453 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0530cb7e00574d54eb0cd968778cf7ac318d49bf5a53f2626233f8e850144df5" Jul 22 03:04:46.027647 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:46.027620 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lv5sb" Jul 22 03:04:46.165211 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:46.165127 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bce77674-28ac-4fcf-8d29-73c7178ed338-bundle\") pod \"bce77674-28ac-4fcf-8d29-73c7178ed338\" (UID: \"bce77674-28ac-4fcf-8d29-73c7178ed338\") " Jul 22 03:04:46.165359 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:46.165215 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bce77674-28ac-4fcf-8d29-73c7178ed338-util\") pod \"bce77674-28ac-4fcf-8d29-73c7178ed338\" (UID: \"bce77674-28ac-4fcf-8d29-73c7178ed338\") " Jul 22 03:04:46.165359 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:46.165253 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4x9rv\" (UniqueName: \"kubernetes.io/projected/bce77674-28ac-4fcf-8d29-73c7178ed338-kube-api-access-4x9rv\") pod \"bce77674-28ac-4fcf-8d29-73c7178ed338\" (UID: \"bce77674-28ac-4fcf-8d29-73c7178ed338\") " Jul 22 03:04:46.165888 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:46.165855 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bce77674-28ac-4fcf-8d29-73c7178ed338-bundle" (OuterVolumeSpecName: "bundle") pod "bce77674-28ac-4fcf-8d29-73c7178ed338" (UID: "bce77674-28ac-4fcf-8d29-73c7178ed338"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:04:46.167353 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:46.167324 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bce77674-28ac-4fcf-8d29-73c7178ed338-kube-api-access-4x9rv" (OuterVolumeSpecName: "kube-api-access-4x9rv") pod "bce77674-28ac-4fcf-8d29-73c7178ed338" (UID: "bce77674-28ac-4fcf-8d29-73c7178ed338"). InnerVolumeSpecName "kube-api-access-4x9rv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 03:04:46.170851 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:46.170803 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bce77674-28ac-4fcf-8d29-73c7178ed338-util" (OuterVolumeSpecName: "util") pod "bce77674-28ac-4fcf-8d29-73c7178ed338" (UID: "bce77674-28ac-4fcf-8d29-73c7178ed338"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:04:46.266057 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:46.266007 2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bce77674-28ac-4fcf-8d29-73c7178ed338-util\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 03:04:46.266057 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:46.266053 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4x9rv\" (UniqueName: \"kubernetes.io/projected/bce77674-28ac-4fcf-8d29-73c7178ed338-kube-api-access-4x9rv\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 03:04:46.266057 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:46.266065 2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bce77674-28ac-4fcf-8d29-73c7178ed338-bundle\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 03:04:46.885401 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:46.885374 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lv5sb" Jul 22 03:04:46.885768 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:46.885378 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lv5sb" event={"ID":"bce77674-28ac-4fcf-8d29-73c7178ed338","Type":"ContainerDied","Data":"de133c676f0fb2cf609010a476c84c650b4756a00da078466c5f59f281c753eb"} Jul 22 03:04:46.885768 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:46.885484 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de133c676f0fb2cf609010a476c84c650b4756a00da078466c5f59f281c753eb" Jul 22 03:04:48.843666 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:48.843631 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:04:48.922870 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:48.922839 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:04:50.810137 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:50.810101 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:04:52.836482 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:52.836448 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:04:55.976330 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:55.976293 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:04:56.563072 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:56.563037 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:04:57.585696 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:57.585664 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-z7ttp"] Jul 22 03:04:57.586092 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:57.585995 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90fe6071-e5e0-4797-8252-17607fbc5def" containerName="pull" Jul 22 03:04:57.586092 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:57.586005 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="90fe6071-e5e0-4797-8252-17607fbc5def" containerName="pull" Jul 22 03:04:57.586092 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:57.586038 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f7e9741-e996-4fd5-8363-c54e86d3dd05" containerName="extract" Jul 22 03:04:57.586092 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:57.586044 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f7e9741-e996-4fd5-8363-c54e86d3dd05" containerName="extract" Jul 22 03:04:57.586092 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:57.586053 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f608664-0cb4-47f7-9e41-ca87b2abe2cf" containerName="extract" Jul 22 03:04:57.586092 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:57.586058 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f608664-0cb4-47f7-9e41-ca87b2abe2cf" containerName="extract" Jul 22 03:04:57.586092 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:57.586066 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bce77674-28ac-4fcf-8d29-73c7178ed338" containerName="pull" Jul 22 03:04:57.586092 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:57.586071 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="bce77674-28ac-4fcf-8d29-73c7178ed338" containerName="pull" Jul 22 03:04:57.586092 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:57.586077 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bce77674-28ac-4fcf-8d29-73c7178ed338" containerName="extract" Jul 22 03:04:57.586092 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:57.586082 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="bce77674-28ac-4fcf-8d29-73c7178ed338" containerName="extract" Jul 22 03:04:57.586092 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:57.586091 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90fe6071-e5e0-4797-8252-17607fbc5def" containerName="extract" Jul 22 03:04:57.586092 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:57.586095 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="90fe6071-e5e0-4797-8252-17607fbc5def" containerName="extract" Jul 22 03:04:57.586453 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:57.586104 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f7e9741-e996-4fd5-8363-c54e86d3dd05" containerName="util" Jul 22 03:04:57.586453 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:57.586109 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f7e9741-e996-4fd5-8363-c54e86d3dd05" containerName="util" Jul 22 03:04:57.586453 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:57.586117 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90fe6071-e5e0-4797-8252-17607fbc5def" containerName="util" Jul 22 03:04:57.586453 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:57.586122 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="90fe6071-e5e0-4797-8252-17607fbc5def" containerName="util" Jul 22 03:04:57.586453 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:57.586130 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bce77674-28ac-4fcf-8d29-73c7178ed338" containerName="util" Jul 22 03:04:57.586453 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:57.586134 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="bce77674-28ac-4fcf-8d29-73c7178ed338" containerName="util" Jul 22 03:04:57.586453 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:57.586141 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f608664-0cb4-47f7-9e41-ca87b2abe2cf" containerName="pull" Jul 22 03:04:57.586453 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:57.586146 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f608664-0cb4-47f7-9e41-ca87b2abe2cf" containerName="pull" Jul 22 03:04:57.586453 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:57.586153 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f608664-0cb4-47f7-9e41-ca87b2abe2cf" containerName="util" Jul 22 03:04:57.586453 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:57.586158 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f608664-0cb4-47f7-9e41-ca87b2abe2cf" containerName="util" Jul 22 03:04:57.586453 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:57.586164 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f7e9741-e996-4fd5-8363-c54e86d3dd05" containerName="pull" Jul 22 03:04:57.586453 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:57.586168 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f7e9741-e996-4fd5-8363-c54e86d3dd05" containerName="pull" Jul 22 03:04:57.586453 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:57.586232 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f608664-0cb4-47f7-9e41-ca87b2abe2cf" containerName="extract" Jul 22 03:04:57.586453 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:57.586241 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f7e9741-e996-4fd5-8363-c54e86d3dd05" containerName="extract" Jul 22 03:04:57.586453 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:57.586248 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="90fe6071-e5e0-4797-8252-17607fbc5def" containerName="extract" Jul 22 03:04:57.586453 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:57.586254 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="bce77674-28ac-4fcf-8d29-73c7178ed338" containerName="extract" Jul 22 03:04:57.590557 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:57.590535 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-z7ttp" Jul 22 03:04:57.593000 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:57.592983 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-j5crl\"" Jul 22 03:04:57.601800 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:57.601778 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-z7ttp"] Jul 22 03:04:57.667489 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:57.667452 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/17f6a5ea-7498-49de-ab19-12d84afedb14-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-z7ttp\" (UID: \"17f6a5ea-7498-49de-ab19-12d84afedb14\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-z7ttp" Jul 22 03:04:57.667676 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:57.667537 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7dz4\" (UniqueName: \"kubernetes.io/projected/17f6a5ea-7498-49de-ab19-12d84afedb14-kube-api-access-r7dz4\") pod \"kuadrant-operator-controller-manager-8684c9ff49-z7ttp\" (UID: \"17f6a5ea-7498-49de-ab19-12d84afedb14\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-z7ttp" Jul 22 03:04:57.768320 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:57.768285 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7dz4\" (UniqueName: \"kubernetes.io/projected/17f6a5ea-7498-49de-ab19-12d84afedb14-kube-api-access-r7dz4\") pod \"kuadrant-operator-controller-manager-8684c9ff49-z7ttp\" (UID: \"17f6a5ea-7498-49de-ab19-12d84afedb14\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-z7ttp" Jul 22 03:04:57.768485 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:57.768343 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/17f6a5ea-7498-49de-ab19-12d84afedb14-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-z7ttp\" (UID: \"17f6a5ea-7498-49de-ab19-12d84afedb14\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-z7ttp" Jul 22 03:04:57.768662 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:57.768644 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/17f6a5ea-7498-49de-ab19-12d84afedb14-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-z7ttp\" (UID: \"17f6a5ea-7498-49de-ab19-12d84afedb14\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-z7ttp" Jul 22 03:04:57.782356 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:57.782332 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7dz4\" (UniqueName: \"kubernetes.io/projected/17f6a5ea-7498-49de-ab19-12d84afedb14-kube-api-access-r7dz4\") pod \"kuadrant-operator-controller-manager-8684c9ff49-z7ttp\" (UID: \"17f6a5ea-7498-49de-ab19-12d84afedb14\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-z7ttp" Jul 22 03:04:57.900662 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:57.900638 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-z7ttp" Jul 22 03:04:58.041516 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:58.041491 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-z7ttp"] Jul 22 03:04:58.043882 ip-10-0-131-67 kubenswrapper[2582]: W0722 03:04:58.043852 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17f6a5ea_7498_49de_ab19_12d84afedb14.slice/crio-54abddde823b23922ba5e081ae18c3a9a5197927535ae5954c03a5cad4c447aa WatchSource:0}: Error finding container 54abddde823b23922ba5e081ae18c3a9a5197927535ae5954c03a5cad4c447aa: Status 404 returned error can't find the container with id 54abddde823b23922ba5e081ae18c3a9a5197927535ae5954c03a5cad4c447aa Jul 22 03:04:58.227439 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:58.227359 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:04:58.949870 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:58.949831 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-z7ttp" event={"ID":"17f6a5ea-7498-49de-ab19-12d84afedb14","Type":"ContainerStarted","Data":"54abddde823b23922ba5e081ae18c3a9a5197927535ae5954c03a5cad4c447aa"} Jul 22 03:04:59.597496 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:59.597454 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-q4rcm"] Jul 22 03:04:59.599984 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:59.599957 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-q4rcm" Jul 22 03:04:59.602491 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:59.602462 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 22 03:04:59.602619 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:59.602592 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-kqcnj\"" Jul 22 03:04:59.611738 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:59.611712 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-q4rcm"] Jul 22 03:04:59.681140 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:59.681107 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnbbr\" (UniqueName: \"kubernetes.io/projected/941e3af6-8144-4317-bec3-9e26bf4248f4-kube-api-access-dnbbr\") pod \"dns-operator-controller-manager-864dd78c4c-q4rcm\" (UID: \"941e3af6-8144-4317-bec3-9e26bf4248f4\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-q4rcm" Jul 22 03:04:59.782315 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:59.782277 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnbbr\" (UniqueName: \"kubernetes.io/projected/941e3af6-8144-4317-bec3-9e26bf4248f4-kube-api-access-dnbbr\") pod \"dns-operator-controller-manager-864dd78c4c-q4rcm\" (UID: \"941e3af6-8144-4317-bec3-9e26bf4248f4\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-q4rcm" Jul 22 03:04:59.793086 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:59.793056 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnbbr\" (UniqueName: \"kubernetes.io/projected/941e3af6-8144-4317-bec3-9e26bf4248f4-kube-api-access-dnbbr\") pod \"dns-operator-controller-manager-864dd78c4c-q4rcm\" (UID: \"941e3af6-8144-4317-bec3-9e26bf4248f4\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-q4rcm" Jul 22 03:04:59.910728 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:04:59.910695 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-q4rcm" Jul 22 03:05:00.046081 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:00.046056 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-q4rcm"] Jul 22 03:05:00.048724 ip-10-0-131-67 kubenswrapper[2582]: W0722 03:05:00.048696 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod941e3af6_8144_4317_bec3_9e26bf4248f4.slice/crio-fee234cf1fe9fefccee3d31406b696cb3f0faf7ad51db2e83ee2933792236c8c WatchSource:0}: Error finding container fee234cf1fe9fefccee3d31406b696cb3f0faf7ad51db2e83ee2933792236c8c: Status 404 returned error can't find the container with id fee234cf1fe9fefccee3d31406b696cb3f0faf7ad51db2e83ee2933792236c8c Jul 22 03:05:00.075706 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:00.075655 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:05:00.267832 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:00.267675 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:05:00.963528 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:00.963480 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-q4rcm" event={"ID":"941e3af6-8144-4317-bec3-9e26bf4248f4","Type":"ContainerStarted","Data":"fee234cf1fe9fefccee3d31406b696cb3f0faf7ad51db2e83ee2933792236c8c"} Jul 22 03:05:02.365661 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:02.365629 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wxs8w"] Jul 22 03:05:02.368142 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:02.368125 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wxs8w" Jul 22 03:05:02.370646 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:02.370628 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-6js5c\"" Jul 22 03:05:02.378895 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:02.378875 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wxs8w"] Jul 22 03:05:02.401396 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:02.401349 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kcnk\" (UniqueName: \"kubernetes.io/projected/1083b153-64a5-4e38-a869-2fcf759eafda-kube-api-access-6kcnk\") pod \"limitador-operator-controller-manager-859d7b78bd-wxs8w\" (UID: \"1083b153-64a5-4e38-a869-2fcf759eafda\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wxs8w" Jul 22 03:05:02.495484 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:02.495452 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:05:02.501773 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:02.501747 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kcnk\" (UniqueName: \"kubernetes.io/projected/1083b153-64a5-4e38-a869-2fcf759eafda-kube-api-access-6kcnk\") pod \"limitador-operator-controller-manager-859d7b78bd-wxs8w\" (UID: \"1083b153-64a5-4e38-a869-2fcf759eafda\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wxs8w" Jul 22 03:05:02.510245 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:02.510164 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kcnk\" (UniqueName: \"kubernetes.io/projected/1083b153-64a5-4e38-a869-2fcf759eafda-kube-api-access-6kcnk\") pod \"limitador-operator-controller-manager-859d7b78bd-wxs8w\" (UID: \"1083b153-64a5-4e38-a869-2fcf759eafda\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wxs8w" Jul 22 03:05:02.620790 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:02.620757 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:05:02.678472 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:02.678443 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wxs8w" Jul 22 03:05:02.978378 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:02.978333 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-q4rcm" event={"ID":"941e3af6-8144-4317-bec3-9e26bf4248f4","Type":"ContainerStarted","Data":"bc0eae3dfa4100af0c44c1de9d474e57b1dde032c813c0d0db95f09bfc99d63f"} Jul 22 03:05:02.978587 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:02.978560 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-q4rcm" Jul 22 03:05:03.006144 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:03.006078 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-q4rcm" podStartSLOduration=1.802983234 podStartE2EDuration="4.006058072s" totalImagesPullingTime="2.203074838s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:04:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:05:03.004408571 +0000 UTC m=+413.736029444" watchObservedRunningTime="2026-07-22 03:05:03.006058072 +0000 UTC m=+413.737678920" Jul 22 03:05:03.423455 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:03.423411 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:05:03.898316 ip-10-0-131-67 kubenswrapper[2582]: W0722 03:05:03.898286 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1083b153_64a5_4e38_a869_2fcf759eafda.slice/crio-16dad79a52f7a37a0bb4dcc8d2c83bbad3f88cf5c0a893e5f6c4688035998576 WatchSource:0}: Error finding container 16dad79a52f7a37a0bb4dcc8d2c83bbad3f88cf5c0a893e5f6c4688035998576: Status 404 returned error can't find the container with id 16dad79a52f7a37a0bb4dcc8d2c83bbad3f88cf5c0a893e5f6c4688035998576 Jul 22 03:05:03.901765 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:03.901747 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wxs8w"] Jul 22 03:05:03.985397 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:03.985360 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wxs8w" event={"ID":"1083b153-64a5-4e38-a869-2fcf759eafda","Type":"ContainerStarted","Data":"16dad79a52f7a37a0bb4dcc8d2c83bbad3f88cf5c0a893e5f6c4688035998576"} Jul 22 03:05:03.987225 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:03.987191 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-z7ttp" event={"ID":"17f6a5ea-7498-49de-ab19-12d84afedb14","Type":"ContainerStarted","Data":"12c7f2a2fa078817802c35468499243dec76b3c7152cbcc3499bfe36f40cdb04"} Jul 22 03:05:03.987451 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:03.987435 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-z7ttp" Jul 22 03:05:04.007353 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:04.007298 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-z7ttp" podStartSLOduration=1.2573420020000001 podStartE2EDuration="7.007276331s" totalImagesPullingTime="5.749934329s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:04:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:05:04.006194862 +0000 UTC m=+414.737815708" watchObservedRunningTime="2026-07-22 03:05:04.007276331 +0000 UTC m=+414.738897180" Jul 22 03:05:04.938209 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:04.938160 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:05:05.021937 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:05.021897 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:05:05.954611 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:05.954577 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-l458v"] Jul 22 03:05:05.956930 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:05.956915 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-l458v" Jul 22 03:05:05.961656 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:05.961637 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-cdb4w\"" Jul 22 03:05:05.974401 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:05.974373 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-l458v"] Jul 22 03:05:06.000208 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:06.000175 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wxs8w" event={"ID":"1083b153-64a5-4e38-a869-2fcf759eafda","Type":"ContainerStarted","Data":"28bda036bf816d78f371829ab2d6baa2cf36a14ce6fd6121c7fc78df729a11c3"} Jul 22 03:05:06.000350 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:06.000315 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wxs8w" Jul 22 03:05:06.027295 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:06.027200 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmq7q\" (UniqueName: \"kubernetes.io/projected/0577bf79-9074-4ad0-a3f5-5f8bc286831e-kube-api-access-tmq7q\") pod \"authorino-operator-75cc8d5cb7-l458v\" (UID: \"0577bf79-9074-4ad0-a3f5-5f8bc286831e\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-l458v" Jul 22 03:05:06.028479 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:06.028431 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wxs8w" podStartSLOduration=2.55127268 podStartE2EDuration="4.028415709s" totalImagesPullingTime="1.477143029s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:05:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:05:06.026842691 +0000 UTC m=+416.758463548" watchObservedRunningTime="2026-07-22 03:05:06.028415709 +0000 UTC m=+416.760036555" Jul 22 03:05:06.128311 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:06.128271 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmq7q\" (UniqueName: \"kubernetes.io/projected/0577bf79-9074-4ad0-a3f5-5f8bc286831e-kube-api-access-tmq7q\") pod \"authorino-operator-75cc8d5cb7-l458v\" (UID: \"0577bf79-9074-4ad0-a3f5-5f8bc286831e\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-l458v" Jul 22 03:05:06.140242 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:06.140204 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmq7q\" (UniqueName: \"kubernetes.io/projected/0577bf79-9074-4ad0-a3f5-5f8bc286831e-kube-api-access-tmq7q\") pod \"authorino-operator-75cc8d5cb7-l458v\" (UID: \"0577bf79-9074-4ad0-a3f5-5f8bc286831e\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-l458v" Jul 22 03:05:06.266719 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:06.266635 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-l458v" Jul 22 03:05:06.399789 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:06.399445 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-l458v"] Jul 22 03:05:06.402500 ip-10-0-131-67 kubenswrapper[2582]: W0722 03:05:06.402471 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0577bf79_9074_4ad0_a3f5_5f8bc286831e.slice/crio-6ce5e19257241d5835dd7c66bfd77527ea6b3bccd20634e6a97c58221f4bc26b WatchSource:0}: Error finding container 6ce5e19257241d5835dd7c66bfd77527ea6b3bccd20634e6a97c58221f4bc26b: Status 404 returned error can't find the container with id 6ce5e19257241d5835dd7c66bfd77527ea6b3bccd20634e6a97c58221f4bc26b Jul 22 03:05:06.871572 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:06.871539 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-dvrw9"] Jul 22 03:05:06.874765 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:06.874749 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dvrw9" Jul 22 03:05:06.876996 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:06.876972 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 22 03:05:06.877163 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:06.877003 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-mp52x\"" Jul 22 03:05:06.877284 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:06.877266 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 22 03:05:06.888367 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:06.888338 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:05:06.888862 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:06.888841 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-dvrw9"] Jul 22 03:05:06.934369 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:06.934343 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/32f52f23-526c-4e4b-a988-9d69c264dca9-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-dvrw9\" (UID: \"32f52f23-526c-4e4b-a988-9d69c264dca9\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dvrw9" Jul 22 03:05:06.934506 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:06.934421 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/32f52f23-526c-4e4b-a988-9d69c264dca9-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-dvrw9\" (UID: \"32f52f23-526c-4e4b-a988-9d69c264dca9\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dvrw9" Jul 22 03:05:06.934506 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:06.934463 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djtbn\" (UniqueName: \"kubernetes.io/projected/32f52f23-526c-4e4b-a988-9d69c264dca9-kube-api-access-djtbn\") pod \"kuadrant-console-plugin-66b5774985-dvrw9\" (UID: \"32f52f23-526c-4e4b-a988-9d69c264dca9\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dvrw9" Jul 22 03:05:07.016572 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:07.016504 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-l458v" event={"ID":"0577bf79-9074-4ad0-a3f5-5f8bc286831e","Type":"ContainerStarted","Data":"6ce5e19257241d5835dd7c66bfd77527ea6b3bccd20634e6a97c58221f4bc26b"} Jul 22 03:05:07.029529 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:07.029484 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:05:07.035713 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:07.035682 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/32f52f23-526c-4e4b-a988-9d69c264dca9-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-dvrw9\" (UID: \"32f52f23-526c-4e4b-a988-9d69c264dca9\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dvrw9" Jul 22 03:05:07.035891 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:07.035751 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/32f52f23-526c-4e4b-a988-9d69c264dca9-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-dvrw9\" (UID: \"32f52f23-526c-4e4b-a988-9d69c264dca9\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dvrw9" Jul 22 03:05:07.035891 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:07.035799 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djtbn\" (UniqueName: \"kubernetes.io/projected/32f52f23-526c-4e4b-a988-9d69c264dca9-kube-api-access-djtbn\") pod \"kuadrant-console-plugin-66b5774985-dvrw9\" (UID: \"32f52f23-526c-4e4b-a988-9d69c264dca9\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dvrw9" Jul 22 03:05:07.036641 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:07.036613 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/32f52f23-526c-4e4b-a988-9d69c264dca9-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-dvrw9\" (UID: \"32f52f23-526c-4e4b-a988-9d69c264dca9\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dvrw9" Jul 22 03:05:07.038406 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:07.038385 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/32f52f23-526c-4e4b-a988-9d69c264dca9-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-dvrw9\" (UID: \"32f52f23-526c-4e4b-a988-9d69c264dca9\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dvrw9" Jul 22 03:05:07.051787 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:07.051758 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djtbn\" (UniqueName: \"kubernetes.io/projected/32f52f23-526c-4e4b-a988-9d69c264dca9-kube-api-access-djtbn\") pod \"kuadrant-console-plugin-66b5774985-dvrw9\" (UID: \"32f52f23-526c-4e4b-a988-9d69c264dca9\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dvrw9" Jul 22 03:05:07.185381 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:07.185341 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dvrw9" Jul 22 03:05:07.358995 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:07.358918 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-dvrw9"] Jul 22 03:05:07.360250 ip-10-0-131-67 kubenswrapper[2582]: W0722 03:05:07.360218 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod32f52f23_526c_4e4b_a988_9d69c264dca9.slice/crio-5c339d7f2fcd2b58035ff7d0c8b968a14c943cef53d5cdac5c5f7f3b4604bb2f WatchSource:0}: Error finding container 5c339d7f2fcd2b58035ff7d0c8b968a14c943cef53d5cdac5c5f7f3b4604bb2f: Status 404 returned error can't find the container with id 5c339d7f2fcd2b58035ff7d0c8b968a14c943cef53d5cdac5c5f7f3b4604bb2f Jul 22 03:05:08.026447 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:08.026407 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dvrw9" event={"ID":"32f52f23-526c-4e4b-a988-9d69c264dca9","Type":"ContainerStarted","Data":"5c339d7f2fcd2b58035ff7d0c8b968a14c943cef53d5cdac5c5f7f3b4604bb2f"} Jul 22 03:05:08.073998 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:08.073964 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:05:08.174797 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:08.174763 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:05:08.243701 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:08.243668 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:05:09.033983 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:09.033895 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-l458v" event={"ID":"0577bf79-9074-4ad0-a3f5-5f8bc286831e","Type":"ContainerStarted","Data":"c1731df7988bbe5d8db569e3e2fdbcce34c0a2e0959a69e03852acef50eab3bd"} Jul 22 03:05:09.034432 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:09.034047 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-l458v" Jul 22 03:05:09.054920 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:09.054798 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-l458v" podStartSLOduration=1.796987828 podStartE2EDuration="4.054778578s" totalImagesPullingTime="2.25779075s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:05:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:05:09.052491463 +0000 UTC m=+419.784112311" watchObservedRunningTime="2026-07-22 03:05:09.054778578 +0000 UTC m=+419.786399428" Jul 22 03:05:13.990060 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:13.989996 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-q4rcm" Jul 22 03:05:14.996177 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:14.996145 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-z7ttp" Jul 22 03:05:15.504583 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:15.504550 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:05:15.605414 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:15.605379 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:05:16.937344 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:16.937310 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-z7ttp"] Jul 22 03:05:16.937797 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:16.937522 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-z7ttp" podUID="17f6a5ea-7498-49de-ab19-12d84afedb14" containerName="manager" containerID="cri-o://12c7f2a2fa078817802c35468499243dec76b3c7152cbcc3499bfe36f40cdb04" gracePeriod=2 Jul 22 03:05:16.946611 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:16.946535 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-z7ttp"] Jul 22 03:05:16.970865 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:16.970818 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rdltl"] Jul 22 03:05:16.971467 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:16.971442 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17f6a5ea-7498-49de-ab19-12d84afedb14" containerName="manager" Jul 22 03:05:16.971652 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:16.971631 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="17f6a5ea-7498-49de-ab19-12d84afedb14" containerName="manager" Jul 22 03:05:16.971812 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:16.971794 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="17f6a5ea-7498-49de-ab19-12d84afedb14" containerName="manager" Jul 22 03:05:16.975246 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:16.975224 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rdltl" Jul 22 03:05:16.977432 ip-10-0-131-67 kubenswrapper[2582]: E0722 03:05:16.977391 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-z7ttp\" is forbidden: User \"system:node:ip-10-0-131-67.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-67.ec2.internal' and this object" podUID="17f6a5ea-7498-49de-ab19-12d84afedb14" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-z7ttp" Jul 22 03:05:16.990641 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:16.990185 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wxs8w"] Jul 22 03:05:16.990641 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:16.990428 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wxs8w" podUID="1083b153-64a5-4e38-a869-2fcf759eafda" containerName="manager" containerID="cri-o://28bda036bf816d78f371829ab2d6baa2cf36a14ce6fd6121c7fc78df729a11c3" gracePeriod=2 Jul 22 03:05:16.992854 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:16.992818 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wxs8w" Jul 22 03:05:16.994004 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:16.993555 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rdltl"] Jul 22 03:05:17.003157 ip-10-0-131-67 kubenswrapper[2582]: E0722 03:05:17.003097 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-z7ttp\" is forbidden: User \"system:node:ip-10-0-131-67.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-67.ec2.internal' and this object" podUID="17f6a5ea-7498-49de-ab19-12d84afedb14" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-z7ttp" Jul 22 03:05:17.003328 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:17.003273 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wxs8w"] Jul 22 03:05:17.005292 ip-10-0-131-67 kubenswrapper[2582]: E0722 03:05:17.005256 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-wxs8w\" is forbidden: User \"system:node:ip-10-0-131-67.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-67.ec2.internal' and this object" podUID="1083b153-64a5-4e38-a869-2fcf759eafda" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wxs8w" Jul 22 03:05:17.028691 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:17.028536 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cb9c3e6d-7da0-421f-82ef-e085f5e7e012-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-rdltl\" (UID: \"cb9c3e6d-7da0-421f-82ef-e085f5e7e012\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rdltl" Jul 22 03:05:17.028691 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:17.028597 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvj6s\" (UniqueName: \"kubernetes.io/projected/cb9c3e6d-7da0-421f-82ef-e085f5e7e012-kube-api-access-dvj6s\") pod \"kuadrant-operator-controller-manager-8684c9ff49-rdltl\" (UID: \"cb9c3e6d-7da0-421f-82ef-e085f5e7e012\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rdltl" Jul 22 03:05:17.050005 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:17.049974 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9mb6v"] Jul 22 03:05:17.050653 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:17.050619 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1083b153-64a5-4e38-a869-2fcf759eafda" containerName="manager" Jul 22 03:05:17.050653 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:17.050647 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1083b153-64a5-4e38-a869-2fcf759eafda" containerName="manager" Jul 22 03:05:17.050839 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:17.050770 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1083b153-64a5-4e38-a869-2fcf759eafda" containerName="manager" Jul 22 03:05:17.054090 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:17.054071 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9mb6v" Jul 22 03:05:17.058275 ip-10-0-131-67 kubenswrapper[2582]: E0722 03:05:17.058237 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-wxs8w\" is forbidden: User \"system:node:ip-10-0-131-67.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-67.ec2.internal' and this object" podUID="1083b153-64a5-4e38-a869-2fcf759eafda" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wxs8w" Jul 22 03:05:17.060233 ip-10-0-131-67 kubenswrapper[2582]: E0722 03:05:17.060201 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-z7ttp\" is forbidden: User \"system:node:ip-10-0-131-67.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-67.ec2.internal' and this object" podUID="17f6a5ea-7498-49de-ab19-12d84afedb14" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-z7ttp" Jul 22 03:05:17.067141 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:17.067116 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9mb6v"] Jul 22 03:05:17.130104 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:17.130067 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cb9c3e6d-7da0-421f-82ef-e085f5e7e012-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-rdltl\" (UID: \"cb9c3e6d-7da0-421f-82ef-e085f5e7e012\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rdltl" Jul 22 03:05:17.130290 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:17.130120 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrb99\" (UniqueName: \"kubernetes.io/projected/36a3a681-1af5-4069-bb00-34d6ccdf9842-kube-api-access-qrb99\") pod \"limitador-operator-controller-manager-859d7b78bd-9mb6v\" (UID: \"36a3a681-1af5-4069-bb00-34d6ccdf9842\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9mb6v" Jul 22 03:05:17.130290 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:17.130159 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvj6s\" (UniqueName: \"kubernetes.io/projected/cb9c3e6d-7da0-421f-82ef-e085f5e7e012-kube-api-access-dvj6s\") pod \"kuadrant-operator-controller-manager-8684c9ff49-rdltl\" (UID: \"cb9c3e6d-7da0-421f-82ef-e085f5e7e012\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rdltl" Jul 22 03:05:17.130545 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:17.130521 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cb9c3e6d-7da0-421f-82ef-e085f5e7e012-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-rdltl\" (UID: \"cb9c3e6d-7da0-421f-82ef-e085f5e7e012\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rdltl" Jul 22 03:05:17.142726 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:17.142698 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvj6s\" (UniqueName: \"kubernetes.io/projected/cb9c3e6d-7da0-421f-82ef-e085f5e7e012-kube-api-access-dvj6s\") pod \"kuadrant-operator-controller-manager-8684c9ff49-rdltl\" (UID: \"cb9c3e6d-7da0-421f-82ef-e085f5e7e012\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rdltl" Jul 22 03:05:17.231541 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:17.231453 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrb99\" (UniqueName: \"kubernetes.io/projected/36a3a681-1af5-4069-bb00-34d6ccdf9842-kube-api-access-qrb99\") pod \"limitador-operator-controller-manager-859d7b78bd-9mb6v\" (UID: \"36a3a681-1af5-4069-bb00-34d6ccdf9842\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9mb6v" Jul 22 03:05:17.240977 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:17.240943 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrb99\" (UniqueName: \"kubernetes.io/projected/36a3a681-1af5-4069-bb00-34d6ccdf9842-kube-api-access-qrb99\") pod \"limitador-operator-controller-manager-859d7b78bd-9mb6v\" (UID: \"36a3a681-1af5-4069-bb00-34d6ccdf9842\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9mb6v" Jul 22 03:05:17.361021 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:17.360985 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rdltl" Jul 22 03:05:17.369405 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:17.369376 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9mb6v" Jul 22 03:05:20.044265 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:20.044234 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-l458v" Jul 22 03:05:20.273720 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:20.273651 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:05:20.376156 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:20.376119 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:05:25.042060 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:25.042023 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:05:25.144755 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:25.144725 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:05:29.044732 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:29.044681 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:05:29.116679 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:29.116646 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:05:30.679688 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:30.679606 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-z7ttp" Jul 22 03:05:30.682205 ip-10-0-131-67 kubenswrapper[2582]: E0722 03:05:30.682164 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-z7ttp\" is forbidden: User \"system:node:ip-10-0-131-67.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-67.ec2.internal' and this object" podUID="17f6a5ea-7498-49de-ab19-12d84afedb14" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-z7ttp" Jul 22 03:05:30.696470 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:30.696445 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9mb6v"] Jul 22 03:05:30.702319 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:30.702297 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wxs8w" Jul 22 03:05:30.704300 ip-10-0-131-67 kubenswrapper[2582]: E0722 03:05:30.704272 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-z7ttp\" is forbidden: User \"system:node:ip-10-0-131-67.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-67.ec2.internal' and this object" podUID="17f6a5ea-7498-49de-ab19-12d84afedb14" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-z7ttp" Jul 22 03:05:30.706165 ip-10-0-131-67 kubenswrapper[2582]: E0722 03:05:30.706134 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-wxs8w\" is forbidden: User \"system:node:ip-10-0-131-67.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-67.ec2.internal' and this object" podUID="1083b153-64a5-4e38-a869-2fcf759eafda" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wxs8w" Jul 22 03:05:30.729080 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:30.729053 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rdltl"] Jul 22 03:05:30.731678 ip-10-0-131-67 kubenswrapper[2582]: W0722 03:05:30.731653 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb9c3e6d_7da0_421f_82ef_e085f5e7e012.slice/crio-627d1ab80df045c70f0b337dd4974d2a1c4a21ecd69123faf6ac3144b87811d0 WatchSource:0}: Error finding container 627d1ab80df045c70f0b337dd4974d2a1c4a21ecd69123faf6ac3144b87811d0: Status 404 returned error can't find the container with id 627d1ab80df045c70f0b337dd4974d2a1c4a21ecd69123faf6ac3144b87811d0 Jul 22 03:05:30.765321 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:30.765294 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/17f6a5ea-7498-49de-ab19-12d84afedb14-extensions-socket-volume\") pod \"17f6a5ea-7498-49de-ab19-12d84afedb14\" (UID: \"17f6a5ea-7498-49de-ab19-12d84afedb14\") " Jul 22 03:05:30.765462 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:30.765424 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r7dz4\" (UniqueName: \"kubernetes.io/projected/17f6a5ea-7498-49de-ab19-12d84afedb14-kube-api-access-r7dz4\") pod \"17f6a5ea-7498-49de-ab19-12d84afedb14\" (UID: \"17f6a5ea-7498-49de-ab19-12d84afedb14\") " Jul 22 03:05:30.765775 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:30.765753 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17f6a5ea-7498-49de-ab19-12d84afedb14-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "17f6a5ea-7498-49de-ab19-12d84afedb14" (UID: "17f6a5ea-7498-49de-ab19-12d84afedb14"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:05:30.767641 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:30.767618 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17f6a5ea-7498-49de-ab19-12d84afedb14-kube-api-access-r7dz4" (OuterVolumeSpecName: "kube-api-access-r7dz4") pod "17f6a5ea-7498-49de-ab19-12d84afedb14" (UID: "17f6a5ea-7498-49de-ab19-12d84afedb14"). InnerVolumeSpecName "kube-api-access-r7dz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 03:05:30.865918 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:30.865888 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6kcnk\" (UniqueName: \"kubernetes.io/projected/1083b153-64a5-4e38-a869-2fcf759eafda-kube-api-access-6kcnk\") pod \"1083b153-64a5-4e38-a869-2fcf759eafda\" (UID: \"1083b153-64a5-4e38-a869-2fcf759eafda\") " Jul 22 03:05:30.866112 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:30.866087 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7dz4\" (UniqueName: \"kubernetes.io/projected/17f6a5ea-7498-49de-ab19-12d84afedb14-kube-api-access-r7dz4\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 03:05:30.866112 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:30.866098 2582 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/17f6a5ea-7498-49de-ab19-12d84afedb14-extensions-socket-volume\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 03:05:30.867990 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:30.867954 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1083b153-64a5-4e38-a869-2fcf759eafda-kube-api-access-6kcnk" (OuterVolumeSpecName: "kube-api-access-6kcnk") pod "1083b153-64a5-4e38-a869-2fcf759eafda" (UID: "1083b153-64a5-4e38-a869-2fcf759eafda"). InnerVolumeSpecName "kube-api-access-6kcnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 03:05:30.966887 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:30.966851 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kcnk\" (UniqueName: \"kubernetes.io/projected/1083b153-64a5-4e38-a869-2fcf759eafda-kube-api-access-6kcnk\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 03:05:31.181305 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:31.181219 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9mb6v" event={"ID":"36a3a681-1af5-4069-bb00-34d6ccdf9842","Type":"ContainerStarted","Data":"d01970a81a293666c128b408c016c759ab40212737abfb4db1637c28e9e94bdc"} Jul 22 03:05:31.181305 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:31.181272 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9mb6v" event={"ID":"36a3a681-1af5-4069-bb00-34d6ccdf9842","Type":"ContainerStarted","Data":"fe74735e9ed3dcf1941a4074958b0449088cd8429d45a0442915a94762d8c13e"} Jul 22 03:05:31.181510 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:31.181347 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9mb6v" Jul 22 03:05:31.182909 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:31.182885 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dvrw9" event={"ID":"32f52f23-526c-4e4b-a988-9d69c264dca9","Type":"ContainerStarted","Data":"271d29997bc26c73985bfefcfc8dc1771f839c9ff10560a3270d5e7229d99596"} Jul 22 03:05:31.183737 ip-10-0-131-67 kubenswrapper[2582]: E0722 03:05:31.183713 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-z7ttp\" is forbidden: User \"system:node:ip-10-0-131-67.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-67.ec2.internal' and this object" podUID="17f6a5ea-7498-49de-ab19-12d84afedb14" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-z7ttp" Jul 22 03:05:31.184177 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:31.184159 2582 generic.go:356] "Generic (PLEG): container finished" podID="1083b153-64a5-4e38-a869-2fcf759eafda" containerID="28bda036bf816d78f371829ab2d6baa2cf36a14ce6fd6121c7fc78df729a11c3" exitCode=0 Jul 22 03:05:31.184254 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:31.184190 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wxs8w" Jul 22 03:05:31.184312 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:31.184261 2582 scope.go:117] "RemoveContainer" containerID="28bda036bf816d78f371829ab2d6baa2cf36a14ce6fd6121c7fc78df729a11c3" Jul 22 03:05:31.185631 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:31.185606 2582 generic.go:356] "Generic (PLEG): container finished" podID="17f6a5ea-7498-49de-ab19-12d84afedb14" containerID="12c7f2a2fa078817802c35468499243dec76b3c7152cbcc3499bfe36f40cdb04" exitCode=0 Jul 22 03:05:31.185748 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:31.185647 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-z7ttp" Jul 22 03:05:31.187548 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:31.187469 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rdltl" event={"ID":"cb9c3e6d-7da0-421f-82ef-e085f5e7e012","Type":"ContainerStarted","Data":"df3838bcfc2beaf2b9ea9274e350bc1295b7a9b39879c07b8857e775c2fe11af"} Jul 22 03:05:31.187548 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:31.187514 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rdltl" event={"ID":"cb9c3e6d-7da0-421f-82ef-e085f5e7e012","Type":"ContainerStarted","Data":"627d1ab80df045c70f0b337dd4974d2a1c4a21ecd69123faf6ac3144b87811d0"} Jul 22 03:05:31.187721 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:31.187599 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rdltl" Jul 22 03:05:31.198101 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:31.198085 2582 scope.go:117] "RemoveContainer" containerID="28bda036bf816d78f371829ab2d6baa2cf36a14ce6fd6121c7fc78df729a11c3" Jul 22 03:05:31.198366 ip-10-0-131-67 kubenswrapper[2582]: E0722 03:05:31.198346 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28bda036bf816d78f371829ab2d6baa2cf36a14ce6fd6121c7fc78df729a11c3\": container with ID starting with 28bda036bf816d78f371829ab2d6baa2cf36a14ce6fd6121c7fc78df729a11c3 not found: ID does not exist" containerID="28bda036bf816d78f371829ab2d6baa2cf36a14ce6fd6121c7fc78df729a11c3" Jul 22 03:05:31.198460 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:31.198373 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28bda036bf816d78f371829ab2d6baa2cf36a14ce6fd6121c7fc78df729a11c3"} err="failed to get container status \"28bda036bf816d78f371829ab2d6baa2cf36a14ce6fd6121c7fc78df729a11c3\": rpc error: code = NotFound desc = could not find container \"28bda036bf816d78f371829ab2d6baa2cf36a14ce6fd6121c7fc78df729a11c3\": container with ID starting with 28bda036bf816d78f371829ab2d6baa2cf36a14ce6fd6121c7fc78df729a11c3 not found: ID does not exist" Jul 22 03:05:31.198460 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:31.198392 2582 scope.go:117] "RemoveContainer" containerID="12c7f2a2fa078817802c35468499243dec76b3c7152cbcc3499bfe36f40cdb04" Jul 22 03:05:31.201533 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:31.201486 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9mb6v" podStartSLOduration=15.201469897 podStartE2EDuration="15.201469897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:05:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:05:31.200516445 +0000 UTC m=+441.932137313" watchObservedRunningTime="2026-07-22 03:05:31.201469897 +0000 UTC m=+441.933090745" Jul 22 03:05:31.202567 ip-10-0-131-67 kubenswrapper[2582]: E0722 03:05:31.202542 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-wxs8w\" is forbidden: User \"system:node:ip-10-0-131-67.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-67.ec2.internal' and this object" podUID="1083b153-64a5-4e38-a869-2fcf759eafda" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wxs8w" Jul 22 03:05:31.208483 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:31.208462 2582 scope.go:117] "RemoveContainer" containerID="12c7f2a2fa078817802c35468499243dec76b3c7152cbcc3499bfe36f40cdb04" Jul 22 03:05:31.208775 ip-10-0-131-67 kubenswrapper[2582]: E0722 03:05:31.208755 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12c7f2a2fa078817802c35468499243dec76b3c7152cbcc3499bfe36f40cdb04\": container with ID starting with 12c7f2a2fa078817802c35468499243dec76b3c7152cbcc3499bfe36f40cdb04 not found: ID does not exist" containerID="12c7f2a2fa078817802c35468499243dec76b3c7152cbcc3499bfe36f40cdb04" Jul 22 03:05:31.208882 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:31.208785 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12c7f2a2fa078817802c35468499243dec76b3c7152cbcc3499bfe36f40cdb04"} err="failed to get container status \"12c7f2a2fa078817802c35468499243dec76b3c7152cbcc3499bfe36f40cdb04\": rpc error: code = NotFound desc = could not find container \"12c7f2a2fa078817802c35468499243dec76b3c7152cbcc3499bfe36f40cdb04\": container with ID starting with 12c7f2a2fa078817802c35468499243dec76b3c7152cbcc3499bfe36f40cdb04 not found: ID does not exist" Jul 22 03:05:31.225292 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:31.225244 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rdltl" podStartSLOduration=15.225228381 podStartE2EDuration="15.225228381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:05:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:05:31.223384826 +0000 UTC m=+441.955005720" watchObservedRunningTime="2026-07-22 03:05:31.225228381 +0000 UTC m=+441.956849227" Jul 22 03:05:31.225609 ip-10-0-131-67 kubenswrapper[2582]: E0722 03:05:31.225578 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-wxs8w\" is forbidden: User \"system:node:ip-10-0-131-67.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-67.ec2.internal' and this object" podUID="1083b153-64a5-4e38-a869-2fcf759eafda" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wxs8w" Jul 22 03:05:31.247333 ip-10-0-131-67 kubenswrapper[2582]: E0722 03:05:31.247298 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-z7ttp\" is forbidden: User \"system:node:ip-10-0-131-67.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-67.ec2.internal' and this object" podUID="17f6a5ea-7498-49de-ab19-12d84afedb14" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-z7ttp" Jul 22 03:05:31.247518 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:31.247480 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dvrw9" podStartSLOduration=2.010031953 podStartE2EDuration="25.247464374s" totalImagesPullingTime="23.237432421s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:05:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:05:31.245313737 +0000 UTC m=+441.976934604" watchObservedRunningTime="2026-07-22 03:05:31.247464374 +0000 UTC m=+441.979085221" Jul 22 03:05:31.902295 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:31.902257 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1083b153-64a5-4e38-a869-2fcf759eafda" path="/var/lib/kubelet/pods/1083b153-64a5-4e38-a869-2fcf759eafda/volumes" Jul 22 03:05:31.902769 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:31.902740 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17f6a5ea-7498-49de-ab19-12d84afedb14" path="/var/lib/kubelet/pods/17f6a5ea-7498-49de-ab19-12d84afedb14/volumes" Jul 22 03:05:33.534609 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:33.534560 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rdltl"] Jul 22 03:05:33.535108 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:33.534787 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rdltl" podUID="cb9c3e6d-7da0-421f-82ef-e085f5e7e012" containerName="manager" containerID="cri-o://df3838bcfc2beaf2b9ea9274e350bc1295b7a9b39879c07b8857e775c2fe11af" gracePeriod=10 Jul 22 03:05:33.790861 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:33.790796 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rdltl" Jul 22 03:05:33.894589 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:33.894551 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dvj6s\" (UniqueName: \"kubernetes.io/projected/cb9c3e6d-7da0-421f-82ef-e085f5e7e012-kube-api-access-dvj6s\") pod \"cb9c3e6d-7da0-421f-82ef-e085f5e7e012\" (UID: \"cb9c3e6d-7da0-421f-82ef-e085f5e7e012\") " Jul 22 03:05:33.894782 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:33.894602 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cb9c3e6d-7da0-421f-82ef-e085f5e7e012-extensions-socket-volume\") pod \"cb9c3e6d-7da0-421f-82ef-e085f5e7e012\" (UID: \"cb9c3e6d-7da0-421f-82ef-e085f5e7e012\") " Jul 22 03:05:33.894999 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:33.894971 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb9c3e6d-7da0-421f-82ef-e085f5e7e012-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "cb9c3e6d-7da0-421f-82ef-e085f5e7e012" (UID: "cb9c3e6d-7da0-421f-82ef-e085f5e7e012"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:05:33.896755 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:33.896726 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb9c3e6d-7da0-421f-82ef-e085f5e7e012-kube-api-access-dvj6s" (OuterVolumeSpecName: "kube-api-access-dvj6s") pod "cb9c3e6d-7da0-421f-82ef-e085f5e7e012" (UID: "cb9c3e6d-7da0-421f-82ef-e085f5e7e012"). InnerVolumeSpecName "kube-api-access-dvj6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 03:05:33.996282 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:33.996244 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvj6s\" (UniqueName: \"kubernetes.io/projected/cb9c3e6d-7da0-421f-82ef-e085f5e7e012-kube-api-access-dvj6s\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 03:05:33.996282 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:33.996273 2582 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cb9c3e6d-7da0-421f-82ef-e085f5e7e012-extensions-socket-volume\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 03:05:34.207198 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:34.207161 2582 generic.go:356] "Generic (PLEG): container finished" podID="cb9c3e6d-7da0-421f-82ef-e085f5e7e012" containerID="df3838bcfc2beaf2b9ea9274e350bc1295b7a9b39879c07b8857e775c2fe11af" exitCode=0 Jul 22 03:05:34.207358 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:34.207204 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rdltl" Jul 22 03:05:34.207358 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:34.207251 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rdltl" event={"ID":"cb9c3e6d-7da0-421f-82ef-e085f5e7e012","Type":"ContainerDied","Data":"df3838bcfc2beaf2b9ea9274e350bc1295b7a9b39879c07b8857e775c2fe11af"} Jul 22 03:05:34.207358 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:34.207305 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rdltl" event={"ID":"cb9c3e6d-7da0-421f-82ef-e085f5e7e012","Type":"ContainerDied","Data":"627d1ab80df045c70f0b337dd4974d2a1c4a21ecd69123faf6ac3144b87811d0"} Jul 22 03:05:34.207358 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:34.207331 2582 scope.go:117] "RemoveContainer" containerID="df3838bcfc2beaf2b9ea9274e350bc1295b7a9b39879c07b8857e775c2fe11af" Jul 22 03:05:34.216777 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:34.216759 2582 scope.go:117] "RemoveContainer" containerID="df3838bcfc2beaf2b9ea9274e350bc1295b7a9b39879c07b8857e775c2fe11af" Jul 22 03:05:34.217065 ip-10-0-131-67 kubenswrapper[2582]: E0722 03:05:34.217046 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df3838bcfc2beaf2b9ea9274e350bc1295b7a9b39879c07b8857e775c2fe11af\": container with ID starting with df3838bcfc2beaf2b9ea9274e350bc1295b7a9b39879c07b8857e775c2fe11af not found: ID does not exist" containerID="df3838bcfc2beaf2b9ea9274e350bc1295b7a9b39879c07b8857e775c2fe11af" Jul 22 03:05:34.217146 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:34.217073 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df3838bcfc2beaf2b9ea9274e350bc1295b7a9b39879c07b8857e775c2fe11af"} err="failed to get container status \"df3838bcfc2beaf2b9ea9274e350bc1295b7a9b39879c07b8857e775c2fe11af\": rpc error: code = NotFound desc = could not find container \"df3838bcfc2beaf2b9ea9274e350bc1295b7a9b39879c07b8857e775c2fe11af\": container with ID starting with df3838bcfc2beaf2b9ea9274e350bc1295b7a9b39879c07b8857e775c2fe11af not found: ID does not exist" Jul 22 03:05:34.235299 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:34.235272 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rdltl"] Jul 22 03:05:34.239244 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:34.239221 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rdltl"] Jul 22 03:05:35.901563 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:35.901529 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb9c3e6d-7da0-421f-82ef-e085f5e7e012" path="/var/lib/kubelet/pods/cb9c3e6d-7da0-421f-82ef-e085f5e7e012/volumes" Jul 22 03:05:36.189343 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:36.189261 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:05:36.844542 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:36.844506 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:05:36.911445 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:36.911412 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:05:40.657115 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:40.657079 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:05:40.757790 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:40.757756 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:05:42.197308 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:42.197279 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9mb6v" Jul 22 03:05:45.447268 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:45.447233 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:05:45.548160 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:45.548128 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:05:50.058495 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:50.058462 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lgdw9"] Jul 22 03:05:50.058975 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:50.058867 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb9c3e6d-7da0-421f-82ef-e085f5e7e012" containerName="manager" Jul 22 03:05:50.058975 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:50.058882 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb9c3e6d-7da0-421f-82ef-e085f5e7e012" containerName="manager" Jul 22 03:05:50.058975 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:50.058968 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb9c3e6d-7da0-421f-82ef-e085f5e7e012" containerName="manager" Jul 22 03:05:50.094695 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:50.094667 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:05:50.100472 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:50.100404 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lgdw9"] Jul 22 03:05:50.100629 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:50.100503 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lgdw9" Jul 22 03:05:50.103145 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:50.103124 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-n7vx2\"" Jul 22 03:05:50.103145 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:50.103133 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 22 03:05:50.139233 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:50.139203 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ca7bc57-768b-42fd-98df-ddddde7d3b08-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-lgdw9\" (UID: \"9ca7bc57-768b-42fd-98df-ddddde7d3b08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lgdw9" Jul 22 03:05:50.139413 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:50.139255 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ca7bc57-768b-42fd-98df-ddddde7d3b08-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-lgdw9\" (UID: \"9ca7bc57-768b-42fd-98df-ddddde7d3b08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lgdw9" Jul 22 03:05:50.139413 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:50.139311 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ca7bc57-768b-42fd-98df-ddddde7d3b08-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-lgdw9\" (UID: \"9ca7bc57-768b-42fd-98df-ddddde7d3b08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lgdw9" Jul 22 03:05:50.139413 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:50.139403 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ca7bc57-768b-42fd-98df-ddddde7d3b08-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-lgdw9\" (UID: \"9ca7bc57-768b-42fd-98df-ddddde7d3b08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lgdw9" Jul 22 03:05:50.139565 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:50.139442 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ca7bc57-768b-42fd-98df-ddddde7d3b08-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-lgdw9\" (UID: \"9ca7bc57-768b-42fd-98df-ddddde7d3b08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lgdw9" Jul 22 03:05:50.139565 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:50.139461 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpmzp\" (UniqueName: \"kubernetes.io/projected/9ca7bc57-768b-42fd-98df-ddddde7d3b08-kube-api-access-jpmzp\") pod \"maas-default-gateway-openshift-default-56654dcf7-lgdw9\" (UID: \"9ca7bc57-768b-42fd-98df-ddddde7d3b08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lgdw9" Jul 22 03:05:50.139649 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:50.139578 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ca7bc57-768b-42fd-98df-ddddde7d3b08-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-lgdw9\" (UID: \"9ca7bc57-768b-42fd-98df-ddddde7d3b08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lgdw9" Jul 22 03:05:50.139649 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:50.139608 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ca7bc57-768b-42fd-98df-ddddde7d3b08-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-lgdw9\" (UID: \"9ca7bc57-768b-42fd-98df-ddddde7d3b08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lgdw9" Jul 22 03:05:50.139649 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:50.139627 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ca7bc57-768b-42fd-98df-ddddde7d3b08-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-lgdw9\" (UID: \"9ca7bc57-768b-42fd-98df-ddddde7d3b08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lgdw9" Jul 22 03:05:50.240477 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:50.240440 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ca7bc57-768b-42fd-98df-ddddde7d3b08-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-lgdw9\" (UID: \"9ca7bc57-768b-42fd-98df-ddddde7d3b08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lgdw9" Jul 22 03:05:50.240477 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:50.240476 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ca7bc57-768b-42fd-98df-ddddde7d3b08-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-lgdw9\" (UID: \"9ca7bc57-768b-42fd-98df-ddddde7d3b08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lgdw9" Jul 22 03:05:50.240781 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:50.240506 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ca7bc57-768b-42fd-98df-ddddde7d3b08-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-lgdw9\" (UID: \"9ca7bc57-768b-42fd-98df-ddddde7d3b08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lgdw9" Jul 22 03:05:50.240781 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:50.240546 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ca7bc57-768b-42fd-98df-ddddde7d3b08-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-lgdw9\" (UID: \"9ca7bc57-768b-42fd-98df-ddddde7d3b08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lgdw9" Jul 22 03:05:50.240781 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:50.240586 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ca7bc57-768b-42fd-98df-ddddde7d3b08-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-lgdw9\" (UID: \"9ca7bc57-768b-42fd-98df-ddddde7d3b08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lgdw9" Jul 22 03:05:50.240781 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:50.240639 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ca7bc57-768b-42fd-98df-ddddde7d3b08-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-lgdw9\" (UID: \"9ca7bc57-768b-42fd-98df-ddddde7d3b08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lgdw9" Jul 22 03:05:50.240781 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:50.240675 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ca7bc57-768b-42fd-98df-ddddde7d3b08-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-lgdw9\" (UID: \"9ca7bc57-768b-42fd-98df-ddddde7d3b08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lgdw9" Jul 22 03:05:50.240781 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:50.240702 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpmzp\" (UniqueName: \"kubernetes.io/projected/9ca7bc57-768b-42fd-98df-ddddde7d3b08-kube-api-access-jpmzp\") pod \"maas-default-gateway-openshift-default-56654dcf7-lgdw9\" (UID: \"9ca7bc57-768b-42fd-98df-ddddde7d3b08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lgdw9" Jul 22 03:05:50.240781 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:50.240780 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ca7bc57-768b-42fd-98df-ddddde7d3b08-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-lgdw9\" (UID: \"9ca7bc57-768b-42fd-98df-ddddde7d3b08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lgdw9" Jul 22 03:05:50.241183 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:50.240928 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ca7bc57-768b-42fd-98df-ddddde7d3b08-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-lgdw9\" (UID: \"9ca7bc57-768b-42fd-98df-ddddde7d3b08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lgdw9" Jul 22 03:05:50.241183 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:50.241004 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ca7bc57-768b-42fd-98df-ddddde7d3b08-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-lgdw9\" (UID: \"9ca7bc57-768b-42fd-98df-ddddde7d3b08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lgdw9" Jul 22 03:05:50.241183 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:50.241112 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ca7bc57-768b-42fd-98df-ddddde7d3b08-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-lgdw9\" (UID: \"9ca7bc57-768b-42fd-98df-ddddde7d3b08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lgdw9" Jul 22 03:05:50.241354 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:50.241336 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ca7bc57-768b-42fd-98df-ddddde7d3b08-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-lgdw9\" (UID: \"9ca7bc57-768b-42fd-98df-ddddde7d3b08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lgdw9" Jul 22 03:05:50.241416 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:50.241384 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ca7bc57-768b-42fd-98df-ddddde7d3b08-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-lgdw9\" (UID: \"9ca7bc57-768b-42fd-98df-ddddde7d3b08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lgdw9" Jul 22 03:05:50.242965 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:50.242939 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ca7bc57-768b-42fd-98df-ddddde7d3b08-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-lgdw9\" (UID: \"9ca7bc57-768b-42fd-98df-ddddde7d3b08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lgdw9" Jul 22 03:05:50.243095 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:50.243056 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ca7bc57-768b-42fd-98df-ddddde7d3b08-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-lgdw9\" (UID: \"9ca7bc57-768b-42fd-98df-ddddde7d3b08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lgdw9" Jul 22 03:05:50.249526 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:50.249499 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ca7bc57-768b-42fd-98df-ddddde7d3b08-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-lgdw9\" (UID: \"9ca7bc57-768b-42fd-98df-ddddde7d3b08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lgdw9" Jul 22 03:05:50.249638 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:50.249624 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpmzp\" (UniqueName: \"kubernetes.io/projected/9ca7bc57-768b-42fd-98df-ddddde7d3b08-kube-api-access-jpmzp\") pod \"maas-default-gateway-openshift-default-56654dcf7-lgdw9\" (UID: \"9ca7bc57-768b-42fd-98df-ddddde7d3b08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lgdw9" Jul 22 03:05:50.411151 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:50.411120 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lgdw9" Jul 22 03:05:50.548896 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:50.548868 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:05:50.611416 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:50.611391 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lgdw9"] Jul 22 03:05:50.613833 ip-10-0-131-67 kubenswrapper[2582]: W0722 03:05:50.613803 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ca7bc57_768b_42fd_98df_ddddde7d3b08.slice/crio-bd455d7caf09ef7a676de3215b189f733015aa71c1d0836efc528f2b0be57938 WatchSource:0}: Error finding container bd455d7caf09ef7a676de3215b189f733015aa71c1d0836efc528f2b0be57938: Status 404 returned error can't find the container with id bd455d7caf09ef7a676de3215b189f733015aa71c1d0836efc528f2b0be57938 Jul 22 03:05:51.309698 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:51.309662 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lgdw9" event={"ID":"9ca7bc57-768b-42fd-98df-ddddde7d3b08","Type":"ContainerStarted","Data":"bd455d7caf09ef7a676de3215b189f733015aa71c1d0836efc528f2b0be57938"} Jul 22 03:05:53.270667 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:53.270622 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 03:05:53.270972 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:53.270759 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 03:05:53.270972 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:53.270796 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 03:05:54.330736 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:54.330700 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lgdw9" event={"ID":"9ca7bc57-768b-42fd-98df-ddddde7d3b08","Type":"ContainerStarted","Data":"ef22f733354d3a77ab411cea9a6d0fa1196d6770439cc59f7854ebb7814db000"} Jul 22 03:05:54.355507 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:54.355451 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lgdw9" podStartSLOduration=1.701010153 podStartE2EDuration="4.355430284s" totalImagesPullingTime="2.654420131s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:05:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:05:54.352893947 +0000 UTC m=+465.084514794" watchObservedRunningTime="2026-07-22 03:05:54.355430284 +0000 UTC m=+465.087051133" Jul 22 03:05:54.411257 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:54.411218 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lgdw9" Jul 22 03:05:54.416212 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:54.416186 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lgdw9" Jul 22 03:05:55.336675 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:55.336644 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lgdw9" Jul 22 03:05:55.337615 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:55.337596 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lgdw9" Jul 22 03:05:55.513519 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:55.513480 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:05:55.530959 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:55.530929 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-f6twk"] Jul 22 03:05:55.560884 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:55.560856 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-f6twk"] Jul 22 03:05:55.561063 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:55.560974 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-f6twk" Jul 22 03:05:55.565807 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:55.565785 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 22 03:05:55.590716 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:55.590625 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/27784f14-2882-4e66-9972-201bbdec53c7-config-file\") pod \"limitador-limitador-64c94fd64f-f6twk\" (UID: \"27784f14-2882-4e66-9972-201bbdec53c7\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-f6twk" Jul 22 03:05:55.590894 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:55.590732 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zwnl\" (UniqueName: \"kubernetes.io/projected/27784f14-2882-4e66-9972-201bbdec53c7-kube-api-access-2zwnl\") pod \"limitador-limitador-64c94fd64f-f6twk\" (UID: \"27784f14-2882-4e66-9972-201bbdec53c7\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-f6twk" Jul 22 03:05:55.629042 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:55.628984 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:05:55.630506 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:55.630483 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-f6twk"] Jul 22 03:05:55.691343 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:55.691297 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/27784f14-2882-4e66-9972-201bbdec53c7-config-file\") pod \"limitador-limitador-64c94fd64f-f6twk\" (UID: \"27784f14-2882-4e66-9972-201bbdec53c7\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-f6twk" Jul 22 03:05:55.691518 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:55.691379 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zwnl\" (UniqueName: \"kubernetes.io/projected/27784f14-2882-4e66-9972-201bbdec53c7-kube-api-access-2zwnl\") pod \"limitador-limitador-64c94fd64f-f6twk\" (UID: \"27784f14-2882-4e66-9972-201bbdec53c7\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-f6twk" Jul 22 03:05:55.691918 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:55.691900 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/27784f14-2882-4e66-9972-201bbdec53c7-config-file\") pod \"limitador-limitador-64c94fd64f-f6twk\" (UID: \"27784f14-2882-4e66-9972-201bbdec53c7\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-f6twk" Jul 22 03:05:55.700492 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:55.700460 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zwnl\" (UniqueName: \"kubernetes.io/projected/27784f14-2882-4e66-9972-201bbdec53c7-kube-api-access-2zwnl\") pod \"limitador-limitador-64c94fd64f-f6twk\" (UID: \"27784f14-2882-4e66-9972-201bbdec53c7\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-f6twk" Jul 22 03:05:55.729287 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:55.729225 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:05:55.871249 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:55.871156 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-f6twk" Jul 22 03:05:55.979610 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:55.979582 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:05:55.984296 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:55.984274 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-8k8vz" Jul 22 03:05:55.993525 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:55.993499 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:05:56.025311 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:56.025275 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:05:56.028590 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:56.028569 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-f6twk"] Jul 22 03:05:56.030607 ip-10-0-131-67 kubenswrapper[2582]: W0722 03:05:56.030577 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27784f14_2882_4e66_9972_201bbdec53c7.slice/crio-2769b328dfa0e577a5c736dcea624db02d42e2676e8d0c192db2dfefab7cab7c WatchSource:0}: Error finding container 2769b328dfa0e577a5c736dcea624db02d42e2676e8d0c192db2dfefab7cab7c: Status 404 returned error can't find the container with id 2769b328dfa0e577a5c736dcea624db02d42e2676e8d0c192db2dfefab7cab7c Jul 22 03:05:56.094332 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:56.094297 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/18482f74-ab9c-4f92-b1f6-ea302e2419a2-config-file\") pod \"limitador-limitador-54d6d988d5-8k8vz\" (UID: \"18482f74-ab9c-4f92-b1f6-ea302e2419a2\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-8k8vz" Jul 22 03:05:56.094508 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:56.094340 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsd8l\" (UniqueName: \"kubernetes.io/projected/18482f74-ab9c-4f92-b1f6-ea302e2419a2-kube-api-access-fsd8l\") pod \"limitador-limitador-54d6d988d5-8k8vz\" (UID: \"18482f74-ab9c-4f92-b1f6-ea302e2419a2\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-8k8vz" Jul 22 03:05:56.195494 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:56.195463 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsd8l\" (UniqueName: \"kubernetes.io/projected/18482f74-ab9c-4f92-b1f6-ea302e2419a2-kube-api-access-fsd8l\") pod \"limitador-limitador-54d6d988d5-8k8vz\" (UID: \"18482f74-ab9c-4f92-b1f6-ea302e2419a2\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-8k8vz" Jul 22 03:05:56.195701 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:56.195615 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/18482f74-ab9c-4f92-b1f6-ea302e2419a2-config-file\") pod \"limitador-limitador-54d6d988d5-8k8vz\" (UID: \"18482f74-ab9c-4f92-b1f6-ea302e2419a2\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-8k8vz" Jul 22 03:05:56.196238 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:56.196214 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/18482f74-ab9c-4f92-b1f6-ea302e2419a2-config-file\") pod \"limitador-limitador-54d6d988d5-8k8vz\" (UID: \"18482f74-ab9c-4f92-b1f6-ea302e2419a2\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-8k8vz" Jul 22 03:05:56.203823 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:56.203797 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsd8l\" (UniqueName: \"kubernetes.io/projected/18482f74-ab9c-4f92-b1f6-ea302e2419a2-kube-api-access-fsd8l\") pod \"limitador-limitador-54d6d988d5-8k8vz\" (UID: \"18482f74-ab9c-4f92-b1f6-ea302e2419a2\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-8k8vz" Jul 22 03:05:56.298967 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:56.298925 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-8k8vz" Jul 22 03:05:56.344451 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:56.344416 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-f6twk" event={"ID":"27784f14-2882-4e66-9972-201bbdec53c7","Type":"ContainerStarted","Data":"2769b328dfa0e577a5c736dcea624db02d42e2676e8d0c192db2dfefab7cab7c"} Jul 22 03:05:56.430145 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:56.430104 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:05:56.445701 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:56.445600 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:05:56.534088 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:56.534053 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:05:57.353829 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:57.353773 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-8k8vz" event={"ID":"18482f74-ab9c-4f92-b1f6-ea302e2419a2","Type":"ContainerStarted","Data":"faa57f9aad17ee844f8c679c3036858a3e9a756390d022c84d1e49774519d094"} Jul 22 03:05:59.367136 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:59.366996 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-8k8vz" event={"ID":"18482f74-ab9c-4f92-b1f6-ea302e2419a2","Type":"ContainerStarted","Data":"63e7921b0c774dcf5591160f73411385c6053753d99a50d64c4d24402ad19598"} Jul 22 03:05:59.367567 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:59.367137 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-8k8vz" Jul 22 03:05:59.368738 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:59.368709 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-f6twk" event={"ID":"27784f14-2882-4e66-9972-201bbdec53c7","Type":"ContainerStarted","Data":"cfc34e3245b17a0664cb9675e20ad50e355afa15a99629458f62c606034273fb"} Jul 22 03:05:59.368868 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:59.368770 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-f6twk" Jul 22 03:05:59.389438 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:59.389390 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-8k8vz" podStartSLOduration=1.841666387 podStartE2EDuration="4.38937357s" totalImagesPullingTime="2.547707183s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:05:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:05:59.385779347 +0000 UTC m=+470.117400195" watchObservedRunningTime="2026-07-22 03:05:59.38937357 +0000 UTC m=+470.120994416" Jul 22 03:05:59.409677 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:05:59.409630 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-f6twk" podStartSLOduration=1.449785672 podStartE2EDuration="4.409615754s" totalImagesPullingTime="2.959830082s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:05:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:05:59.407495747 +0000 UTC m=+470.139116634" watchObservedRunningTime="2026-07-22 03:05:59.409615754 +0000 UTC m=+470.141236601" Jul 22 03:06:08.978569 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:08.978535 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:06:09.066886 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:09.066854 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:06:10.375350 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:10.375315 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-f6twk" Jul 22 03:06:10.375822 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:10.375386 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-8k8vz" Jul 22 03:06:10.447721 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:10.447685 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-f6twk"] Jul 22 03:06:10.447946 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:10.447911 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-f6twk" podUID="27784f14-2882-4e66-9972-201bbdec53c7" containerName="limitador" containerID="cri-o://cfc34e3245b17a0664cb9675e20ad50e355afa15a99629458f62c606034273fb" gracePeriod=30 Jul 22 03:06:11.401795 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:11.401770 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-f6twk" Jul 22 03:06:11.435949 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:11.435917 2582 generic.go:356] "Generic (PLEG): container finished" podID="27784f14-2882-4e66-9972-201bbdec53c7" containerID="cfc34e3245b17a0664cb9675e20ad50e355afa15a99629458f62c606034273fb" exitCode=0 Jul 22 03:06:11.436101 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:11.435965 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-f6twk" Jul 22 03:06:11.436101 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:11.435987 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-f6twk" event={"ID":"27784f14-2882-4e66-9972-201bbdec53c7","Type":"ContainerDied","Data":"cfc34e3245b17a0664cb9675e20ad50e355afa15a99629458f62c606034273fb"} Jul 22 03:06:11.436101 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:11.436079 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-f6twk" event={"ID":"27784f14-2882-4e66-9972-201bbdec53c7","Type":"ContainerDied","Data":"2769b328dfa0e577a5c736dcea624db02d42e2676e8d0c192db2dfefab7cab7c"} Jul 22 03:06:11.436265 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:11.436110 2582 scope.go:117] "RemoveContainer" containerID="cfc34e3245b17a0664cb9675e20ad50e355afa15a99629458f62c606034273fb" Jul 22 03:06:11.446475 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:11.446331 2582 scope.go:117] "RemoveContainer" containerID="cfc34e3245b17a0664cb9675e20ad50e355afa15a99629458f62c606034273fb" Jul 22 03:06:11.446686 ip-10-0-131-67 kubenswrapper[2582]: E0722 03:06:11.446589 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfc34e3245b17a0664cb9675e20ad50e355afa15a99629458f62c606034273fb\": container with ID starting with cfc34e3245b17a0664cb9675e20ad50e355afa15a99629458f62c606034273fb not found: ID does not exist" containerID="cfc34e3245b17a0664cb9675e20ad50e355afa15a99629458f62c606034273fb" Jul 22 03:06:11.446686 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:11.446623 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfc34e3245b17a0664cb9675e20ad50e355afa15a99629458f62c606034273fb"} err="failed to get container status \"cfc34e3245b17a0664cb9675e20ad50e355afa15a99629458f62c606034273fb\": rpc error: code = NotFound desc = could not find container \"cfc34e3245b17a0664cb9675e20ad50e355afa15a99629458f62c606034273fb\": container with ID starting with cfc34e3245b17a0664cb9675e20ad50e355afa15a99629458f62c606034273fb not found: ID does not exist" Jul 22 03:06:11.535832 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:11.535741 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/27784f14-2882-4e66-9972-201bbdec53c7-config-file\") pod \"27784f14-2882-4e66-9972-201bbdec53c7\" (UID: \"27784f14-2882-4e66-9972-201bbdec53c7\") " Jul 22 03:06:11.535832 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:11.535808 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2zwnl\" (UniqueName: \"kubernetes.io/projected/27784f14-2882-4e66-9972-201bbdec53c7-kube-api-access-2zwnl\") pod \"27784f14-2882-4e66-9972-201bbdec53c7\" (UID: \"27784f14-2882-4e66-9972-201bbdec53c7\") " Jul 22 03:06:11.536186 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:11.536161 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27784f14-2882-4e66-9972-201bbdec53c7-config-file" (OuterVolumeSpecName: "config-file") pod "27784f14-2882-4e66-9972-201bbdec53c7" (UID: "27784f14-2882-4e66-9972-201bbdec53c7"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 03:06:11.537801 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:11.537774 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27784f14-2882-4e66-9972-201bbdec53c7-kube-api-access-2zwnl" (OuterVolumeSpecName: "kube-api-access-2zwnl") pod "27784f14-2882-4e66-9972-201bbdec53c7" (UID: "27784f14-2882-4e66-9972-201bbdec53c7"). InnerVolumeSpecName "kube-api-access-2zwnl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 03:06:11.637150 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:11.637122 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zwnl\" (UniqueName: \"kubernetes.io/projected/27784f14-2882-4e66-9972-201bbdec53c7-kube-api-access-2zwnl\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 03:06:11.637150 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:11.637147 2582 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/27784f14-2882-4e66-9972-201bbdec53c7-config-file\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 03:06:11.697587 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:11.697557 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:06:11.763146 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:11.763113 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-f6twk"] Jul 22 03:06:11.770291 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:11.770266 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-f6twk"] Jul 22 03:06:11.807604 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:11.807522 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:06:11.901640 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:11.901609 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27784f14-2882-4e66-9972-201bbdec53c7" path="/var/lib/kubelet/pods/27784f14-2882-4e66-9972-201bbdec53c7/volumes" Jul 22 03:06:12.260516 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:12.260487 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:06:19.926917 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:19.926879 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:06:20.023966 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:20.023934 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:06:27.099592 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:27.099553 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:06:27.294409 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:27.294373 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:06:34.550176 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:34.550142 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:06:34.574083 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:34.574052 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-9cd744bc7-zcbk4"] Jul 22 03:06:34.574478 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:34.574465 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27784f14-2882-4e66-9972-201bbdec53c7" containerName="limitador" Jul 22 03:06:34.574522 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:34.574480 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="27784f14-2882-4e66-9972-201bbdec53c7" containerName="limitador" Jul 22 03:06:34.574573 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:34.574550 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="27784f14-2882-4e66-9972-201bbdec53c7" containerName="limitador" Jul 22 03:06:34.577694 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:34.577679 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9cd744bc7-zcbk4" Jul 22 03:06:34.580043 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:34.580001 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 22 03:06:34.580193 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:34.580131 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-fmzc5\"" Jul 22 03:06:34.581078 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:34.581064 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 22 03:06:34.587346 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:34.587325 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-9cd744bc7-zcbk4"] Jul 22 03:06:34.664926 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:34.664894 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:06:34.739653 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:34.739611 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/11dbe7b1-2b01-40f9-a980-98c4dbfaafa8-webhook-cert\") pod \"maas-controller-9cd744bc7-zcbk4\" (UID: \"11dbe7b1-2b01-40f9-a980-98c4dbfaafa8\") " pod="opendatahub/maas-controller-9cd744bc7-zcbk4" Jul 22 03:06:34.739847 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:34.739663 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghzmn\" (UniqueName: \"kubernetes.io/projected/11dbe7b1-2b01-40f9-a980-98c4dbfaafa8-kube-api-access-ghzmn\") pod \"maas-controller-9cd744bc7-zcbk4\" (UID: \"11dbe7b1-2b01-40f9-a980-98c4dbfaafa8\") " pod="opendatahub/maas-controller-9cd744bc7-zcbk4" Jul 22 03:06:34.769832 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:34.769798 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:06:34.782516 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:34.782490 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6446b4cb87-mk6rs"] Jul 22 03:06:34.785991 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:34.785966 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6446b4cb87-mk6rs" Jul 22 03:06:34.799182 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:34.799157 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6446b4cb87-mk6rs"] Jul 22 03:06:34.841175 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:34.841082 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/11dbe7b1-2b01-40f9-a980-98c4dbfaafa8-webhook-cert\") pod \"maas-controller-9cd744bc7-zcbk4\" (UID: \"11dbe7b1-2b01-40f9-a980-98c4dbfaafa8\") " pod="opendatahub/maas-controller-9cd744bc7-zcbk4" Jul 22 03:06:34.841175 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:34.841150 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghzmn\" (UniqueName: \"kubernetes.io/projected/11dbe7b1-2b01-40f9-a980-98c4dbfaafa8-kube-api-access-ghzmn\") pod \"maas-controller-9cd744bc7-zcbk4\" (UID: \"11dbe7b1-2b01-40f9-a980-98c4dbfaafa8\") " pod="opendatahub/maas-controller-9cd744bc7-zcbk4" Jul 22 03:06:34.841800 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:34.841778 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:06:34.843525 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:34.843499 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/11dbe7b1-2b01-40f9-a980-98c4dbfaafa8-webhook-cert\") pod \"maas-controller-9cd744bc7-zcbk4\" (UID: \"11dbe7b1-2b01-40f9-a980-98c4dbfaafa8\") " pod="opendatahub/maas-controller-9cd744bc7-zcbk4" Jul 22 03:06:34.852508 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:34.852475 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghzmn\" (UniqueName: \"kubernetes.io/projected/11dbe7b1-2b01-40f9-a980-98c4dbfaafa8-kube-api-access-ghzmn\") pod \"maas-controller-9cd744bc7-zcbk4\" (UID: \"11dbe7b1-2b01-40f9-a980-98c4dbfaafa8\") " pod="opendatahub/maas-controller-9cd744bc7-zcbk4" Jul 22 03:06:34.888917 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:34.888879 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9cd744bc7-zcbk4" Jul 22 03:06:34.941893 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:34.941859 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fvrj\" (UniqueName: \"kubernetes.io/projected/76cb9a05-537f-45d7-bb11-facc17791c13-kube-api-access-5fvrj\") pod \"maas-controller-6446b4cb87-mk6rs\" (UID: \"76cb9a05-537f-45d7-bb11-facc17791c13\") " pod="opendatahub/maas-controller-6446b4cb87-mk6rs" Jul 22 03:06:34.942043 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:34.941968 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/76cb9a05-537f-45d7-bb11-facc17791c13-webhook-cert\") pod \"maas-controller-6446b4cb87-mk6rs\" (UID: \"76cb9a05-537f-45d7-bb11-facc17791c13\") " pod="opendatahub/maas-controller-6446b4cb87-mk6rs" Jul 22 03:06:35.030845 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:35.030819 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-9cd744bc7-zcbk4"] Jul 22 03:06:35.032657 ip-10-0-131-67 kubenswrapper[2582]: W0722 03:06:35.032625 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11dbe7b1_2b01_40f9_a980_98c4dbfaafa8.slice/crio-b89e8afe85b4f5b50197815b090e4e9fee11a767a27fbd7198cf6e96a82146a0 WatchSource:0}: Error finding container b89e8afe85b4f5b50197815b090e4e9fee11a767a27fbd7198cf6e96a82146a0: Status 404 returned error can't find the container with id b89e8afe85b4f5b50197815b090e4e9fee11a767a27fbd7198cf6e96a82146a0 Jul 22 03:06:35.043369 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:35.043347 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fvrj\" (UniqueName: \"kubernetes.io/projected/76cb9a05-537f-45d7-bb11-facc17791c13-kube-api-access-5fvrj\") pod \"maas-controller-6446b4cb87-mk6rs\" (UID: \"76cb9a05-537f-45d7-bb11-facc17791c13\") " pod="opendatahub/maas-controller-6446b4cb87-mk6rs" Jul 22 03:06:35.043473 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:35.043393 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/76cb9a05-537f-45d7-bb11-facc17791c13-webhook-cert\") pod \"maas-controller-6446b4cb87-mk6rs\" (UID: \"76cb9a05-537f-45d7-bb11-facc17791c13\") " pod="opendatahub/maas-controller-6446b4cb87-mk6rs" Jul 22 03:06:35.045733 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:35.045713 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/76cb9a05-537f-45d7-bb11-facc17791c13-webhook-cert\") pod \"maas-controller-6446b4cb87-mk6rs\" (UID: \"76cb9a05-537f-45d7-bb11-facc17791c13\") " pod="opendatahub/maas-controller-6446b4cb87-mk6rs" Jul 22 03:06:35.051573 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:35.051553 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fvrj\" (UniqueName: \"kubernetes.io/projected/76cb9a05-537f-45d7-bb11-facc17791c13-kube-api-access-5fvrj\") pod \"maas-controller-6446b4cb87-mk6rs\" (UID: \"76cb9a05-537f-45d7-bb11-facc17791c13\") " pod="opendatahub/maas-controller-6446b4cb87-mk6rs" Jul 22 03:06:35.096649 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:35.096562 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6446b4cb87-mk6rs" Jul 22 03:06:35.243293 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:35.243271 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6446b4cb87-mk6rs"] Jul 22 03:06:35.245260 ip-10-0-131-67 kubenswrapper[2582]: W0722 03:06:35.245230 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76cb9a05_537f_45d7_bb11_facc17791c13.slice/crio-18282631d84e21c9abd12d4d5bcd4dd46007c6519686bbf8064b2de41c7557a8 WatchSource:0}: Error finding container 18282631d84e21c9abd12d4d5bcd4dd46007c6519686bbf8064b2de41c7557a8: Status 404 returned error can't find the container with id 18282631d84e21c9abd12d4d5bcd4dd46007c6519686bbf8064b2de41c7557a8 Jul 22 03:06:35.572139 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:35.572102 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9cd744bc7-zcbk4" event={"ID":"11dbe7b1-2b01-40f9-a980-98c4dbfaafa8","Type":"ContainerStarted","Data":"b89e8afe85b4f5b50197815b090e4e9fee11a767a27fbd7198cf6e96a82146a0"} Jul 22 03:06:35.573271 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:35.573243 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6446b4cb87-mk6rs" event={"ID":"76cb9a05-537f-45d7-bb11-facc17791c13","Type":"ContainerStarted","Data":"18282631d84e21c9abd12d4d5bcd4dd46007c6519686bbf8064b2de41c7557a8"} Jul 22 03:06:37.995927 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:37.995840 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:06:38.108348 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:38.108317 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:06:38.600489 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:38.600443 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9cd744bc7-zcbk4" event={"ID":"11dbe7b1-2b01-40f9-a980-98c4dbfaafa8","Type":"ContainerStarted","Data":"32a1243b4fe9864810299205837c418741b2193126ce722ec56474e17ea3c439"} Jul 22 03:06:38.600676 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:38.600545 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-9cd744bc7-zcbk4" Jul 22 03:06:38.601928 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:38.601901 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6446b4cb87-mk6rs" event={"ID":"76cb9a05-537f-45d7-bb11-facc17791c13","Type":"ContainerStarted","Data":"59c715cd73f62129834f86b54e3a687d6762558d4de0398200298edbd6f9c3de"} Jul 22 03:06:38.602107 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:38.602049 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6446b4cb87-mk6rs" Jul 22 03:06:38.620796 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:38.620746 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-9cd744bc7-zcbk4" podStartSLOduration=1.934725198 podStartE2EDuration="4.620731281s" totalImagesPullingTime="2.686006083s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:06:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:06:38.619612998 +0000 UTC m=+509.351233858" watchObservedRunningTime="2026-07-22 03:06:38.620731281 +0000 UTC m=+509.352352127" Jul 22 03:06:38.646088 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:38.646034 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6446b4cb87-mk6rs" podStartSLOduration=2.165607435 podStartE2EDuration="4.646001811s" totalImagesPullingTime="2.480394376s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:06:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:06:38.64448846 +0000 UTC m=+509.376109327" watchObservedRunningTime="2026-07-22 03:06:38.646001811 +0000 UTC m=+509.377622659" Jul 22 03:06:49.611606 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:49.611576 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6446b4cb87-mk6rs" Jul 22 03:06:49.612027 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:49.611643 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-9cd744bc7-zcbk4" Jul 22 03:06:49.679119 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:49.679086 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-9cd744bc7-zcbk4"] Jul 22 03:06:49.679328 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:49.679293 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-9cd744bc7-zcbk4" podUID="11dbe7b1-2b01-40f9-a980-98c4dbfaafa8" containerName="manager" containerID="cri-o://32a1243b4fe9864810299205837c418741b2193126ce722ec56474e17ea3c439" gracePeriod=30 Jul 22 03:06:49.941886 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:49.941861 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9cd744bc7-zcbk4" Jul 22 03:06:49.976503 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:49.976471 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ghzmn\" (UniqueName: \"kubernetes.io/projected/11dbe7b1-2b01-40f9-a980-98c4dbfaafa8-kube-api-access-ghzmn\") pod \"11dbe7b1-2b01-40f9-a980-98c4dbfaafa8\" (UID: \"11dbe7b1-2b01-40f9-a980-98c4dbfaafa8\") " Jul 22 03:06:49.976705 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:49.976513 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/11dbe7b1-2b01-40f9-a980-98c4dbfaafa8-webhook-cert\") pod \"11dbe7b1-2b01-40f9-a980-98c4dbfaafa8\" (UID: \"11dbe7b1-2b01-40f9-a980-98c4dbfaafa8\") " Jul 22 03:06:49.978741 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:49.978710 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11dbe7b1-2b01-40f9-a980-98c4dbfaafa8-kube-api-access-ghzmn" (OuterVolumeSpecName: "kube-api-access-ghzmn") pod "11dbe7b1-2b01-40f9-a980-98c4dbfaafa8" (UID: "11dbe7b1-2b01-40f9-a980-98c4dbfaafa8"). InnerVolumeSpecName "kube-api-access-ghzmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 03:06:49.978860 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:49.978752 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11dbe7b1-2b01-40f9-a980-98c4dbfaafa8-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "11dbe7b1-2b01-40f9-a980-98c4dbfaafa8" (UID: "11dbe7b1-2b01-40f9-a980-98c4dbfaafa8"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 03:06:50.077998 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:50.077968 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghzmn\" (UniqueName: \"kubernetes.io/projected/11dbe7b1-2b01-40f9-a980-98c4dbfaafa8-kube-api-access-ghzmn\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 03:06:50.077998 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:50.077995 2582 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/11dbe7b1-2b01-40f9-a980-98c4dbfaafa8-webhook-cert\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 03:06:50.674113 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:50.674082 2582 generic.go:356] "Generic (PLEG): container finished" podID="11dbe7b1-2b01-40f9-a980-98c4dbfaafa8" containerID="32a1243b4fe9864810299205837c418741b2193126ce722ec56474e17ea3c439" exitCode=0 Jul 22 03:06:50.674518 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:50.674123 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9cd744bc7-zcbk4" Jul 22 03:06:50.674518 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:50.674143 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9cd744bc7-zcbk4" event={"ID":"11dbe7b1-2b01-40f9-a980-98c4dbfaafa8","Type":"ContainerDied","Data":"32a1243b4fe9864810299205837c418741b2193126ce722ec56474e17ea3c439"} Jul 22 03:06:50.674518 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:50.674185 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9cd744bc7-zcbk4" event={"ID":"11dbe7b1-2b01-40f9-a980-98c4dbfaafa8","Type":"ContainerDied","Data":"b89e8afe85b4f5b50197815b090e4e9fee11a767a27fbd7198cf6e96a82146a0"} Jul 22 03:06:50.674518 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:50.674207 2582 scope.go:117] "RemoveContainer" containerID="32a1243b4fe9864810299205837c418741b2193126ce722ec56474e17ea3c439" Jul 22 03:06:50.684157 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:50.683919 2582 scope.go:117] "RemoveContainer" containerID="32a1243b4fe9864810299205837c418741b2193126ce722ec56474e17ea3c439" Jul 22 03:06:50.684239 ip-10-0-131-67 kubenswrapper[2582]: E0722 03:06:50.684222 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32a1243b4fe9864810299205837c418741b2193126ce722ec56474e17ea3c439\": container with ID starting with 32a1243b4fe9864810299205837c418741b2193126ce722ec56474e17ea3c439 not found: ID does not exist" containerID="32a1243b4fe9864810299205837c418741b2193126ce722ec56474e17ea3c439" Jul 22 03:06:50.684277 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:50.684249 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32a1243b4fe9864810299205837c418741b2193126ce722ec56474e17ea3c439"} err="failed to get container status \"32a1243b4fe9864810299205837c418741b2193126ce722ec56474e17ea3c439\": rpc error: code = NotFound desc = could not find container \"32a1243b4fe9864810299205837c418741b2193126ce722ec56474e17ea3c439\": container with ID starting with 32a1243b4fe9864810299205837c418741b2193126ce722ec56474e17ea3c439 not found: ID does not exist" Jul 22 03:06:50.706100 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:50.706075 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-9cd744bc7-zcbk4"] Jul 22 03:06:50.709856 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:50.709834 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-9cd744bc7-zcbk4"] Jul 22 03:06:51.901708 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:06:51.901675 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11dbe7b1-2b01-40f9-a980-98c4dbfaafa8" path="/var/lib/kubelet/pods/11dbe7b1-2b01-40f9-a980-98c4dbfaafa8/volumes" Jul 22 03:07:09.995600 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:09.995518 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:07:10.095571 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:10.095535 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:07:10.632366 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:10.632300 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:07:10.773148 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:10.773108 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:07:11.311081 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:11.311046 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:07:11.408896 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:11.408867 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:07:11.438713 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:11.438673 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-tf965"] Jul 22 03:07:11.439505 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:11.439482 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11dbe7b1-2b01-40f9-a980-98c4dbfaafa8" containerName="manager" Jul 22 03:07:11.439505 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:11.439505 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="11dbe7b1-2b01-40f9-a980-98c4dbfaafa8" containerName="manager" Jul 22 03:07:11.439680 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:11.439672 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="11dbe7b1-2b01-40f9-a980-98c4dbfaafa8" containerName="manager" Jul 22 03:07:11.448162 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:11.448139 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-tf965" Jul 22 03:07:11.452760 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:11.452734 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 22 03:07:11.453228 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:11.453050 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-kbrsf\"" Jul 22 03:07:11.454865 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:11.454752 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-tf965"] Jul 22 03:07:11.484987 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:11.484957 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:07:11.494342 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:11.494314 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-szpl4"] Jul 22 03:07:11.499189 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:11.499168 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-szpl4" Jul 22 03:07:11.515464 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:11.515433 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-szpl4"] Jul 22 03:07:11.568883 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:11.568795 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56ca64c4-0973-4ed3-acb1-2d2ebfc80814-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-szpl4\" (UID: \"56ca64c4-0973-4ed3-acb1-2d2ebfc80814\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-szpl4" Jul 22 03:07:11.569075 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:11.568953 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/877a033d-62c3-46d6-a0d5-f23139602b32-plugins-config-volume\") pod \"payload-processing-5bf47cd988-tf965\" (UID: \"877a033d-62c3-46d6-a0d5-f23139602b32\") " pod="openshift-ingress/payload-processing-5bf47cd988-tf965" Jul 22 03:07:11.569075 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:11.569024 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chxb8\" (UniqueName: \"kubernetes.io/projected/877a033d-62c3-46d6-a0d5-f23139602b32-kube-api-access-chxb8\") pod \"payload-processing-5bf47cd988-tf965\" (UID: \"877a033d-62c3-46d6-a0d5-f23139602b32\") " pod="openshift-ingress/payload-processing-5bf47cd988-tf965" Jul 22 03:07:11.569229 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:11.569136 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b847m\" (UniqueName: \"kubernetes.io/projected/56ca64c4-0973-4ed3-acb1-2d2ebfc80814-kube-api-access-b847m\") pod \"payload-pre-processing-7677b774bb-szpl4\" (UID: \"56ca64c4-0973-4ed3-acb1-2d2ebfc80814\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-szpl4" Jul 22 03:07:11.670886 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:11.670847 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56ca64c4-0973-4ed3-acb1-2d2ebfc80814-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-szpl4\" (UID: \"56ca64c4-0973-4ed3-acb1-2d2ebfc80814\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-szpl4" Jul 22 03:07:11.671137 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:11.670896 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/877a033d-62c3-46d6-a0d5-f23139602b32-plugins-config-volume\") pod \"payload-processing-5bf47cd988-tf965\" (UID: \"877a033d-62c3-46d6-a0d5-f23139602b32\") " pod="openshift-ingress/payload-processing-5bf47cd988-tf965" Jul 22 03:07:11.671137 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:11.670940 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chxb8\" (UniqueName: \"kubernetes.io/projected/877a033d-62c3-46d6-a0d5-f23139602b32-kube-api-access-chxb8\") pod \"payload-processing-5bf47cd988-tf965\" (UID: \"877a033d-62c3-46d6-a0d5-f23139602b32\") " pod="openshift-ingress/payload-processing-5bf47cd988-tf965" Jul 22 03:07:11.671137 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:11.670986 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b847m\" (UniqueName: \"kubernetes.io/projected/56ca64c4-0973-4ed3-acb1-2d2ebfc80814-kube-api-access-b847m\") pod \"payload-pre-processing-7677b774bb-szpl4\" (UID: \"56ca64c4-0973-4ed3-acb1-2d2ebfc80814\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-szpl4" Jul 22 03:07:11.671639 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:11.671617 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/877a033d-62c3-46d6-a0d5-f23139602b32-plugins-config-volume\") pod \"payload-processing-5bf47cd988-tf965\" (UID: \"877a033d-62c3-46d6-a0d5-f23139602b32\") " pod="openshift-ingress/payload-processing-5bf47cd988-tf965" Jul 22 03:07:11.671717 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:11.671665 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56ca64c4-0973-4ed3-acb1-2d2ebfc80814-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-szpl4\" (UID: \"56ca64c4-0973-4ed3-acb1-2d2ebfc80814\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-szpl4" Jul 22 03:07:11.680477 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:11.680445 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chxb8\" (UniqueName: \"kubernetes.io/projected/877a033d-62c3-46d6-a0d5-f23139602b32-kube-api-access-chxb8\") pod \"payload-processing-5bf47cd988-tf965\" (UID: \"877a033d-62c3-46d6-a0d5-f23139602b32\") " pod="openshift-ingress/payload-processing-5bf47cd988-tf965" Jul 22 03:07:11.680582 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:11.680451 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b847m\" (UniqueName: \"kubernetes.io/projected/56ca64c4-0973-4ed3-acb1-2d2ebfc80814-kube-api-access-b847m\") pod \"payload-pre-processing-7677b774bb-szpl4\" (UID: \"56ca64c4-0973-4ed3-acb1-2d2ebfc80814\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-szpl4" Jul 22 03:07:11.761519 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:11.761484 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-tf965" Jul 22 03:07:11.810154 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:11.810115 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-szpl4" Jul 22 03:07:11.971039 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:11.955337 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-tf965"] Jul 22 03:07:11.972086 ip-10-0-131-67 kubenswrapper[2582]: W0722 03:07:11.971715 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod877a033d_62c3_46d6_a0d5_f23139602b32.slice/crio-028b0e2e37683c2616a66defaa7cde81971418b9cf66d044d4c37b4b55f8fbdb WatchSource:0}: Error finding container 028b0e2e37683c2616a66defaa7cde81971418b9cf66d044d4c37b4b55f8fbdb: Status 404 returned error can't find the container with id 028b0e2e37683c2616a66defaa7cde81971418b9cf66d044d4c37b4b55f8fbdb Jul 22 03:07:12.025576 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:12.025552 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-szpl4"] Jul 22 03:07:12.028096 ip-10-0-131-67 kubenswrapper[2582]: W0722 03:07:12.028063 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56ca64c4_0973_4ed3_acb1_2d2ebfc80814.slice/crio-d4992d4fd079745b01671976b6e93abe81ab132259fa5f9fc3df7b6084240ab2 WatchSource:0}: Error finding container d4992d4fd079745b01671976b6e93abe81ab132259fa5f9fc3df7b6084240ab2: Status 404 returned error can't find the container with id d4992d4fd079745b01671976b6e93abe81ab132259fa5f9fc3df7b6084240ab2 Jul 22 03:07:12.805185 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:12.805060 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-szpl4" event={"ID":"56ca64c4-0973-4ed3-acb1-2d2ebfc80814","Type":"ContainerStarted","Data":"d4992d4fd079745b01671976b6e93abe81ab132259fa5f9fc3df7b6084240ab2"} Jul 22 03:07:12.810022 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:12.809906 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-tf965" event={"ID":"877a033d-62c3-46d6-a0d5-f23139602b32","Type":"ContainerStarted","Data":"028b0e2e37683c2616a66defaa7cde81971418b9cf66d044d4c37b4b55f8fbdb"} Jul 22 03:07:14.822660 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:14.822625 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-szpl4" event={"ID":"56ca64c4-0973-4ed3-acb1-2d2ebfc80814","Type":"ContainerStarted","Data":"481260ec5d89713d594578f57599f312b5947e2344fb8c3c6489301de8029c3a"} Jul 22 03:07:14.823171 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:14.822828 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-szpl4" Jul 22 03:07:14.824099 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:14.824079 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-tf965" event={"ID":"877a033d-62c3-46d6-a0d5-f23139602b32","Type":"ContainerStarted","Data":"7349fc51c0eaf4480be5fc1ed8546a7e327ccc69635217ec44014e637d26b992"} Jul 22 03:07:14.824202 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:14.824192 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5bf47cd988-tf965" Jul 22 03:07:14.844755 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:14.844688 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7677b774bb-szpl4" podStartSLOduration=1.603933367 podStartE2EDuration="3.844666585s" totalImagesPullingTime="2.240733218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:07:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:07:14.840206634 +0000 UTC m=+545.571827518" watchObservedRunningTime="2026-07-22 03:07:14.844666585 +0000 UTC m=+545.576287436" Jul 22 03:07:14.862562 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:14.862486 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5bf47cd988-tf965" podStartSLOduration=1.574088232 podStartE2EDuration="3.862462645s" totalImagesPullingTime="2.288374413s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:07:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:07:14.860876958 +0000 UTC m=+545.592497805" watchObservedRunningTime="2026-07-22 03:07:14.862462645 +0000 UTC m=+545.594083492" Jul 22 03:07:18.848771 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:18.848688 2582 generic.go:356] "Generic (PLEG): container finished" podID="9ca7bc57-768b-42fd-98df-ddddde7d3b08" containerID="ef22f733354d3a77ab411cea9a6d0fa1196d6770439cc59f7854ebb7814db000" exitCode=0 Jul 22 03:07:18.848771 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:18.848759 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lgdw9" event={"ID":"9ca7bc57-768b-42fd-98df-ddddde7d3b08","Type":"ContainerDied","Data":"ef22f733354d3a77ab411cea9a6d0fa1196d6770439cc59f7854ebb7814db000"} Jul 22 03:07:18.849247 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:18.849165 2582 scope.go:117] "RemoveContainer" containerID="ef22f733354d3a77ab411cea9a6d0fa1196d6770439cc59f7854ebb7814db000" Jul 22 03:07:18.850179 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:18.850153 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 03:07:18.850239 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:18.850220 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 03:07:18.850285 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:18.850269 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 03:07:19.864670 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:19.864623 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lgdw9" event={"ID":"9ca7bc57-768b-42fd-98df-ddddde7d3b08","Type":"ContainerStarted","Data":"4e65af0c8b91f034fe31ea88c397250e8e7c57755e655b49ca0ce9865d6a894b"} Jul 22 03:07:20.411492 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:20.411451 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lgdw9" Jul 22 03:07:20.411891 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:20.411868 2582 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-lgdw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 22 03:07:20.411944 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:20.411923 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lgdw9" podUID="9ca7bc57-768b-42fd-98df-ddddde7d3b08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 22 03:07:21.412339 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:21.412300 2582 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-lgdw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 22 03:07:21.412787 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:21.412355 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lgdw9" podUID="9ca7bc57-768b-42fd-98df-ddddde7d3b08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 22 03:07:21.527616 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:21.527581 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:07:21.632436 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:21.632402 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:07:22.412365 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:22.412324 2582 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-lgdw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 22 03:07:22.412773 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:22.412389 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lgdw9" podUID="9ca7bc57-768b-42fd-98df-ddddde7d3b08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 22 03:07:23.412064 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:23.412004 2582 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-lgdw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 22 03:07:23.412229 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:23.412085 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lgdw9" podUID="9ca7bc57-768b-42fd-98df-ddddde7d3b08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 22 03:07:24.416348 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:24.416318 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lgdw9" Jul 22 03:07:24.416759 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:24.416611 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lgdw9" Jul 22 03:07:24.417405 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:24.417386 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lgdw9" Jul 22 03:07:25.831200 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:25.831167 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-szpl4" Jul 22 03:07:25.831677 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:25.831462 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5bf47cd988-tf965" Jul 22 03:07:29.147820 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:29.147783 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:07:29.787283 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:29.787248 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:07:35.829228 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:35.829188 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:07:42.298436 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:42.298403 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-mnctl"] Jul 22 03:07:42.302279 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:42.302252 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mnctl" Jul 22 03:07:42.305341 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:42.305306 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 22 03:07:42.306193 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:42.306169 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 22 03:07:42.306193 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:42.306188 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-hkxsv\"" Jul 22 03:07:42.306375 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:42.306171 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 22 03:07:42.312379 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:42.312356 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-mnctl"] Jul 22 03:07:42.327703 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:42.327666 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:07:42.350077 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:42.350045 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfa663d5-84bd-49ec-bdf6-6e964a2acc7c-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mnctl\" (UID: \"cfa663d5-84bd-49ec-bdf6-6e964a2acc7c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mnctl" Jul 22 03:07:42.350252 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:42.350091 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfa663d5-84bd-49ec-bdf6-6e964a2acc7c-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mnctl\" (UID: \"cfa663d5-84bd-49ec-bdf6-6e964a2acc7c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mnctl" Jul 22 03:07:42.350252 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:42.350113 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfa663d5-84bd-49ec-bdf6-6e964a2acc7c-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mnctl\" (UID: \"cfa663d5-84bd-49ec-bdf6-6e964a2acc7c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mnctl" Jul 22 03:07:42.350252 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:42.350130 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-975zb\" (UniqueName: \"kubernetes.io/projected/cfa663d5-84bd-49ec-bdf6-6e964a2acc7c-kube-api-access-975zb\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mnctl\" (UID: \"cfa663d5-84bd-49ec-bdf6-6e964a2acc7c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mnctl" Jul 22 03:07:42.350252 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:42.350162 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfa663d5-84bd-49ec-bdf6-6e964a2acc7c-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mnctl\" (UID: \"cfa663d5-84bd-49ec-bdf6-6e964a2acc7c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mnctl" Jul 22 03:07:42.350252 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:42.350191 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfa663d5-84bd-49ec-bdf6-6e964a2acc7c-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mnctl\" (UID: \"cfa663d5-84bd-49ec-bdf6-6e964a2acc7c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mnctl" Jul 22 03:07:42.450703 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:42.450667 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfa663d5-84bd-49ec-bdf6-6e964a2acc7c-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mnctl\" (UID: \"cfa663d5-84bd-49ec-bdf6-6e964a2acc7c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mnctl" Jul 22 03:07:42.450703 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:42.450701 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-975zb\" (UniqueName: \"kubernetes.io/projected/cfa663d5-84bd-49ec-bdf6-6e964a2acc7c-kube-api-access-975zb\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mnctl\" (UID: \"cfa663d5-84bd-49ec-bdf6-6e964a2acc7c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mnctl" Jul 22 03:07:42.450911 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:42.450735 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfa663d5-84bd-49ec-bdf6-6e964a2acc7c-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mnctl\" (UID: \"cfa663d5-84bd-49ec-bdf6-6e964a2acc7c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mnctl" Jul 22 03:07:42.450911 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:42.450767 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfa663d5-84bd-49ec-bdf6-6e964a2acc7c-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mnctl\" (UID: \"cfa663d5-84bd-49ec-bdf6-6e964a2acc7c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mnctl" Jul 22 03:07:42.450911 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:42.450810 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfa663d5-84bd-49ec-bdf6-6e964a2acc7c-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mnctl\" (UID: \"cfa663d5-84bd-49ec-bdf6-6e964a2acc7c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mnctl" Jul 22 03:07:42.450911 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:42.450859 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfa663d5-84bd-49ec-bdf6-6e964a2acc7c-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mnctl\" (UID: \"cfa663d5-84bd-49ec-bdf6-6e964a2acc7c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mnctl" Jul 22 03:07:42.451282 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:42.451263 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfa663d5-84bd-49ec-bdf6-6e964a2acc7c-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mnctl\" (UID: \"cfa663d5-84bd-49ec-bdf6-6e964a2acc7c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mnctl" Jul 22 03:07:42.451355 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:42.451309 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfa663d5-84bd-49ec-bdf6-6e964a2acc7c-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mnctl\" (UID: \"cfa663d5-84bd-49ec-bdf6-6e964a2acc7c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mnctl" Jul 22 03:07:42.451355 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:42.451322 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfa663d5-84bd-49ec-bdf6-6e964a2acc7c-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mnctl\" (UID: \"cfa663d5-84bd-49ec-bdf6-6e964a2acc7c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mnctl" Jul 22 03:07:42.453157 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:42.453130 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfa663d5-84bd-49ec-bdf6-6e964a2acc7c-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mnctl\" (UID: \"cfa663d5-84bd-49ec-bdf6-6e964a2acc7c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mnctl" Jul 22 03:07:42.453284 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:42.453270 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfa663d5-84bd-49ec-bdf6-6e964a2acc7c-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mnctl\" (UID: \"cfa663d5-84bd-49ec-bdf6-6e964a2acc7c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mnctl" Jul 22 03:07:42.458786 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:42.458757 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-975zb\" (UniqueName: \"kubernetes.io/projected/cfa663d5-84bd-49ec-bdf6-6e964a2acc7c-kube-api-access-975zb\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mnctl\" (UID: \"cfa663d5-84bd-49ec-bdf6-6e964a2acc7c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mnctl" Jul 22 03:07:42.614845 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:42.614751 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mnctl" Jul 22 03:07:42.751609 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:42.751572 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-mnctl"] Jul 22 03:07:42.755235 ip-10-0-131-67 kubenswrapper[2582]: W0722 03:07:42.755199 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfa663d5_84bd_49ec_bdf6_6e964a2acc7c.slice/crio-aedb361129f8589da481b1d2a5978749fbc5080f055765443c4d210c57caef7a WatchSource:0}: Error finding container aedb361129f8589da481b1d2a5978749fbc5080f055765443c4d210c57caef7a: Status 404 returned error can't find the container with id aedb361129f8589da481b1d2a5978749fbc5080f055765443c4d210c57caef7a Jul 22 03:07:42.999969 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:42.999933 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mnctl" event={"ID":"cfa663d5-84bd-49ec-bdf6-6e964a2acc7c","Type":"ContainerStarted","Data":"aedb361129f8589da481b1d2a5978749fbc5080f055765443c4d210c57caef7a"} Jul 22 03:07:43.230347 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:43.230311 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:07:44.007794 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:44.007739 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mnctl" event={"ID":"cfa663d5-84bd-49ec-bdf6-6e964a2acc7c","Type":"ContainerStarted","Data":"fdbfb87cf90179a5378bc3c7ee1e139e4273ad9313c1a36be3968f6155d9de79"} Jul 22 03:07:44.027239 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:44.027183 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mnctl" podStartSLOduration=0.931577255 podStartE2EDuration="2.027165134s" totalImagesPullingTime="1.095587879s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:07:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:07:44.025945558 +0000 UTC m=+574.757566417" watchObservedRunningTime="2026-07-22 03:07:44.027165134 +0000 UTC m=+574.758785981" Jul 22 03:07:52.615362 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:52.615313 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mnctl" Jul 22 03:07:52.615839 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:52.615394 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mnctl" Jul 22 03:07:52.628058 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:52.628035 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mnctl" Jul 22 03:07:53.070199 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:07:53.070168 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mnctl" Jul 22 03:08:01.952702 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:08:01.952670 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:08:03.095466 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:08:03.095432 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:08:13.741667 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:08:13.741635 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:08:13.814399 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:08:13.814370 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:08:18.331819 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:08:18.331776 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:08:20.179703 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:08:20.179666 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:08:33.042260 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:08:33.042173 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:08:35.986068 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:08:35.986034 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:08:39.296232 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:08:39.296197 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-jcc2f"] Jul 22 03:08:39.299074 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:08:39.299053 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jcc2f" Jul 22 03:08:39.301603 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:08:39.301583 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 22 03:08:39.313799 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:08:39.313773 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-jcc2f"] Jul 22 03:08:39.333615 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:08:39.333587 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:08:39.369595 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:08:39.369560 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18577aca-6f29-47a0-ac67-697b298b664d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jcc2f\" (UID: \"18577aca-6f29-47a0-ac67-697b298b664d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jcc2f" Jul 22 03:08:39.369767 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:08:39.369603 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18577aca-6f29-47a0-ac67-697b298b664d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jcc2f\" (UID: \"18577aca-6f29-47a0-ac67-697b298b664d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jcc2f" Jul 22 03:08:39.369767 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:08:39.369672 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18577aca-6f29-47a0-ac67-697b298b664d-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jcc2f\" (UID: \"18577aca-6f29-47a0-ac67-697b298b664d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jcc2f" Jul 22 03:08:39.369767 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:08:39.369702 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzlhm\" (UniqueName: \"kubernetes.io/projected/18577aca-6f29-47a0-ac67-697b298b664d-kube-api-access-xzlhm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jcc2f\" (UID: \"18577aca-6f29-47a0-ac67-697b298b664d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jcc2f" Jul 22 03:08:39.369767 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:08:39.369732 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18577aca-6f29-47a0-ac67-697b298b664d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jcc2f\" (UID: \"18577aca-6f29-47a0-ac67-697b298b664d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jcc2f" Jul 22 03:08:39.369981 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:08:39.369851 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18577aca-6f29-47a0-ac67-697b298b664d-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jcc2f\" (UID: \"18577aca-6f29-47a0-ac67-697b298b664d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jcc2f" Jul 22 03:08:39.470965 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:08:39.470936 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18577aca-6f29-47a0-ac67-697b298b664d-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jcc2f\" (UID: \"18577aca-6f29-47a0-ac67-697b298b664d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jcc2f" Jul 22 03:08:39.471161 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:08:39.470970 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzlhm\" (UniqueName: \"kubernetes.io/projected/18577aca-6f29-47a0-ac67-697b298b664d-kube-api-access-xzlhm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jcc2f\" (UID: \"18577aca-6f29-47a0-ac67-697b298b664d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jcc2f" Jul 22 03:08:39.471161 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:08:39.470998 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18577aca-6f29-47a0-ac67-697b298b664d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jcc2f\" (UID: \"18577aca-6f29-47a0-ac67-697b298b664d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jcc2f" Jul 22 03:08:39.471161 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:08:39.471069 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18577aca-6f29-47a0-ac67-697b298b664d-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jcc2f\" (UID: \"18577aca-6f29-47a0-ac67-697b298b664d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jcc2f" Jul 22 03:08:39.471161 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:08:39.471110 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18577aca-6f29-47a0-ac67-697b298b664d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jcc2f\" (UID: \"18577aca-6f29-47a0-ac67-697b298b664d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jcc2f" Jul 22 03:08:39.471161 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:08:39.471142 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18577aca-6f29-47a0-ac67-697b298b664d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jcc2f\" (UID: \"18577aca-6f29-47a0-ac67-697b298b664d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jcc2f" Jul 22 03:08:39.471507 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:08:39.471477 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18577aca-6f29-47a0-ac67-697b298b664d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jcc2f\" (UID: \"18577aca-6f29-47a0-ac67-697b298b664d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jcc2f" Jul 22 03:08:39.471636 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:08:39.471520 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18577aca-6f29-47a0-ac67-697b298b664d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jcc2f\" (UID: \"18577aca-6f29-47a0-ac67-697b298b664d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jcc2f" Jul 22 03:08:39.471636 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:08:39.471555 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18577aca-6f29-47a0-ac67-697b298b664d-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jcc2f\" (UID: \"18577aca-6f29-47a0-ac67-697b298b664d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jcc2f" Jul 22 03:08:39.473503 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:08:39.473484 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18577aca-6f29-47a0-ac67-697b298b664d-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jcc2f\" (UID: \"18577aca-6f29-47a0-ac67-697b298b664d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jcc2f" Jul 22 03:08:39.473627 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:08:39.473610 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18577aca-6f29-47a0-ac67-697b298b664d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jcc2f\" (UID: \"18577aca-6f29-47a0-ac67-697b298b664d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jcc2f" Jul 22 03:08:39.480457 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:08:39.480433 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzlhm\" (UniqueName: \"kubernetes.io/projected/18577aca-6f29-47a0-ac67-697b298b664d-kube-api-access-xzlhm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jcc2f\" (UID: \"18577aca-6f29-47a0-ac67-697b298b664d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jcc2f" Jul 22 03:08:39.608655 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:08:39.608576 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jcc2f" Jul 22 03:08:39.751055 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:08:39.751027 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-jcc2f"] Jul 22 03:08:39.753342 ip-10-0-131-67 kubenswrapper[2582]: W0722 03:08:39.753315 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod18577aca_6f29_47a0_ac67_697b298b664d.slice/crio-5cbf7a76322c75b0264b337aaaaccd9b007bbc8e6d9eaa1fccccbb8a0226a427 WatchSource:0}: Error finding container 5cbf7a76322c75b0264b337aaaaccd9b007bbc8e6d9eaa1fccccbb8a0226a427: Status 404 returned error can't find the container with id 5cbf7a76322c75b0264b337aaaaccd9b007bbc8e6d9eaa1fccccbb8a0226a427 Jul 22 03:08:39.754621 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:08:39.754603 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 03:08:40.326929 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:08:40.326829 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jcc2f" event={"ID":"18577aca-6f29-47a0-ac67-697b298b664d","Type":"ContainerStarted","Data":"2eead103217f1b487b4d304e64494338277a277e1fee0738877c645226a85dc4"} Jul 22 03:08:40.326929 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:08:40.326872 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jcc2f" event={"ID":"18577aca-6f29-47a0-ac67-697b298b664d","Type":"ContainerStarted","Data":"5cbf7a76322c75b0264b337aaaaccd9b007bbc8e6d9eaa1fccccbb8a0226a427"} Jul 22 03:08:40.346837 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:08:40.346788 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jcc2f" podStartSLOduration=1.173821272 podStartE2EDuration="1.346771884s" totalImagesPullingTime="172.950612ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:08:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:08:40.345616547 +0000 UTC m=+631.077237394" watchObservedRunningTime="2026-07-22 03:08:40.346771884 +0000 UTC m=+631.078392730" Jul 22 03:08:43.034390 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:08:43.034357 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:08:43.116346 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:08:43.116313 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:08:43.905435 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:08:43.905401 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:08:49.608977 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:08:49.608937 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jcc2f" Jul 22 03:08:49.608977 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:08:49.608984 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jcc2f" Jul 22 03:08:49.621608 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:08:49.621581 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jcc2f" Jul 22 03:08:50.398824 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:08:50.398795 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jcc2f" Jul 22 03:09:28.619632 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:09:28.619601 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:09:37.517320 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:09:37.517288 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:09:37.622266 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:09:37.622236 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:09:38.694667 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:09:38.694640 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:09:38.790770 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:09:38.790737 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:10:02.910457 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:10:02.910375 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:10:08.668493 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:10:08.668462 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:10:17.776962 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:10:17.776924 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:10:18.623038 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:10:18.622980 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:10:19.566933 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:10:19.566892 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:10:19.666294 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:10:19.666258 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:10:25.658639 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:10:25.658601 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6446b4cb87-mk6rs"] Jul 22 03:10:25.659253 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:10:25.659206 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6446b4cb87-mk6rs" podUID="76cb9a05-537f-45d7-bb11-facc17791c13" containerName="manager" containerID="cri-o://59c715cd73f62129834f86b54e3a687d6762558d4de0398200298edbd6f9c3de" gracePeriod=30 Jul 22 03:10:25.920384 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:10:25.920327 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6446b4cb87-mk6rs" Jul 22 03:10:25.931130 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:10:25.931099 2582 generic.go:356] "Generic (PLEG): container finished" podID="76cb9a05-537f-45d7-bb11-facc17791c13" containerID="59c715cd73f62129834f86b54e3a687d6762558d4de0398200298edbd6f9c3de" exitCode=0 Jul 22 03:10:25.931264 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:10:25.931157 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6446b4cb87-mk6rs" event={"ID":"76cb9a05-537f-45d7-bb11-facc17791c13","Type":"ContainerDied","Data":"59c715cd73f62129834f86b54e3a687d6762558d4de0398200298edbd6f9c3de"} Jul 22 03:10:25.931264 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:10:25.931165 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6446b4cb87-mk6rs" Jul 22 03:10:25.931264 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:10:25.931208 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6446b4cb87-mk6rs" event={"ID":"76cb9a05-537f-45d7-bb11-facc17791c13","Type":"ContainerDied","Data":"18282631d84e21c9abd12d4d5bcd4dd46007c6519686bbf8064b2de41c7557a8"} Jul 22 03:10:25.931264 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:10:25.931238 2582 scope.go:117] "RemoveContainer" containerID="59c715cd73f62129834f86b54e3a687d6762558d4de0398200298edbd6f9c3de" Jul 22 03:10:25.941900 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:10:25.941878 2582 scope.go:117] "RemoveContainer" containerID="59c715cd73f62129834f86b54e3a687d6762558d4de0398200298edbd6f9c3de" Jul 22 03:10:25.942319 ip-10-0-131-67 kubenswrapper[2582]: E0722 03:10:25.942296 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59c715cd73f62129834f86b54e3a687d6762558d4de0398200298edbd6f9c3de\": container with ID starting with 59c715cd73f62129834f86b54e3a687d6762558d4de0398200298edbd6f9c3de not found: ID does not exist" containerID="59c715cd73f62129834f86b54e3a687d6762558d4de0398200298edbd6f9c3de" Jul 22 03:10:25.942410 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:10:25.942331 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59c715cd73f62129834f86b54e3a687d6762558d4de0398200298edbd6f9c3de"} err="failed to get container status \"59c715cd73f62129834f86b54e3a687d6762558d4de0398200298edbd6f9c3de\": rpc error: code = NotFound desc = could not find container \"59c715cd73f62129834f86b54e3a687d6762558d4de0398200298edbd6f9c3de\": container with ID starting with 59c715cd73f62129834f86b54e3a687d6762558d4de0398200298edbd6f9c3de not found: ID does not exist" Jul 22 03:10:26.057613 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:10:26.057583 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/76cb9a05-537f-45d7-bb11-facc17791c13-webhook-cert\") pod \"76cb9a05-537f-45d7-bb11-facc17791c13\" (UID: \"76cb9a05-537f-45d7-bb11-facc17791c13\") " Jul 22 03:10:26.057823 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:10:26.057686 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5fvrj\" (UniqueName: \"kubernetes.io/projected/76cb9a05-537f-45d7-bb11-facc17791c13-kube-api-access-5fvrj\") pod \"76cb9a05-537f-45d7-bb11-facc17791c13\" (UID: \"76cb9a05-537f-45d7-bb11-facc17791c13\") " Jul 22 03:10:26.059772 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:10:26.059734 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76cb9a05-537f-45d7-bb11-facc17791c13-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "76cb9a05-537f-45d7-bb11-facc17791c13" (UID: "76cb9a05-537f-45d7-bb11-facc17791c13"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 03:10:26.059884 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:10:26.059825 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76cb9a05-537f-45d7-bb11-facc17791c13-kube-api-access-5fvrj" (OuterVolumeSpecName: "kube-api-access-5fvrj") pod "76cb9a05-537f-45d7-bb11-facc17791c13" (UID: "76cb9a05-537f-45d7-bb11-facc17791c13"). InnerVolumeSpecName "kube-api-access-5fvrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 03:10:26.159132 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:10:26.159100 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fvrj\" (UniqueName: \"kubernetes.io/projected/76cb9a05-537f-45d7-bb11-facc17791c13-kube-api-access-5fvrj\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 03:10:26.159132 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:10:26.159129 2582 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/76cb9a05-537f-45d7-bb11-facc17791c13-webhook-cert\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 03:10:26.254707 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:10:26.254658 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6446b4cb87-mk6rs"] Jul 22 03:10:26.257984 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:10:26.257958 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6446b4cb87-mk6rs"] Jul 22 03:10:27.901971 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:10:27.901931 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76cb9a05-537f-45d7-bb11-facc17791c13" path="/var/lib/kubelet/pods/76cb9a05-537f-45d7-bb11-facc17791c13/volumes" Jul 22 03:10:28.371435 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:10:28.371343 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:11:27.126288 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:11:27.126254 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6446b4cb87-xfkfz"] Jul 22 03:11:27.126794 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:11:27.126655 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76cb9a05-537f-45d7-bb11-facc17791c13" containerName="manager" Jul 22 03:11:27.126794 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:11:27.126666 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="76cb9a05-537f-45d7-bb11-facc17791c13" containerName="manager" Jul 22 03:11:27.126794 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:11:27.126750 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="76cb9a05-537f-45d7-bb11-facc17791c13" containerName="manager" Jul 22 03:11:27.129302 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:11:27.129284 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6446b4cb87-xfkfz" Jul 22 03:11:27.132380 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:11:27.132355 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 22 03:11:27.132500 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:11:27.132378 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 22 03:11:27.132500 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:11:27.132355 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-fmzc5\"" Jul 22 03:11:27.139634 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:11:27.139613 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6446b4cb87-xfkfz"] Jul 22 03:11:27.315919 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:11:27.315878 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0a1b982a-6648-4e77-a334-b24d3950948b-webhook-cert\") pod \"maas-controller-6446b4cb87-xfkfz\" (UID: \"0a1b982a-6648-4e77-a334-b24d3950948b\") " pod="opendatahub/maas-controller-6446b4cb87-xfkfz" Jul 22 03:11:27.316164 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:11:27.316078 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7phk8\" (UniqueName: \"kubernetes.io/projected/0a1b982a-6648-4e77-a334-b24d3950948b-kube-api-access-7phk8\") pod \"maas-controller-6446b4cb87-xfkfz\" (UID: \"0a1b982a-6648-4e77-a334-b24d3950948b\") " pod="opendatahub/maas-controller-6446b4cb87-xfkfz" Jul 22 03:11:27.417596 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:11:27.417508 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7phk8\" (UniqueName: \"kubernetes.io/projected/0a1b982a-6648-4e77-a334-b24d3950948b-kube-api-access-7phk8\") pod \"maas-controller-6446b4cb87-xfkfz\" (UID: \"0a1b982a-6648-4e77-a334-b24d3950948b\") " pod="opendatahub/maas-controller-6446b4cb87-xfkfz" Jul 22 03:11:27.417596 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:11:27.417551 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0a1b982a-6648-4e77-a334-b24d3950948b-webhook-cert\") pod \"maas-controller-6446b4cb87-xfkfz\" (UID: \"0a1b982a-6648-4e77-a334-b24d3950948b\") " pod="opendatahub/maas-controller-6446b4cb87-xfkfz" Jul 22 03:11:27.420023 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:11:27.419984 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0a1b982a-6648-4e77-a334-b24d3950948b-webhook-cert\") pod \"maas-controller-6446b4cb87-xfkfz\" (UID: \"0a1b982a-6648-4e77-a334-b24d3950948b\") " pod="opendatahub/maas-controller-6446b4cb87-xfkfz" Jul 22 03:11:27.425928 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:11:27.425904 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7phk8\" (UniqueName: \"kubernetes.io/projected/0a1b982a-6648-4e77-a334-b24d3950948b-kube-api-access-7phk8\") pod \"maas-controller-6446b4cb87-xfkfz\" (UID: \"0a1b982a-6648-4e77-a334-b24d3950948b\") " pod="opendatahub/maas-controller-6446b4cb87-xfkfz" Jul 22 03:11:27.440917 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:11:27.440891 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6446b4cb87-xfkfz" Jul 22 03:11:27.581678 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:11:27.581654 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6446b4cb87-xfkfz"] Jul 22 03:11:27.583868 ip-10-0-131-67 kubenswrapper[2582]: W0722 03:11:27.583840 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a1b982a_6648_4e77_a334_b24d3950948b.slice/crio-b13cc508f6c927c799992d7c2a2e67366e492f06064edf48121eaca201b57d2e WatchSource:0}: Error finding container b13cc508f6c927c799992d7c2a2e67366e492f06064edf48121eaca201b57d2e: Status 404 returned error can't find the container with id b13cc508f6c927c799992d7c2a2e67366e492f06064edf48121eaca201b57d2e Jul 22 03:11:28.298589 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:11:28.298485 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6446b4cb87-xfkfz" event={"ID":"0a1b982a-6648-4e77-a334-b24d3950948b","Type":"ContainerStarted","Data":"6b4d62d2f140623fc89a60aaf6492b75b8285a13975ecd20dd5825bef8d11412"} Jul 22 03:11:28.298589 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:11:28.298539 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6446b4cb87-xfkfz" event={"ID":"0a1b982a-6648-4e77-a334-b24d3950948b","Type":"ContainerStarted","Data":"b13cc508f6c927c799992d7c2a2e67366e492f06064edf48121eaca201b57d2e"} Jul 22 03:11:28.298589 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:11:28.298568 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6446b4cb87-xfkfz" Jul 22 03:11:28.317201 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:11:28.317140 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6446b4cb87-xfkfz" podStartSLOduration=0.897663593 podStartE2EDuration="1.317106791s" totalImagesPullingTime="419.443198ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:11:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:11:28.316476461 +0000 UTC m=+799.048097333" watchObservedRunningTime="2026-07-22 03:11:28.317106791 +0000 UTC m=+799.048727639" Jul 22 03:11:39.305945 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:11:39.305908 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6446b4cb87-xfkfz" Jul 22 03:11:43.894599 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:11:43.894559 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:11:44.016402 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:11:44.016360 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6446b4cb87-xfkfz"] Jul 22 03:11:44.016665 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:11:44.016639 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6446b4cb87-xfkfz" podUID="0a1b982a-6648-4e77-a334-b24d3950948b" containerName="manager" containerID="cri-o://6b4d62d2f140623fc89a60aaf6492b75b8285a13975ecd20dd5825bef8d11412" gracePeriod=30 Jul 22 03:11:44.266422 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:11:44.266395 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6446b4cb87-xfkfz" Jul 22 03:11:44.347513 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:11:44.347477 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0a1b982a-6648-4e77-a334-b24d3950948b-webhook-cert\") pod \"0a1b982a-6648-4e77-a334-b24d3950948b\" (UID: \"0a1b982a-6648-4e77-a334-b24d3950948b\") " Jul 22 03:11:44.347701 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:11:44.347541 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7phk8\" (UniqueName: \"kubernetes.io/projected/0a1b982a-6648-4e77-a334-b24d3950948b-kube-api-access-7phk8\") pod \"0a1b982a-6648-4e77-a334-b24d3950948b\" (UID: \"0a1b982a-6648-4e77-a334-b24d3950948b\") " Jul 22 03:11:44.349703 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:11:44.349661 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a1b982a-6648-4e77-a334-b24d3950948b-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "0a1b982a-6648-4e77-a334-b24d3950948b" (UID: "0a1b982a-6648-4e77-a334-b24d3950948b"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 03:11:44.349844 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:11:44.349771 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a1b982a-6648-4e77-a334-b24d3950948b-kube-api-access-7phk8" (OuterVolumeSpecName: "kube-api-access-7phk8") pod "0a1b982a-6648-4e77-a334-b24d3950948b" (UID: "0a1b982a-6648-4e77-a334-b24d3950948b"). InnerVolumeSpecName "kube-api-access-7phk8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 03:11:44.390250 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:11:44.390213 2582 generic.go:356] "Generic (PLEG): container finished" podID="0a1b982a-6648-4e77-a334-b24d3950948b" containerID="6b4d62d2f140623fc89a60aaf6492b75b8285a13975ecd20dd5825bef8d11412" exitCode=0 Jul 22 03:11:44.390418 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:11:44.390263 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6446b4cb87-xfkfz" Jul 22 03:11:44.390418 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:11:44.390295 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6446b4cb87-xfkfz" event={"ID":"0a1b982a-6648-4e77-a334-b24d3950948b","Type":"ContainerDied","Data":"6b4d62d2f140623fc89a60aaf6492b75b8285a13975ecd20dd5825bef8d11412"} Jul 22 03:11:44.390418 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:11:44.390346 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6446b4cb87-xfkfz" event={"ID":"0a1b982a-6648-4e77-a334-b24d3950948b","Type":"ContainerDied","Data":"b13cc508f6c927c799992d7c2a2e67366e492f06064edf48121eaca201b57d2e"} Jul 22 03:11:44.390418 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:11:44.390368 2582 scope.go:117] "RemoveContainer" containerID="6b4d62d2f140623fc89a60aaf6492b75b8285a13975ecd20dd5825bef8d11412" Jul 22 03:11:44.404301 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:11:44.404284 2582 scope.go:117] "RemoveContainer" containerID="6b4d62d2f140623fc89a60aaf6492b75b8285a13975ecd20dd5825bef8d11412" Jul 22 03:11:44.404609 ip-10-0-131-67 kubenswrapper[2582]: E0722 03:11:44.404591 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b4d62d2f140623fc89a60aaf6492b75b8285a13975ecd20dd5825bef8d11412\": container with ID starting with 6b4d62d2f140623fc89a60aaf6492b75b8285a13975ecd20dd5825bef8d11412 not found: ID does not exist" containerID="6b4d62d2f140623fc89a60aaf6492b75b8285a13975ecd20dd5825bef8d11412" Jul 22 03:11:44.404699 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:11:44.404619 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b4d62d2f140623fc89a60aaf6492b75b8285a13975ecd20dd5825bef8d11412"} err="failed to get container status \"6b4d62d2f140623fc89a60aaf6492b75b8285a13975ecd20dd5825bef8d11412\": rpc error: code = NotFound desc = could not find container \"6b4d62d2f140623fc89a60aaf6492b75b8285a13975ecd20dd5825bef8d11412\": container with ID starting with 6b4d62d2f140623fc89a60aaf6492b75b8285a13975ecd20dd5825bef8d11412 not found: ID does not exist" Jul 22 03:11:44.417438 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:11:44.417413 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6446b4cb87-xfkfz"] Jul 22 03:11:44.421062 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:11:44.421039 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6446b4cb87-xfkfz"] Jul 22 03:11:44.448544 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:11:44.448462 2582 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0a1b982a-6648-4e77-a334-b24d3950948b-webhook-cert\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 03:11:44.448544 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:11:44.448488 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7phk8\" (UniqueName: \"kubernetes.io/projected/0a1b982a-6648-4e77-a334-b24d3950948b-kube-api-access-7phk8\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 03:11:45.168625 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:11:45.168595 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:11:45.298594 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:11:45.298529 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6446b4cb87-mx6mt"] Jul 22 03:11:45.299151 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:11:45.299127 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a1b982a-6648-4e77-a334-b24d3950948b" containerName="manager" Jul 22 03:11:45.299151 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:11:45.299147 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a1b982a-6648-4e77-a334-b24d3950948b" containerName="manager" Jul 22 03:11:45.299368 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:11:45.299248 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a1b982a-6648-4e77-a334-b24d3950948b" containerName="manager" Jul 22 03:11:45.301965 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:11:45.301944 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6446b4cb87-mx6mt" Jul 22 03:11:45.304420 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:11:45.304372 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 22 03:11:45.305279 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:11:45.305265 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 22 03:11:45.305593 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:11:45.305576 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-fmzc5\"" Jul 22 03:11:45.313646 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:11:45.313622 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6446b4cb87-mx6mt"] Jul 22 03:11:45.355585 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:11:45.355551 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ef264272-2474-4344-9d30-5ad28556cae8-webhook-cert\") pod \"maas-controller-6446b4cb87-mx6mt\" (UID: \"ef264272-2474-4344-9d30-5ad28556cae8\") " pod="opendatahub/maas-controller-6446b4cb87-mx6mt" Jul 22 03:11:45.355767 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:11:45.355598 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwdlr\" (UniqueName: \"kubernetes.io/projected/ef264272-2474-4344-9d30-5ad28556cae8-kube-api-access-wwdlr\") pod \"maas-controller-6446b4cb87-mx6mt\" (UID: \"ef264272-2474-4344-9d30-5ad28556cae8\") " pod="opendatahub/maas-controller-6446b4cb87-mx6mt" Jul 22 03:11:45.456238 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:11:45.456147 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ef264272-2474-4344-9d30-5ad28556cae8-webhook-cert\") pod \"maas-controller-6446b4cb87-mx6mt\" (UID: \"ef264272-2474-4344-9d30-5ad28556cae8\") " pod="opendatahub/maas-controller-6446b4cb87-mx6mt" Jul 22 03:11:45.456238 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:11:45.456206 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwdlr\" (UniqueName: \"kubernetes.io/projected/ef264272-2474-4344-9d30-5ad28556cae8-kube-api-access-wwdlr\") pod \"maas-controller-6446b4cb87-mx6mt\" (UID: \"ef264272-2474-4344-9d30-5ad28556cae8\") " pod="opendatahub/maas-controller-6446b4cb87-mx6mt" Jul 22 03:11:45.458701 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:11:45.458677 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ef264272-2474-4344-9d30-5ad28556cae8-webhook-cert\") pod \"maas-controller-6446b4cb87-mx6mt\" (UID: \"ef264272-2474-4344-9d30-5ad28556cae8\") " pod="opendatahub/maas-controller-6446b4cb87-mx6mt" Jul 22 03:11:45.465263 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:11:45.465238 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwdlr\" (UniqueName: \"kubernetes.io/projected/ef264272-2474-4344-9d30-5ad28556cae8-kube-api-access-wwdlr\") pod \"maas-controller-6446b4cb87-mx6mt\" (UID: \"ef264272-2474-4344-9d30-5ad28556cae8\") " pod="opendatahub/maas-controller-6446b4cb87-mx6mt" Jul 22 03:11:45.615201 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:11:45.615162 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6446b4cb87-mx6mt" Jul 22 03:11:45.752694 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:11:45.752663 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6446b4cb87-mx6mt"] Jul 22 03:11:45.755208 ip-10-0-131-67 kubenswrapper[2582]: W0722 03:11:45.755181 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef264272_2474_4344_9d30_5ad28556cae8.slice/crio-983e8f45684d269a3e002f65d776c16a13f08776894f9ad9af9f23a2749a3580 WatchSource:0}: Error finding container 983e8f45684d269a3e002f65d776c16a13f08776894f9ad9af9f23a2749a3580: Status 404 returned error can't find the container with id 983e8f45684d269a3e002f65d776c16a13f08776894f9ad9af9f23a2749a3580 Jul 22 03:11:45.903114 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:11:45.903082 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a1b982a-6648-4e77-a334-b24d3950948b" path="/var/lib/kubelet/pods/0a1b982a-6648-4e77-a334-b24d3950948b/volumes" Jul 22 03:11:46.404368 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:11:46.404335 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6446b4cb87-mx6mt" event={"ID":"ef264272-2474-4344-9d30-5ad28556cae8","Type":"ContainerStarted","Data":"638ccae0b6cd8dc52dd4280e126f7e23e6b9d481e25048e5f7dcb3845f464b1c"} Jul 22 03:11:46.404748 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:11:46.404385 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6446b4cb87-mx6mt" event={"ID":"ef264272-2474-4344-9d30-5ad28556cae8","Type":"ContainerStarted","Data":"983e8f45684d269a3e002f65d776c16a13f08776894f9ad9af9f23a2749a3580"} Jul 22 03:11:46.404748 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:11:46.404467 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6446b4cb87-mx6mt" Jul 22 03:11:46.422866 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:11:46.422818 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6446b4cb87-mx6mt" podStartSLOduration=0.987822442 podStartE2EDuration="1.422801792s" totalImagesPullingTime="434.97935ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:11:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:11:46.421391905 +0000 UTC m=+817.153012773" watchObservedRunningTime="2026-07-22 03:11:46.422801792 +0000 UTC m=+817.154422700" Jul 22 03:11:57.412611 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:11:57.412582 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6446b4cb87-mx6mt" Jul 22 03:12:09.682947 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:12:09.682911 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:12:17.686729 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:12:17.682502 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:12:21.142104 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:12:21.142051 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:12:21.242324 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:12:21.242286 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:12:21.389465 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:12:21.389418 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:12:27.715966 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:12:27.715919 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:12:27.780265 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:12:27.780232 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:12:27.860160 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:12:27.860122 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:12:27.961246 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:12:27.961218 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:12:28.081361 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:12:28.081253 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:12:34.123369 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:12:34.123338 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:12:34.185566 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:12:34.185514 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:12:51.582378 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:12:51.582346 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:13:00.974560 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:13:00.974519 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:13:11.089875 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:13:11.089841 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:13:20.971407 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:13:20.971373 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:13:30.600524 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:13:30.600488 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:13:41.907742 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:13:41.907706 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:13:53.579092 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:13:53.579052 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:14:03.083121 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:14:03.082509 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:14:26.009676 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:14:26.009639 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:14:35.005958 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:14:35.005872 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:14:43.862076 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:14:43.862034 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:15:00.491464 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:15:00.491420 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:15:11.386065 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:15:11.386025 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:15:27.290187 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:15:27.290140 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:15:36.838427 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:15:36.838388 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:15:47.776104 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:15:47.776069 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:15:57.284329 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:15:57.284295 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:16:06.871830 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:16:06.871749 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:16:41.570176 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:16:41.570142 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:17:54.179671 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:17:54.179632 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:18:06.074382 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:18:06.074348 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:18:14.776729 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:18:14.776694 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:18:27.175698 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:18:27.175660 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:18:38.670480 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:18:38.670445 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:18:48.266354 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:18:48.266316 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:18:57.271502 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:18:57.271459 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:19:08.180165 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:19:08.180082 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:19:20.266957 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:19:20.266925 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:19:28.975104 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:19:28.975064 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:19:39.573936 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:19:39.573892 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:19:48.361067 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:19:48.361036 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:19:49.591027 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:19:49.590983 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:19:56.750699 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:19:56.750667 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:19:56.851301 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:19:56.851263 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:20:03.775041 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:20:03.774990 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:20:11.969180 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:20:11.969123 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:20:16.250700 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:20:16.250662 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:20:20.969281 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:20:20.969247 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:20:24.005135 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:20:24.005103 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:20:35.880466 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:20:35.880376 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:20:54.540291 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:20:54.540258 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:20:56.078662 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:20:56.078623 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:20:58.389536 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:20:58.389504 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:21:12.502220 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:21:12.502184 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:21:19.497886 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:21:19.497852 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:21:37.600414 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:21:37.600378 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:21:49.092433 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:21:49.092403 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:21:58.815921 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:21:58.815881 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:22:01.375636 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:22:01.375550 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:22:07.799219 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:22:07.799184 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:22:19.286124 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:22:19.286079 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:22:28.855005 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:22:28.854965 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:22:42.358889 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:22:42.358856 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:22:52.108785 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:22:52.108749 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:23:01.877434 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:23:01.877396 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:23:14.172909 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:23:14.172872 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:23:21.885110 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:23:21.885078 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:23:26.385628 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:23:26.385591 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:23:35.515872 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:23:35.515791 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:23:47.335333 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:23:47.335299 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:23:59.278750 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:23:59.278713 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:24:08.387071 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:24:08.387036 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:24:10.823186 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:24:10.823149 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:24:21.370515 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:24:21.370476 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:24:32.473428 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:24:32.473394 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:24:32.577999 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:24:32.577964 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:24:35.668135 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:24:35.668105 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:24:35.763617 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:24:35.763586 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:24:36.163684 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:24:36.163649 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:24:41.275462 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:24:41.275418 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:24:42.189164 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:24:42.189129 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:24:42.288135 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:24:42.288103 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:24:42.441767 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:24:42.441685 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:24:42.778245 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:24:42.778164 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:24:43.012991 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:24:43.012960 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:24:45.427533 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:24:45.427499 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:24:45.526705 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:24:45.526672 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:24:45.655094 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:24:45.655061 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:24:45.774688 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:24:45.774599 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:24:46.342490 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:24:46.342432 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:24:46.443598 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:24:46.443565 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:24:51.378529 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:24:51.378493 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:25:11.120097 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:25:11.119992 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c81ed29-4xg4q"] Jul 22 03:25:11.123641 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:25:11.123617 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c81ed29-4xg4q" Jul 22 03:25:11.125934 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:25:11.125914 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 03:25:11.126088 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:25:11.125969 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cw8l4\"" Jul 22 03:25:11.131318 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:25:11.131285 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c81ed29-4xg4q"] Jul 22 03:25:11.228127 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:25:11.228088 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/194bfcb1-cd70-491f-b36e-17f3289d09b0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0c81ed29-4xg4q\" (UID: \"194bfcb1-cd70-491f-b36e-17f3289d09b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c81ed29-4xg4q" Jul 22 03:25:11.329140 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:25:11.329104 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/194bfcb1-cd70-491f-b36e-17f3289d09b0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0c81ed29-4xg4q\" (UID: \"194bfcb1-cd70-491f-b36e-17f3289d09b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c81ed29-4xg4q" Jul 22 03:25:11.329791 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:25:11.329763 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/194bfcb1-cd70-491f-b36e-17f3289d09b0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0c81ed29-4xg4q\" (UID: \"194bfcb1-cd70-491f-b36e-17f3289d09b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c81ed29-4xg4q" Jul 22 03:25:11.438849 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:25:11.438819 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c81ed29-4xg4q" Jul 22 03:25:11.576889 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:25:11.576858 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c81ed29-4xg4q"] Jul 22 03:25:11.578053 ip-10-0-131-67 kubenswrapper[2582]: W0722 03:25:11.577984 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod194bfcb1_cd70_491f_b36e_17f3289d09b0.slice/crio-e84f9af0f302e783daf3cb092d9a5b7b5d50315bf635e6093e37ee07f01ba370 WatchSource:0}: Error finding container e84f9af0f302e783daf3cb092d9a5b7b5d50315bf635e6093e37ee07f01ba370: Status 404 returned error can't find the container with id e84f9af0f302e783daf3cb092d9a5b7b5d50315bf635e6093e37ee07f01ba370 Jul 22 03:25:11.579855 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:25:11.579838 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 03:25:12.010828 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:25:12.010785 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c81ed29-4xg4q" event={"ID":"194bfcb1-cd70-491f-b36e-17f3289d09b0","Type":"ContainerStarted","Data":"e84f9af0f302e783daf3cb092d9a5b7b5d50315bf635e6093e37ee07f01ba370"} Jul 22 03:25:15.031008 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:25:15.030912 2582 generic.go:356] "Generic (PLEG): container finished" podID="194bfcb1-cd70-491f-b36e-17f3289d09b0" containerID="3ae0c38f4ca92b6030cc9d34ec585a11605c7ec2ac2ee0aa2b9a0f09ff656ef9" exitCode=0 Jul 22 03:25:15.031008 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:25:15.030982 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c81ed29-4xg4q" event={"ID":"194bfcb1-cd70-491f-b36e-17f3289d09b0","Type":"ContainerDied","Data":"3ae0c38f4ca92b6030cc9d34ec585a11605c7ec2ac2ee0aa2b9a0f09ff656ef9"} Jul 22 03:25:16.196803 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:25:16.196766 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c81ed29-4xg4q" Jul 22 03:25:16.270613 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:25:16.270581 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/194bfcb1-cd70-491f-b36e-17f3289d09b0-maas-api-service-ca\") pod \"194bfcb1-cd70-491f-b36e-17f3289d09b0\" (UID: \"194bfcb1-cd70-491f-b36e-17f3289d09b0\") " Jul 22 03:25:16.270921 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:25:16.270898 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/194bfcb1-cd70-491f-b36e-17f3289d09b0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "194bfcb1-cd70-491f-b36e-17f3289d09b0" (UID: "194bfcb1-cd70-491f-b36e-17f3289d09b0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 03:25:16.371785 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:25:16.371699 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/194bfcb1-cd70-491f-b36e-17f3289d09b0-maas-api-service-ca\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 03:25:17.043813 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:25:17.043780 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c81ed29-4xg4q" event={"ID":"194bfcb1-cd70-491f-b36e-17f3289d09b0","Type":"ContainerDied","Data":"e84f9af0f302e783daf3cb092d9a5b7b5d50315bf635e6093e37ee07f01ba370"} Jul 22 03:25:17.043813 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:25:17.043813 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c81ed29-4xg4q" Jul 22 03:25:17.044051 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:25:17.043827 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e84f9af0f302e783daf3cb092d9a5b7b5d50315bf635e6093e37ee07f01ba370" Jul 22 03:25:21.778161 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:25:21.778125 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:25:24.576066 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:25:24.574626 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:25:26.208424 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:25:26.208387 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:25:26.215902 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:25:26.215877 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c81ed29-4xg4q"] Jul 22 03:25:26.218916 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:25:26.218894 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c81ed29-4xg4q"] Jul 22 03:25:26.397341 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:25:26.397314 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:25:26.646373 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:25:26.646338 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:25:26.932411 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:25:26.932302 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:25:27.025170 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:25:27.025139 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:25:27.150958 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:25:27.150920 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:25:27.519003 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:25:27.518972 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:25:27.618861 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:25:27.618827 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:25:27.902330 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:25:27.902292 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="194bfcb1-cd70-491f-b36e-17f3289d09b0" path="/var/lib/kubelet/pods/194bfcb1-cd70-491f-b36e-17f3289d09b0/volumes" Jul 22 03:25:29.767340 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:25:29.767310 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:25:29.886835 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:25:29.886801 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:25:30.227672 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:25:30.227643 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:25:30.470284 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:25:30.470237 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:25:30.571171 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:25:30.571089 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:25:37.076985 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:25:37.076951 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:25:54.853558 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:25:54.853515 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-321cbb2c-hq2dj"] Jul 22 03:25:54.854008 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:25:54.853936 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="194bfcb1-cd70-491f-b36e-17f3289d09b0" containerName="revoke-keys" Jul 22 03:25:54.854008 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:25:54.853947 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="194bfcb1-cd70-491f-b36e-17f3289d09b0" containerName="revoke-keys" Jul 22 03:25:54.854128 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:25:54.854052 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="194bfcb1-cd70-491f-b36e-17f3289d09b0" containerName="revoke-keys" Jul 22 03:25:54.856955 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:25:54.856934 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-321cbb2c-hq2dj" Jul 22 03:25:54.859530 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:25:54.859513 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 03:25:54.859633 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:25:54.859553 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cw8l4\"" Jul 22 03:25:54.868564 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:25:54.868538 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-321cbb2c-hq2dj"] Jul 22 03:25:55.019240 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:25:55.019204 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d6bc3da-e002-4227-8907-1d75ea9a3258-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-321cbb2c-hq2dj\" (UID: \"3d6bc3da-e002-4227-8907-1d75ea9a3258\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-321cbb2c-hq2dj" Jul 22 03:25:55.120659 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:25:55.120579 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d6bc3da-e002-4227-8907-1d75ea9a3258-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-321cbb2c-hq2dj\" (UID: \"3d6bc3da-e002-4227-8907-1d75ea9a3258\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-321cbb2c-hq2dj" Jul 22 03:25:55.121214 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:25:55.121195 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d6bc3da-e002-4227-8907-1d75ea9a3258-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-321cbb2c-hq2dj\" (UID: \"3d6bc3da-e002-4227-8907-1d75ea9a3258\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-321cbb2c-hq2dj" Jul 22 03:25:55.167384 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:25:55.167347 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-321cbb2c-hq2dj" Jul 22 03:25:55.321451 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:25:55.321420 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-321cbb2c-hq2dj"] Jul 22 03:25:55.323647 ip-10-0-131-67 kubenswrapper[2582]: W0722 03:25:55.323615 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d6bc3da_e002_4227_8907_1d75ea9a3258.slice/crio-9d753898fe45371b5d91e348085ada7479ddcb700b059e2678129f3ade7d4bc0 WatchSource:0}: Error finding container 9d753898fe45371b5d91e348085ada7479ddcb700b059e2678129f3ade7d4bc0: Status 404 returned error can't find the container with id 9d753898fe45371b5d91e348085ada7479ddcb700b059e2678129f3ade7d4bc0 Jul 22 03:25:56.273297 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:25:56.273264 2582 generic.go:356] "Generic (PLEG): container finished" podID="3d6bc3da-e002-4227-8907-1d75ea9a3258" containerID="779a59cc3ffef710d38626f880ddbc3215a59cdad90238514d12bbcff580636c" exitCode=0 Jul 22 03:25:56.273741 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:25:56.273356 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-321cbb2c-hq2dj" event={"ID":"3d6bc3da-e002-4227-8907-1d75ea9a3258","Type":"ContainerDied","Data":"779a59cc3ffef710d38626f880ddbc3215a59cdad90238514d12bbcff580636c"} Jul 22 03:25:56.273741 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:25:56.273422 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-321cbb2c-hq2dj" event={"ID":"3d6bc3da-e002-4227-8907-1d75ea9a3258","Type":"ContainerStarted","Data":"9d753898fe45371b5d91e348085ada7479ddcb700b059e2678129f3ade7d4bc0"} Jul 22 03:25:57.415240 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:25:57.415216 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-321cbb2c-hq2dj" Jul 22 03:25:57.544144 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:25:57.544048 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d6bc3da-e002-4227-8907-1d75ea9a3258-maas-api-service-ca\") pod \"3d6bc3da-e002-4227-8907-1d75ea9a3258\" (UID: \"3d6bc3da-e002-4227-8907-1d75ea9a3258\") " Jul 22 03:25:57.544346 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:25:57.544324 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d6bc3da-e002-4227-8907-1d75ea9a3258-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3d6bc3da-e002-4227-8907-1d75ea9a3258" (UID: "3d6bc3da-e002-4227-8907-1d75ea9a3258"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 03:25:57.544416 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:25:57.544403 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d6bc3da-e002-4227-8907-1d75ea9a3258-maas-api-service-ca\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 03:25:58.287169 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:25:58.287083 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-321cbb2c-hq2dj" Jul 22 03:25:58.287169 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:25:58.287120 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-321cbb2c-hq2dj" event={"ID":"3d6bc3da-e002-4227-8907-1d75ea9a3258","Type":"ContainerDied","Data":"9d753898fe45371b5d91e348085ada7479ddcb700b059e2678129f3ade7d4bc0"} Jul 22 03:25:58.287169 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:25:58.287166 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d753898fe45371b5d91e348085ada7479ddcb700b059e2678129f3ade7d4bc0" Jul 22 03:26:05.530813 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:26:05.530778 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:26:08.497239 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:26:08.497198 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:26:09.951566 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:26:09.951534 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:26:09.953852 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:26:09.953833 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-321cbb2c-hq2dj"] Jul 22 03:26:09.956772 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:26:09.956747 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-321cbb2c-hq2dj"] Jul 22 03:26:10.111853 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:26:10.111821 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:26:10.381476 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:26:10.381440 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:26:10.601284 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:26:10.601254 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:26:10.938073 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:26:10.937986 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:26:11.389304 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:26:11.389276 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:26:11.902258 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:26:11.902227 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d6bc3da-e002-4227-8907-1d75ea9a3258" path="/var/lib/kubelet/pods/3d6bc3da-e002-4227-8907-1d75ea9a3258/volumes" Jul 22 03:26:12.836949 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:26:12.836917 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:26:12.981469 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:26:12.981432 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:26:13.077531 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:26:13.077496 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:26:13.557996 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:26:13.557957 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:26:13.665859 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:26:13.665826 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:26:16.176381 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:26:16.176344 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:26:16.277954 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:26:16.277921 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:26:19.685004 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:26:19.684967 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:26:38.819559 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:26:38.819448 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67a1aa21-kh9cx"] Jul 22 03:26:38.820209 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:26:38.819887 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d6bc3da-e002-4227-8907-1d75ea9a3258" containerName="revoke-keys" Jul 22 03:26:38.820209 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:26:38.819898 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d6bc3da-e002-4227-8907-1d75ea9a3258" containerName="revoke-keys" Jul 22 03:26:38.820209 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:26:38.819966 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d6bc3da-e002-4227-8907-1d75ea9a3258" containerName="revoke-keys" Jul 22 03:26:38.823458 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:26:38.823437 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67a1aa21-kh9cx" Jul 22 03:26:38.827885 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:26:38.827075 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 03:26:38.827885 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:26:38.827136 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cw8l4\"" Jul 22 03:26:38.838251 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:26:38.838218 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67a1aa21-kh9cx"] Jul 22 03:26:38.924479 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:26:38.924433 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a7e2c50-13dd-4da6-a0d2-c71c16ce902e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-67a1aa21-kh9cx\" (UID: \"9a7e2c50-13dd-4da6-a0d2-c71c16ce902e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67a1aa21-kh9cx" Jul 22 03:26:39.025316 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:26:39.025267 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a7e2c50-13dd-4da6-a0d2-c71c16ce902e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-67a1aa21-kh9cx\" (UID: \"9a7e2c50-13dd-4da6-a0d2-c71c16ce902e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67a1aa21-kh9cx" Jul 22 03:26:39.025940 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:26:39.025918 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a7e2c50-13dd-4da6-a0d2-c71c16ce902e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-67a1aa21-kh9cx\" (UID: \"9a7e2c50-13dd-4da6-a0d2-c71c16ce902e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67a1aa21-kh9cx" Jul 22 03:26:39.140477 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:26:39.140439 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67a1aa21-kh9cx" Jul 22 03:26:39.297895 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:26:39.297864 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67a1aa21-kh9cx"] Jul 22 03:26:39.299943 ip-10-0-131-67 kubenswrapper[2582]: W0722 03:26:39.299916 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a7e2c50_13dd_4da6_a0d2_c71c16ce902e.slice/crio-432f4744a297761759e47677d79166b9d2b4e759d4251ff65bbd5de5bdf57e18 WatchSource:0}: Error finding container 432f4744a297761759e47677d79166b9d2b4e759d4251ff65bbd5de5bdf57e18: Status 404 returned error can't find the container with id 432f4744a297761759e47677d79166b9d2b4e759d4251ff65bbd5de5bdf57e18 Jul 22 03:26:39.530357 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:26:39.530267 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67a1aa21-kh9cx" event={"ID":"9a7e2c50-13dd-4da6-a0d2-c71c16ce902e","Type":"ContainerStarted","Data":"432f4744a297761759e47677d79166b9d2b4e759d4251ff65bbd5de5bdf57e18"} Jul 22 03:26:40.538619 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:26:40.538588 2582 generic.go:356] "Generic (PLEG): container finished" podID="9a7e2c50-13dd-4da6-a0d2-c71c16ce902e" containerID="3d72d4547f86b8f98059f3763c79c2aba0ee6c2fa0ceca0212eed6c076b5e07a" exitCode=0 Jul 22 03:26:40.539039 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:26:40.538652 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67a1aa21-kh9cx" event={"ID":"9a7e2c50-13dd-4da6-a0d2-c71c16ce902e","Type":"ContainerDied","Data":"3d72d4547f86b8f98059f3763c79c2aba0ee6c2fa0ceca0212eed6c076b5e07a"} Jul 22 03:26:41.685676 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:26:41.685653 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67a1aa21-kh9cx" Jul 22 03:26:41.851822 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:26:41.851733 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a7e2c50-13dd-4da6-a0d2-c71c16ce902e-maas-api-service-ca\") pod \"9a7e2c50-13dd-4da6-a0d2-c71c16ce902e\" (UID: \"9a7e2c50-13dd-4da6-a0d2-c71c16ce902e\") " Jul 22 03:26:41.852094 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:26:41.852070 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a7e2c50-13dd-4da6-a0d2-c71c16ce902e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9a7e2c50-13dd-4da6-a0d2-c71c16ce902e" (UID: "9a7e2c50-13dd-4da6-a0d2-c71c16ce902e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 03:26:41.952577 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:26:41.952539 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a7e2c50-13dd-4da6-a0d2-c71c16ce902e-maas-api-service-ca\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 03:26:42.551936 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:26:42.551895 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67a1aa21-kh9cx" event={"ID":"9a7e2c50-13dd-4da6-a0d2-c71c16ce902e","Type":"ContainerDied","Data":"432f4744a297761759e47677d79166b9d2b4e759d4251ff65bbd5de5bdf57e18"} Jul 22 03:26:42.551936 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:26:42.551926 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67a1aa21-kh9cx" Jul 22 03:26:42.552168 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:26:42.551961 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="432f4744a297761759e47677d79166b9d2b4e759d4251ff65bbd5de5bdf57e18" Jul 22 03:26:49.445871 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:26:49.445837 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:26:51.984136 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:26:51.984102 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:26:53.905526 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:26:53.905494 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67a1aa21-kh9cx"] Jul 22 03:26:53.909602 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:26:53.909567 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:26:53.910773 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:26:53.910755 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67a1aa21-kh9cx"] Jul 22 03:26:55.233332 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:26:55.233299 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:26:55.469561 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:26:55.469511 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:26:55.717549 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:26:55.717512 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:26:55.902567 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:26:55.902530 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a7e2c50-13dd-4da6-a0d2-c71c16ce902e" path="/var/lib/kubelet/pods/9a7e2c50-13dd-4da6-a0d2-c71c16ce902e/volumes" Jul 22 03:26:56.154950 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:26:56.154891 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:26:56.318921 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:26:56.318880 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:26:58.092081 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:26:58.092037 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:26:58.191704 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:26:58.191669 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:26:58.284215 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:26:58.284183 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:26:58.388498 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:26:58.388466 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:26:58.801678 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:26:58.801593 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:26:58.901801 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:26:58.901770 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:27:04.377572 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:27:04.377533 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:27:07.618322 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:27:07.618283 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:27:07.716660 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:27:07.716626 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:27:19.648987 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:27:19.648951 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:27:19.749584 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:27:19.749552 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:27:23.043444 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:27:23.043403 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7df930d3-8cslw"] Jul 22 03:27:23.043851 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:27:23.043839 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a7e2c50-13dd-4da6-a0d2-c71c16ce902e" containerName="revoke-keys" Jul 22 03:27:23.043899 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:27:23.043853 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a7e2c50-13dd-4da6-a0d2-c71c16ce902e" containerName="revoke-keys" Jul 22 03:27:23.043934 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:27:23.043922 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a7e2c50-13dd-4da6-a0d2-c71c16ce902e" containerName="revoke-keys" Jul 22 03:27:23.047076 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:27:23.047056 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7df930d3-8cslw" Jul 22 03:27:23.049533 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:27:23.049510 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cw8l4\"" Jul 22 03:27:23.049661 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:27:23.049507 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 03:27:23.055900 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:27:23.055875 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7df930d3-8cslw"] Jul 22 03:27:23.130096 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:27:23.130061 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b628d45-3976-48e0-9954-60218dd5a170-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-7df930d3-8cslw\" (UID: \"8b628d45-3976-48e0-9954-60218dd5a170\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7df930d3-8cslw" Jul 22 03:27:23.230664 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:27:23.230628 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b628d45-3976-48e0-9954-60218dd5a170-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-7df930d3-8cslw\" (UID: \"8b628d45-3976-48e0-9954-60218dd5a170\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7df930d3-8cslw" Jul 22 03:27:23.231335 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:27:23.231312 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b628d45-3976-48e0-9954-60218dd5a170-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-7df930d3-8cslw\" (UID: \"8b628d45-3976-48e0-9954-60218dd5a170\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7df930d3-8cslw" Jul 22 03:27:23.359091 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:27:23.358994 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7df930d3-8cslw" Jul 22 03:27:23.509852 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:27:23.509696 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7df930d3-8cslw"] Jul 22 03:27:23.512226 ip-10-0-131-67 kubenswrapper[2582]: W0722 03:27:23.512193 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b628d45_3976_48e0_9954_60218dd5a170.slice/crio-4b038a1f727e19341a0441deaf2b1e59980edb302ee7ce561b1a1a8f78173296 WatchSource:0}: Error finding container 4b038a1f727e19341a0441deaf2b1e59980edb302ee7ce561b1a1a8f78173296: Status 404 returned error can't find the container with id 4b038a1f727e19341a0441deaf2b1e59980edb302ee7ce561b1a1a8f78173296 Jul 22 03:27:23.789729 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:27:23.789688 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7df930d3-8cslw" event={"ID":"8b628d45-3976-48e0-9954-60218dd5a170","Type":"ContainerStarted","Data":"4b038a1f727e19341a0441deaf2b1e59980edb302ee7ce561b1a1a8f78173296"} Jul 22 03:27:24.797686 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:27:24.797645 2582 generic.go:356] "Generic (PLEG): container finished" podID="8b628d45-3976-48e0-9954-60218dd5a170" containerID="d5d7289ed227e9eb5ea989b1082b65434a6ba013976eaeaa38aee7055eba78da" exitCode=0 Jul 22 03:27:24.798162 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:27:24.797731 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7df930d3-8cslw" event={"ID":"8b628d45-3976-48e0-9954-60218dd5a170","Type":"ContainerDied","Data":"d5d7289ed227e9eb5ea989b1082b65434a6ba013976eaeaa38aee7055eba78da"} Jul 22 03:27:25.957215 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:27:25.957190 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7df930d3-8cslw" Jul 22 03:27:26.059550 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:27:26.059459 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b628d45-3976-48e0-9954-60218dd5a170-maas-api-service-ca\") pod \"8b628d45-3976-48e0-9954-60218dd5a170\" (UID: \"8b628d45-3976-48e0-9954-60218dd5a170\") " Jul 22 03:27:26.059814 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:27:26.059791 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b628d45-3976-48e0-9954-60218dd5a170-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8b628d45-3976-48e0-9954-60218dd5a170" (UID: "8b628d45-3976-48e0-9954-60218dd5a170"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 03:27:26.160866 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:27:26.160832 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b628d45-3976-48e0-9954-60218dd5a170-maas-api-service-ca\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 03:27:26.812171 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:27:26.812141 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7df930d3-8cslw" Jul 22 03:27:26.812171 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:27:26.812152 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7df930d3-8cslw" event={"ID":"8b628d45-3976-48e0-9954-60218dd5a170","Type":"ContainerDied","Data":"4b038a1f727e19341a0441deaf2b1e59980edb302ee7ce561b1a1a8f78173296"} Jul 22 03:27:26.812412 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:27:26.812195 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b038a1f727e19341a0441deaf2b1e59980edb302ee7ce561b1a1a8f78173296" Jul 22 03:27:33.816812 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:27:33.816776 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:27:36.587319 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:27:36.587280 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:27:38.190408 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:27:38.190368 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:27:38.206102 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:27:38.206071 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7df930d3-8cslw"] Jul 22 03:27:38.216781 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:27:38.216754 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7df930d3-8cslw"] Jul 22 03:27:38.430194 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:27:38.430162 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:27:38.676537 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:27:38.676500 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:27:39.193816 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:27:39.193788 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:27:39.292814 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:27:39.292777 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:27:39.398796 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:27:39.398762 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:27:39.903195 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:27:39.903160 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b628d45-3976-48e0-9954-60218dd5a170" path="/var/lib/kubelet/pods/8b628d45-3976-48e0-9954-60218dd5a170/volumes" Jul 22 03:27:41.049886 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:27:41.049854 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:27:50.628640 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:27:50.628581 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:27:50.755659 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:27:50.755623 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:27:50.863192 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:27:50.863155 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:27:50.973076 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:27:50.972986 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:27:51.482825 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:27:51.482796 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:28:17.536357 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:28:17.536320 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:28:24.664681 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:28:24.664647 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:28:24.768211 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:28:24.768179 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:28:24.867901 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:28:24.867865 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:28:24.966595 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:28:24.966502 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:28:31.378630 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:28:31.378591 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:28:36.265632 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:28:36.265596 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:28:40.878201 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:28:40.878162 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:28:47.596051 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:28:47.595997 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:28:47.667686 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:28:47.667652 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:28:47.759005 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:28:47.758963 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:28:47.855959 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:28:47.855873 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:28:58.398801 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:28:58.398762 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:28:59.236440 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:28:59.236407 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:29:06.766962 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:29:06.766925 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:29:06.878073 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:29:06.878034 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:29:06.982609 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:29:06.982580 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:29:07.187508 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:29:07.187475 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:29:07.886102 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:29:07.886066 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:29:07.997320 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:29:07.997285 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:29:08.115287 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:29:08.115254 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:29:09.859339 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:29:09.859304 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:29:09.992846 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:29:09.992812 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:29:15.987977 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:29:15.987942 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:29:16.087567 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:29:16.087538 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:29:16.679978 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:29:16.679943 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:29:22.578379 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:29:22.578338 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:29:23.048733 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:29:23.048699 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:29:23.149356 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:29:23.149327 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:29:23.365721 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:29:23.365623 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:29:23.737104 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:29:23.737068 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:29:24.035313 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:29:24.035229 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:29:26.549860 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:29:26.549825 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:29:26.657930 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:29:26.657891 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:29:26.746782 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:29:26.746750 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:29:26.994429 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:29:26.994397 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:29:27.334083 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:29:27.333973 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:29:27.434621 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:29:27.434566 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:29:31.096331 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:29:31.096290 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:29:31.790266 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:29:31.790235 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:29:31.889558 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:29:31.889521 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:29:32.268436 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:29:32.268407 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:29:34.995559 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:29:34.995524 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:29:38.597886 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:29:38.597850 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:29:38.674350 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:29:38.674320 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:29:39.015870 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:29:39.015837 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fe15ed44-48sw8"] Jul 22 03:29:39.016293 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:29:39.016279 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b628d45-3976-48e0-9954-60218dd5a170" containerName="revoke-keys" Jul 22 03:29:39.016342 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:29:39.016295 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b628d45-3976-48e0-9954-60218dd5a170" containerName="revoke-keys" Jul 22 03:29:39.016383 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:29:39.016370 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b628d45-3976-48e0-9954-60218dd5a170" containerName="revoke-keys" Jul 22 03:29:39.019545 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:29:39.019530 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fe15ed44-48sw8" Jul 22 03:29:39.021963 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:29:39.021937 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 03:29:39.022125 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:29:39.021937 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cw8l4\"" Jul 22 03:29:39.029547 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:29:39.029466 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fe15ed44-48sw8"] Jul 22 03:29:39.060898 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:29:39.060862 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b28a5b7a-4119-49e3-91bc-21c6d057e5c5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-fe15ed44-48sw8\" (UID: \"b28a5b7a-4119-49e3-91bc-21c6d057e5c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fe15ed44-48sw8" Jul 22 03:29:39.161849 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:29:39.161817 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b28a5b7a-4119-49e3-91bc-21c6d057e5c5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-fe15ed44-48sw8\" (UID: \"b28a5b7a-4119-49e3-91bc-21c6d057e5c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fe15ed44-48sw8" Jul 22 03:29:39.162554 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:29:39.162535 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b28a5b7a-4119-49e3-91bc-21c6d057e5c5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-fe15ed44-48sw8\" (UID: \"b28a5b7a-4119-49e3-91bc-21c6d057e5c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fe15ed44-48sw8" Jul 22 03:29:39.332275 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:29:39.332181 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fe15ed44-48sw8" Jul 22 03:29:39.477924 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:29:39.477899 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fe15ed44-48sw8"] Jul 22 03:29:39.480161 ip-10-0-131-67 kubenswrapper[2582]: W0722 03:29:39.480132 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb28a5b7a_4119_49e3_91bc_21c6d057e5c5.slice/crio-47d18519402bc50eee649fad079a8314cf4b65d86822ba570621190dea117bd2 WatchSource:0}: Error finding container 47d18519402bc50eee649fad079a8314cf4b65d86822ba570621190dea117bd2: Status 404 returned error can't find the container with id 47d18519402bc50eee649fad079a8314cf4b65d86822ba570621190dea117bd2 Jul 22 03:29:39.596820 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:29:39.596736 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fe15ed44-48sw8" event={"ID":"b28a5b7a-4119-49e3-91bc-21c6d057e5c5","Type":"ContainerStarted","Data":"47d18519402bc50eee649fad079a8314cf4b65d86822ba570621190dea117bd2"} Jul 22 03:29:40.603864 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:29:40.603762 2582 generic.go:356] "Generic (PLEG): container finished" podID="b28a5b7a-4119-49e3-91bc-21c6d057e5c5" containerID="9d1347677bd4b600f0912df95fd06c814b95efaab46cbcbaba3e7371a6225c45" exitCode=0 Jul 22 03:29:40.604306 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:29:40.603861 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fe15ed44-48sw8" event={"ID":"b28a5b7a-4119-49e3-91bc-21c6d057e5c5","Type":"ContainerDied","Data":"9d1347677bd4b600f0912df95fd06c814b95efaab46cbcbaba3e7371a6225c45"} Jul 22 03:29:41.743664 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:29:41.743634 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fe15ed44-48sw8" Jul 22 03:29:41.885696 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:29:41.885667 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b28a5b7a-4119-49e3-91bc-21c6d057e5c5-maas-api-service-ca\") pod \"b28a5b7a-4119-49e3-91bc-21c6d057e5c5\" (UID: \"b28a5b7a-4119-49e3-91bc-21c6d057e5c5\") " Jul 22 03:29:41.886075 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:29:41.886052 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b28a5b7a-4119-49e3-91bc-21c6d057e5c5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b28a5b7a-4119-49e3-91bc-21c6d057e5c5" (UID: "b28a5b7a-4119-49e3-91bc-21c6d057e5c5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 03:29:41.987429 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:29:41.986848 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b28a5b7a-4119-49e3-91bc-21c6d057e5c5-maas-api-service-ca\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 03:29:42.617049 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:29:42.617007 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fe15ed44-48sw8" Jul 22 03:29:42.617231 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:29:42.617049 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fe15ed44-48sw8" event={"ID":"b28a5b7a-4119-49e3-91bc-21c6d057e5c5","Type":"ContainerDied","Data":"47d18519402bc50eee649fad079a8314cf4b65d86822ba570621190dea117bd2"} Jul 22 03:29:42.617231 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:29:42.617096 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47d18519402bc50eee649fad079a8314cf4b65d86822ba570621190dea117bd2" Jul 22 03:29:49.514336 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:29:49.514301 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:29:52.082176 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:29:52.082145 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:29:54.097970 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:29:54.097937 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:29:54.098729 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:29:54.098702 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fe15ed44-48sw8"] Jul 22 03:29:54.101316 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:29:54.101291 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fe15ed44-48sw8"] Jul 22 03:29:54.509851 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:29:54.509815 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:29:55.902110 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:29:55.902076 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b28a5b7a-4119-49e3-91bc-21c6d057e5c5" path="/var/lib/kubelet/pods/b28a5b7a-4119-49e3-91bc-21c6d057e5c5/volumes" Jul 22 03:30:00.144411 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:30:00.144372 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29744850-vp62n"] Jul 22 03:30:00.144917 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:30:00.144788 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b28a5b7a-4119-49e3-91bc-21c6d057e5c5" containerName="revoke-keys" Jul 22 03:30:00.144917 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:30:00.144798 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b28a5b7a-4119-49e3-91bc-21c6d057e5c5" containerName="revoke-keys" Jul 22 03:30:00.144917 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:30:00.144882 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b28a5b7a-4119-49e3-91bc-21c6d057e5c5" containerName="revoke-keys" Jul 22 03:30:00.148240 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:30:00.148224 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744850-vp62n" Jul 22 03:30:00.150571 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:30:00.150549 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cw8l4\"" Jul 22 03:30:00.156781 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:30:00.156755 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29744850-vp62n"] Jul 22 03:30:00.159551 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:30:00.159529 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744850-vp62n" Jul 22 03:30:00.300717 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:30:00.300689 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29744850-vp62n"] Jul 22 03:30:00.302976 ip-10-0-131-67 kubenswrapper[2582]: W0722 03:30:00.302944 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33141429_e38e_4dff_89d1_29ec69356888.slice/crio-b09e53d96a7c38c6b24b6cfb842c157ff7a3b90788254b6bb52c13884da64e3a WatchSource:0}: Error finding container b09e53d96a7c38c6b24b6cfb842c157ff7a3b90788254b6bb52c13884da64e3a: Status 404 returned error can't find the container with id b09e53d96a7c38c6b24b6cfb842c157ff7a3b90788254b6bb52c13884da64e3a Jul 22 03:30:00.633603 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:30:00.633567 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:30:00.724658 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:30:00.724622 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744850-vp62n" event={"ID":"33141429-e38e-4dff-89d1-29ec69356888","Type":"ContainerStarted","Data":"0b99b3e0a20c6f871ac3e0150bf4cbffce4a0f8b18ddb7cb21cf3aa4d9d27bc3"} Jul 22 03:30:00.724822 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:30:00.724683 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744850-vp62n" event={"ID":"33141429-e38e-4dff-89d1-29ec69356888","Type":"ContainerStarted","Data":"b09e53d96a7c38c6b24b6cfb842c157ff7a3b90788254b6bb52c13884da64e3a"} Jul 22 03:30:00.732221 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:30:00.732186 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:30:00.740561 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:30:00.740505 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744850-vp62n" podStartSLOduration=0.740484496 podStartE2EDuration="740.484496ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:30:00.740195849 +0000 UTC m=+1911.471816715" watchObservedRunningTime="2026-07-22 03:30:00.740484496 +0000 UTC m=+1911.472105344" Jul 22 03:30:00.921069 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:30:00.921039 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:30:01.168650 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:30:01.168610 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:30:01.309107 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:30:01.309054 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:30:01.411818 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:30:01.411788 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:30:01.559091 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:30:01.558985 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:30:01.732706 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:30:01.732673 2582 generic.go:356] "Generic (PLEG): container finished" podID="33141429-e38e-4dff-89d1-29ec69356888" containerID="0b99b3e0a20c6f871ac3e0150bf4cbffce4a0f8b18ddb7cb21cf3aa4d9d27bc3" exitCode=0 Jul 22 03:30:01.732885 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:30:01.732758 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744850-vp62n" event={"ID":"33141429-e38e-4dff-89d1-29ec69356888","Type":"ContainerDied","Data":"0b99b3e0a20c6f871ac3e0150bf4cbffce4a0f8b18ddb7cb21cf3aa4d9d27bc3"} Jul 22 03:30:02.879961 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:30:02.879936 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744850-vp62n" Jul 22 03:30:03.745890 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:30:03.745857 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744850-vp62n" Jul 22 03:30:03.746125 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:30:03.745901 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744850-vp62n" event={"ID":"33141429-e38e-4dff-89d1-29ec69356888","Type":"ContainerDied","Data":"b09e53d96a7c38c6b24b6cfb842c157ff7a3b90788254b6bb52c13884da64e3a"} Jul 22 03:30:03.746125 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:30:03.745991 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b09e53d96a7c38c6b24b6cfb842c157ff7a3b90788254b6bb52c13884da64e3a" Jul 22 03:30:07.796469 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:30:07.796439 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:30:07.861658 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:30:07.861625 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:30:07.954515 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:30:07.954482 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:30:08.055514 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:30:08.055412 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:30:08.184498 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:30:08.184462 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:30:14.337557 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:30:14.337520 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:30:14.436161 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:30:14.436130 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:30:21.335203 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:30:21.335159 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:30:21.663715 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:30:21.663683 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:30:21.780985 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:30:21.780940 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:30:24.556816 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:30:24.556771 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:30:25.817227 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:30:25.817194 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:30:29.081685 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:30:29.081652 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:30:30.775006 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:30:30.774968 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:30:36.200310 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:30:36.200279 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:30:38.877380 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:30:38.877344 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:31:01.368543 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:31:01.368459 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:31:07.811679 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:31:07.811636 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-017b0f74-pjc2c"] Jul 22 03:31:07.812194 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:31:07.812164 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33141429-e38e-4dff-89d1-29ec69356888" containerName="cleanup" Jul 22 03:31:07.812194 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:31:07.812178 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="33141429-e38e-4dff-89d1-29ec69356888" containerName="cleanup" Jul 22 03:31:07.812275 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:31:07.812265 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="33141429-e38e-4dff-89d1-29ec69356888" containerName="cleanup" Jul 22 03:31:07.815475 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:31:07.815454 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-017b0f74-pjc2c" Jul 22 03:31:07.817910 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:31:07.817888 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 03:31:07.818004 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:31:07.817947 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cw8l4\"" Jul 22 03:31:07.825137 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:31:07.825105 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-017b0f74-pjc2c"] Jul 22 03:31:07.871810 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:31:07.871771 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/330e8fe4-a37c-4798-a3d3-c279a1b23840-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-017b0f74-pjc2c\" (UID: \"330e8fe4-a37c-4798-a3d3-c279a1b23840\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-017b0f74-pjc2c" Jul 22 03:31:07.972790 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:31:07.972754 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/330e8fe4-a37c-4798-a3d3-c279a1b23840-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-017b0f74-pjc2c\" (UID: \"330e8fe4-a37c-4798-a3d3-c279a1b23840\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-017b0f74-pjc2c" Jul 22 03:31:07.973414 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:31:07.973394 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/330e8fe4-a37c-4798-a3d3-c279a1b23840-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-017b0f74-pjc2c\" (UID: \"330e8fe4-a37c-4798-a3d3-c279a1b23840\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-017b0f74-pjc2c" Jul 22 03:31:08.131799 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:31:08.131761 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-017b0f74-pjc2c" Jul 22 03:31:08.278528 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:31:08.278494 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-017b0f74-pjc2c"] Jul 22 03:31:08.280485 ip-10-0-131-67 kubenswrapper[2582]: W0722 03:31:08.280458 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod330e8fe4_a37c_4798_a3d3_c279a1b23840.slice/crio-a45515f400e246d44fb9bb75dce8ae71c14b1d653c9f718c32e3dcd53eceb3d1 WatchSource:0}: Error finding container a45515f400e246d44fb9bb75dce8ae71c14b1d653c9f718c32e3dcd53eceb3d1: Status 404 returned error can't find the container with id a45515f400e246d44fb9bb75dce8ae71c14b1d653c9f718c32e3dcd53eceb3d1 Jul 22 03:31:09.141905 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:31:09.141865 2582 generic.go:356] "Generic (PLEG): container finished" podID="330e8fe4-a37c-4798-a3d3-c279a1b23840" containerID="a54f21eced5f9ea43edeb3ce0d899562146aed20cd97093d471dbd7606a5dc8a" exitCode=0 Jul 22 03:31:09.142321 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:31:09.141948 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-017b0f74-pjc2c" event={"ID":"330e8fe4-a37c-4798-a3d3-c279a1b23840","Type":"ContainerDied","Data":"a54f21eced5f9ea43edeb3ce0d899562146aed20cd97093d471dbd7606a5dc8a"} Jul 22 03:31:09.142321 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:31:09.142003 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-017b0f74-pjc2c" event={"ID":"330e8fe4-a37c-4798-a3d3-c279a1b23840","Type":"ContainerStarted","Data":"a45515f400e246d44fb9bb75dce8ae71c14b1d653c9f718c32e3dcd53eceb3d1"} Jul 22 03:31:10.300992 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:31:10.300968 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-017b0f74-pjc2c" Jul 22 03:31:10.393458 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:31:10.393429 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/330e8fe4-a37c-4798-a3d3-c279a1b23840-maas-api-service-ca\") pod \"330e8fe4-a37c-4798-a3d3-c279a1b23840\" (UID: \"330e8fe4-a37c-4798-a3d3-c279a1b23840\") " Jul 22 03:31:10.393829 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:31:10.393801 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/330e8fe4-a37c-4798-a3d3-c279a1b23840-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "330e8fe4-a37c-4798-a3d3-c279a1b23840" (UID: "330e8fe4-a37c-4798-a3d3-c279a1b23840"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 03:31:10.494806 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:31:10.494711 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/330e8fe4-a37c-4798-a3d3-c279a1b23840-maas-api-service-ca\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 03:31:11.155714 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:31:11.155686 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-017b0f74-pjc2c" Jul 22 03:31:11.155714 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:31:11.155703 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-017b0f74-pjc2c" event={"ID":"330e8fe4-a37c-4798-a3d3-c279a1b23840","Type":"ContainerDied","Data":"a45515f400e246d44fb9bb75dce8ae71c14b1d653c9f718c32e3dcd53eceb3d1"} Jul 22 03:31:11.155931 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:31:11.155746 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a45515f400e246d44fb9bb75dce8ae71c14b1d653c9f718c32e3dcd53eceb3d1" Jul 22 03:31:21.675105 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:31:21.675072 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:31:23.314412 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:31:23.314375 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:31:27.890666 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:31:27.890626 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-017b0f74-pjc2c"] Jul 22 03:31:27.892941 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:31:27.892919 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-017b0f74-pjc2c"] Jul 22 03:31:27.901437 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:31:27.901406 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="330e8fe4-a37c-4798-a3d3-c279a1b23840" path="/var/lib/kubelet/pods/330e8fe4-a37c-4798-a3d3-c279a1b23840/volumes" Jul 22 03:31:27.947877 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:31:27.947843 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:31:30.246746 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:31:30.246711 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:31:31.970729 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:31:31.970693 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:31:35.279668 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:31:35.279631 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:32:02.717805 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:32:02.717772 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:32:02.821252 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:32:02.821212 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:32:08.409658 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:32:08.409622 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:32:08.511223 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:32:08.511192 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:32:11.711161 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:32:11.711127 2582 scope.go:117] "RemoveContainer" containerID="779a59cc3ffef710d38626f880ddbc3215a59cdad90238514d12bbcff580636c" Jul 22 03:32:11.720346 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:32:11.720323 2582 scope.go:117] "RemoveContainer" containerID="3ae0c38f4ca92b6030cc9d34ec585a11605c7ec2ac2ee0aa2b9a0f09ff656ef9" Jul 22 03:32:11.963580 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:32:11.963493 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:32:12.578981 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:32:12.578952 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:32:18.257441 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:32:18.257400 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:32:18.361273 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:32:18.361236 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:32:18.580179 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:32:18.580084 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:32:18.844326 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:32:18.844235 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:32:19.963965 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:32:19.963935 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:32:20.059732 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:32:20.059700 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:32:20.160471 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:32:20.160433 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:32:20.985729 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:32:20.985691 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:32:21.083139 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:32:21.083104 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:32:21.180241 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:32:21.180205 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:32:22.645598 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:32:22.645568 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:32:28.786474 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:32:28.786435 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:32:28.887174 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:32:28.887143 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:32:29.081486 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:32:29.081403 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:32:29.493407 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:32:29.493373 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:32:35.588288 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:32:35.588252 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:32:35.694764 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:32:35.694729 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:32:35.794187 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:32:35.794135 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:32:37.686673 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:32:37.686636 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:32:46.397219 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:32:46.397181 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:32:46.788918 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:32:46.788823 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:33:07.803189 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:33:07.803152 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:33:11.751277 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:33:11.751247 2582 scope.go:117] "RemoveContainer" containerID="3d72d4547f86b8f98059f3763c79c2aba0ee6c2fa0ceca0212eed6c076b5e07a" Jul 22 03:33:14.349341 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:33:14.349307 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:33:14.450173 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:33:14.450143 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:33:14.731876 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:33:14.731844 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:33:15.065694 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:33:15.065618 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:33:15.406365 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:33:15.406332 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:33:18.358425 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:33:18.358391 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:33:18.458261 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:33:18.458229 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:33:18.558003 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:33:18.557967 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:33:18.744823 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:33:18.744793 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:33:19.115315 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:33:19.115224 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:33:19.214784 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:33:19.214748 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:33:22.922444 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:33:22.922413 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:33:23.175329 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:33:23.175249 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:33:23.292210 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:33:23.292174 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:33:23.648719 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:33:23.648679 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:33:23.830714 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:33:23.830682 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:33:26.450142 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:33:26.450104 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:33:26.550710 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:33:26.550659 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:33:26.718249 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:33:26.718161 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:33:27.269804 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:33:27.269768 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:33:27.449099 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:33:27.449053 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:33:30.894752 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:33:30.894718 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:33:31.575118 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:33:31.575087 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:33:37.485470 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:33:37.485433 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:33:40.298392 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:33:40.298354 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:33:40.410637 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:33:40.410603 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:33:40.411802 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:33:40.411783 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:33:48.606562 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:33:48.606530 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:33:48.701885 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:33:48.701850 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:33:49.029607 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:33:49.029572 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:33:53.411857 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:33:53.411821 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:34:08.348172 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:34:08.348088 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:34:11.777694 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:34:11.777661 2582 scope.go:117] "RemoveContainer" containerID="d5d7289ed227e9eb5ea989b1082b65434a6ba013976eaeaa38aee7055eba78da" Jul 22 03:34:12.350924 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:34:12.350893 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:34:26.294360 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:34:26.294325 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:34:50.979492 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:34:50.979453 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:34:51.910921 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:34:51.910889 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:34:52.037060 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:34:52.037003 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd4878-d2gp9"] Jul 22 03:34:52.037461 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:34:52.037453 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="330e8fe4-a37c-4798-a3d3-c279a1b23840" containerName="revoke-keys" Jul 22 03:34:52.037511 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:34:52.037465 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="330e8fe4-a37c-4798-a3d3-c279a1b23840" containerName="revoke-keys" Jul 22 03:34:52.037547 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:34:52.037537 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="330e8fe4-a37c-4798-a3d3-c279a1b23840" containerName="revoke-keys" Jul 22 03:34:52.040633 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:34:52.040613 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd4878-d2gp9" Jul 22 03:34:52.043250 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:34:52.043228 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cw8l4\"" Jul 22 03:34:52.043250 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:34:52.043246 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 03:34:52.050888 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:34:52.050863 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd4878-d2gp9"] Jul 22 03:34:52.192271 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:34:52.192166 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30732ab5-a75f-4cd6-bf13-530d377d4e86-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fd4878-d2gp9\" (UID: \"30732ab5-a75f-4cd6-bf13-530d377d4e86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd4878-d2gp9" Jul 22 03:34:52.293575 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:34:52.293540 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30732ab5-a75f-4cd6-bf13-530d377d4e86-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fd4878-d2gp9\" (UID: \"30732ab5-a75f-4cd6-bf13-530d377d4e86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd4878-d2gp9" Jul 22 03:34:52.294184 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:34:52.294164 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30732ab5-a75f-4cd6-bf13-530d377d4e86-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fd4878-d2gp9\" (UID: \"30732ab5-a75f-4cd6-bf13-530d377d4e86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd4878-d2gp9" Jul 22 03:34:52.353362 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:34:52.353331 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd4878-d2gp9" Jul 22 03:34:52.600222 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:34:52.600134 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:34:52.699949 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:34:52.699919 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd4878-d2gp9"] Jul 22 03:34:52.700108 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:34:52.699927 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:34:52.701682 ip-10-0-131-67 kubenswrapper[2582]: W0722 03:34:52.701658 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30732ab5_a75f_4cd6_bf13_530d377d4e86.slice/crio-66e595404bf32d09510e67280a34d9111d1cc32ce28ddb1b8dd1f359c9f79fea WatchSource:0}: Error finding container 66e595404bf32d09510e67280a34d9111d1cc32ce28ddb1b8dd1f359c9f79fea: Status 404 returned error can't find the container with id 66e595404bf32d09510e67280a34d9111d1cc32ce28ddb1b8dd1f359c9f79fea Jul 22 03:34:53.509173 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:34:53.509142 2582 generic.go:356] "Generic (PLEG): container finished" podID="30732ab5-a75f-4cd6-bf13-530d377d4e86" containerID="7b79dca1650811ea711fa9b4de3abcfb15bdabf043897cfb8ca225f48835d28d" exitCode=0 Jul 22 03:34:53.509560 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:34:53.509178 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd4878-d2gp9" event={"ID":"30732ab5-a75f-4cd6-bf13-530d377d4e86","Type":"ContainerDied","Data":"7b79dca1650811ea711fa9b4de3abcfb15bdabf043897cfb8ca225f48835d28d"} Jul 22 03:34:53.509560 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:34:53.509215 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd4878-d2gp9" event={"ID":"30732ab5-a75f-4cd6-bf13-530d377d4e86","Type":"ContainerStarted","Data":"66e595404bf32d09510e67280a34d9111d1cc32ce28ddb1b8dd1f359c9f79fea"} Jul 22 03:34:54.655413 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:34:54.655382 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd4878-d2gp9" Jul 22 03:34:54.816332 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:34:54.816233 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30732ab5-a75f-4cd6-bf13-530d377d4e86-maas-api-service-ca\") pod \"30732ab5-a75f-4cd6-bf13-530d377d4e86\" (UID: \"30732ab5-a75f-4cd6-bf13-530d377d4e86\") " Jul 22 03:34:54.816591 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:34:54.816568 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30732ab5-a75f-4cd6-bf13-530d377d4e86-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "30732ab5-a75f-4cd6-bf13-530d377d4e86" (UID: "30732ab5-a75f-4cd6-bf13-530d377d4e86"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 03:34:54.917366 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:34:54.917319 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30732ab5-a75f-4cd6-bf13-530d377d4e86-maas-api-service-ca\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 03:34:55.381775 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:34:55.381743 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:34:55.521821 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:34:55.521790 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd4878-d2gp9" Jul 22 03:34:55.521821 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:34:55.521819 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd4878-d2gp9" event={"ID":"30732ab5-a75f-4cd6-bf13-530d377d4e86","Type":"ContainerDied","Data":"66e595404bf32d09510e67280a34d9111d1cc32ce28ddb1b8dd1f359c9f79fea"} Jul 22 03:34:55.522052 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:34:55.521866 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66e595404bf32d09510e67280a34d9111d1cc32ce28ddb1b8dd1f359c9f79fea" Jul 22 03:35:02.532163 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:35:02.532127 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:35:07.119281 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:35:07.119245 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:35:07.131027 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:35:07.130980 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd4878-d2gp9"] Jul 22 03:35:07.134049 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:35:07.134023 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd4878-d2gp9"] Jul 22 03:35:07.546601 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:35:07.546566 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:35:07.901708 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:35:07.901678 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30732ab5-a75f-4cd6-bf13-530d377d4e86" path="/var/lib/kubelet/pods/30732ab5-a75f-4cd6-bf13-530d377d4e86/volumes" Jul 22 03:35:11.076188 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:35:11.076146 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:35:13.757820 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:35:13.757782 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:35:13.822570 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:35:13.822535 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:35:14.100902 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:35:14.100810 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:35:14.299577 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:35:14.299544 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:35:14.426261 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:35:14.426230 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-30704e-84c2v"] Jul 22 03:35:14.426688 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:35:14.426675 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30732ab5-a75f-4cd6-bf13-530d377d4e86" containerName="revoke-keys" Jul 22 03:35:14.426733 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:35:14.426689 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="30732ab5-a75f-4cd6-bf13-530d377d4e86" containerName="revoke-keys" Jul 22 03:35:14.426775 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:35:14.426765 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="30732ab5-a75f-4cd6-bf13-530d377d4e86" containerName="revoke-keys" Jul 22 03:35:14.429716 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:35:14.429697 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-30704e-84c2v" Jul 22 03:35:14.432366 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:35:14.432346 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cw8l4\"" Jul 22 03:35:14.432470 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:35:14.432346 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 03:35:14.439329 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:35:14.439308 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-30704e-84c2v"] Jul 22 03:35:14.492178 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:35:14.492147 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70df354d-73fa-4105-b425-c6c91ba41392-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-30704e-84c2v\" (UID: \"70df354d-73fa-4105-b425-c6c91ba41392\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-30704e-84c2v" Jul 22 03:35:14.593193 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:35:14.593160 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70df354d-73fa-4105-b425-c6c91ba41392-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-30704e-84c2v\" (UID: \"70df354d-73fa-4105-b425-c6c91ba41392\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-30704e-84c2v" Jul 22 03:35:14.593765 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:35:14.593747 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70df354d-73fa-4105-b425-c6c91ba41392-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-30704e-84c2v\" (UID: \"70df354d-73fa-4105-b425-c6c91ba41392\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-30704e-84c2v" Jul 22 03:35:14.740850 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:35:14.740763 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-30704e-84c2v" Jul 22 03:35:14.882287 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:35:14.882257 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-30704e-84c2v"] Jul 22 03:35:14.882833 ip-10-0-131-67 kubenswrapper[2582]: W0722 03:35:14.882806 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70df354d_73fa_4105_b425_c6c91ba41392.slice/crio-e7eb3e9d0fa5783ac3c7a2a39705a0ab55e9e3b541a62968e692629ece9c107d WatchSource:0}: Error finding container e7eb3e9d0fa5783ac3c7a2a39705a0ab55e9e3b541a62968e692629ece9c107d: Status 404 returned error can't find the container with id e7eb3e9d0fa5783ac3c7a2a39705a0ab55e9e3b541a62968e692629ece9c107d Jul 22 03:35:15.194892 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:35:15.194857 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:35:15.294711 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:35:15.294669 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:35:15.645262 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:35:15.645231 2582 generic.go:356] "Generic (PLEG): container finished" podID="70df354d-73fa-4105-b425-c6c91ba41392" containerID="52d2da316d8e203bc5e486b911d697fdd234321f856c06f4135734efc843663a" exitCode=0 Jul 22 03:35:15.645436 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:35:15.645286 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-30704e-84c2v" event={"ID":"70df354d-73fa-4105-b425-c6c91ba41392","Type":"ContainerDied","Data":"52d2da316d8e203bc5e486b911d697fdd234321f856c06f4135734efc843663a"} Jul 22 03:35:15.645436 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:35:15.645353 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-30704e-84c2v" event={"ID":"70df354d-73fa-4105-b425-c6c91ba41392","Type":"ContainerStarted","Data":"e7eb3e9d0fa5783ac3c7a2a39705a0ab55e9e3b541a62968e692629ece9c107d"} Jul 22 03:35:16.785645 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:35:16.785620 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-30704e-84c2v" Jul 22 03:35:16.814816 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:35:16.814790 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70df354d-73fa-4105-b425-c6c91ba41392-maas-api-service-ca\") pod \"70df354d-73fa-4105-b425-c6c91ba41392\" (UID: \"70df354d-73fa-4105-b425-c6c91ba41392\") " Jul 22 03:35:16.815140 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:35:16.815118 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70df354d-73fa-4105-b425-c6c91ba41392-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "70df354d-73fa-4105-b425-c6c91ba41392" (UID: "70df354d-73fa-4105-b425-c6c91ba41392"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 03:35:16.915994 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:35:16.915888 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70df354d-73fa-4105-b425-c6c91ba41392-maas-api-service-ca\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 03:35:17.658802 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:35:17.658768 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-30704e-84c2v" event={"ID":"70df354d-73fa-4105-b425-c6c91ba41392","Type":"ContainerDied","Data":"e7eb3e9d0fa5783ac3c7a2a39705a0ab55e9e3b541a62968e692629ece9c107d"} Jul 22 03:35:17.658802 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:35:17.658801 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-30704e-84c2v" Jul 22 03:35:17.659028 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:35:17.658828 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7eb3e9d0fa5783ac3c7a2a39705a0ab55e9e3b541a62968e692629ece9c107d" Jul 22 03:35:24.901617 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:35:24.901583 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:35:27.670435 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:35:27.670391 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:35:29.515389 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:35:29.515357 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-30704e-84c2v"] Jul 22 03:35:29.520194 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:35:29.520165 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:35:29.520839 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:35:29.520819 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-30704e-84c2v"] Jul 22 03:35:29.903475 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:35:29.903444 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70df354d-73fa-4105-b425-c6c91ba41392" path="/var/lib/kubelet/pods/70df354d-73fa-4105-b425-c6c91ba41392/volumes" Jul 22 03:35:30.716147 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:35:30.716117 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:35:35.944559 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:35:35.944526 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:35:36.044967 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:35:36.044935 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:35:36.251476 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:35:36.251392 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:35:36.492396 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:35:36.492361 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:35:36.851583 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:35:36.851546 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:35:37.199977 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:35:37.199938 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:35:37.785236 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:35:37.785201 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:35:40.021586 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:35:40.021549 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:35:40.120426 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:35:40.120390 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:35:40.244833 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:35:40.244803 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:35:40.620113 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:35:40.620080 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:35:40.786932 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:35:40.786894 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:35:40.884358 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:35:40.884326 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:35:43.181203 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:35:43.181167 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:35:44.543514 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:35:44.543482 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:35:45.055935 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:35:45.055903 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:35:45.384067 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:35:45.383996 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:35:45.687999 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:35:45.687918 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:35:48.013731 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:35:48.013693 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:35:48.113393 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:35:48.113363 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:35:48.278998 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:35:48.278913 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:35:48.533609 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:35:48.533529 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:35:48.967666 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:35:48.967633 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:35:49.063700 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:35:49.063663 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:35:52.597942 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:35:52.597903 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:35:55.177357 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:35:55.177321 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:36:00.859171 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:00.859131 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-19cc62-qbn7k"] Jul 22 03:36:00.859701 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:00.859682 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70df354d-73fa-4105-b425-c6c91ba41392" containerName="revoke-keys" Jul 22 03:36:00.859701 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:00.859702 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="70df354d-73fa-4105-b425-c6c91ba41392" containerName="revoke-keys" Jul 22 03:36:00.859805 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:00.859794 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="70df354d-73fa-4105-b425-c6c91ba41392" containerName="revoke-keys" Jul 22 03:36:00.863255 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:00.863237 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-19cc62-qbn7k" Jul 22 03:36:00.865961 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:00.865936 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 03:36:00.866124 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:00.866105 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cw8l4\"" Jul 22 03:36:00.872370 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:00.872346 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-19cc62-qbn7k"] Jul 22 03:36:00.924166 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:00.924132 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbd280c3-cca6-4f1d-a42a-d7c277b3a187-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-19cc62-qbn7k\" (UID: \"bbd280c3-cca6-4f1d-a42a-d7c277b3a187\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-19cc62-qbn7k" Jul 22 03:36:01.024732 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:01.024691 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbd280c3-cca6-4f1d-a42a-d7c277b3a187-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-19cc62-qbn7k\" (UID: \"bbd280c3-cca6-4f1d-a42a-d7c277b3a187\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-19cc62-qbn7k" Jul 22 03:36:01.025429 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:01.025403 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbd280c3-cca6-4f1d-a42a-d7c277b3a187-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-19cc62-qbn7k\" (UID: \"bbd280c3-cca6-4f1d-a42a-d7c277b3a187\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-19cc62-qbn7k" Jul 22 03:36:01.175404 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:01.175367 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-19cc62-qbn7k" Jul 22 03:36:01.315719 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:01.315694 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-19cc62-qbn7k"] Jul 22 03:36:01.318092 ip-10-0-131-67 kubenswrapper[2582]: W0722 03:36:01.318066 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbd280c3_cca6_4f1d_a42a_d7c277b3a187.slice/crio-9f0b0f0a703cd7ad3e77a359f837803b4981dd6dab5810688c0e493e64d59fbe WatchSource:0}: Error finding container 9f0b0f0a703cd7ad3e77a359f837803b4981dd6dab5810688c0e493e64d59fbe: Status 404 returned error can't find the container with id 9f0b0f0a703cd7ad3e77a359f837803b4981dd6dab5810688c0e493e64d59fbe Jul 22 03:36:01.464439 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:01.464314 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:36:01.563727 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:01.563691 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:36:01.925602 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:01.925567 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-19cc62-qbn7k" event={"ID":"bbd280c3-cca6-4f1d-a42a-d7c277b3a187","Type":"ContainerStarted","Data":"39112ebd161208e64c4e83a790f7522b30e1be057c007a201f1db67f5b29f7ba"} Jul 22 03:36:01.925996 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:01.925630 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-19cc62-qbn7k" event={"ID":"bbd280c3-cca6-4f1d-a42a-d7c277b3a187","Type":"ContainerStarted","Data":"9f0b0f0a703cd7ad3e77a359f837803b4981dd6dab5810688c0e493e64d59fbe"} Jul 22 03:36:01.943926 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:01.943878 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-19cc62-qbn7k" podStartSLOduration=1.9438610459999999 podStartE2EDuration="1.943861046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:36:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:36:01.942798567 +0000 UTC m=+2272.674419414" watchObservedRunningTime="2026-07-22 03:36:01.943861046 +0000 UTC m=+2272.675481893" Jul 22 03:36:02.933305 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:02.933271 2582 generic.go:356] "Generic (PLEG): container finished" podID="bbd280c3-cca6-4f1d-a42a-d7c277b3a187" containerID="39112ebd161208e64c4e83a790f7522b30e1be057c007a201f1db67f5b29f7ba" exitCode=0 Jul 22 03:36:02.933702 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:02.933333 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-19cc62-qbn7k" event={"ID":"bbd280c3-cca6-4f1d-a42a-d7c277b3a187","Type":"ContainerDied","Data":"39112ebd161208e64c4e83a790f7522b30e1be057c007a201f1db67f5b29f7ba"} Jul 22 03:36:04.079305 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:04.079280 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-19cc62-qbn7k" Jul 22 03:36:04.155244 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:04.155213 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbd280c3-cca6-4f1d-a42a-d7c277b3a187-maas-api-service-ca\") pod \"bbd280c3-cca6-4f1d-a42a-d7c277b3a187\" (UID: \"bbd280c3-cca6-4f1d-a42a-d7c277b3a187\") " Jul 22 03:36:04.155662 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:04.155633 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbd280c3-cca6-4f1d-a42a-d7c277b3a187-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bbd280c3-cca6-4f1d-a42a-d7c277b3a187" (UID: "bbd280c3-cca6-4f1d-a42a-d7c277b3a187"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 03:36:04.255974 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:04.255883 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbd280c3-cca6-4f1d-a42a-d7c277b3a187-maas-api-service-ca\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 03:36:04.945960 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:04.945930 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-19cc62-qbn7k" Jul 22 03:36:04.945960 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:04.945957 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-19cc62-qbn7k" event={"ID":"bbd280c3-cca6-4f1d-a42a-d7c277b3a187","Type":"ContainerDied","Data":"9f0b0f0a703cd7ad3e77a359f837803b4981dd6dab5810688c0e493e64d59fbe"} Jul 22 03:36:04.946189 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:04.946000 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f0b0f0a703cd7ad3e77a359f837803b4981dd6dab5810688c0e493e64d59fbe" Jul 22 03:36:09.952866 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:09.952830 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:36:10.052748 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:10.052713 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:36:11.530734 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:11.530699 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:36:11.630945 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:11.630918 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:36:11.809253 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:11.809171 2582 scope.go:117] "RemoveContainer" containerID="9d1347677bd4b600f0912df95fd06c814b95efaab46cbcbaba3e7371a6225c45" Jul 22 03:36:16.807556 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:16.807517 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:36:20.978759 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:20.978725 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:36:20.981108 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:20.981073 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-19cc62-qbn7k"] Jul 22 03:36:20.983639 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:20.983612 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-19cc62-qbn7k"] Jul 22 03:36:21.362654 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:21.362575 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:36:21.902575 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:21.902544 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbd280c3-cca6-4f1d-a42a-d7c277b3a187" path="/var/lib/kubelet/pods/bbd280c3-cca6-4f1d-a42a-d7c277b3a187/volumes" Jul 22 03:36:22.987379 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:22.987343 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:36:27.505193 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:27.505162 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:36:27.625139 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:27.625102 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:36:27.786616 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:27.786525 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:36:28.038625 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:28.038536 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:36:28.166641 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:28.166610 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ed08a5-b4qwv"] Jul 22 03:36:28.167085 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:28.167071 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbd280c3-cca6-4f1d-a42a-d7c277b3a187" containerName="revoke-keys" Jul 22 03:36:28.167136 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:28.167086 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbd280c3-cca6-4f1d-a42a-d7c277b3a187" containerName="revoke-keys" Jul 22 03:36:28.167187 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:28.167178 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbd280c3-cca6-4f1d-a42a-d7c277b3a187" containerName="revoke-keys" Jul 22 03:36:28.170601 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:28.170584 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ed08a5-b4qwv" Jul 22 03:36:28.172976 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:28.172945 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cw8l4\"" Jul 22 03:36:28.173720 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:28.173698 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 03:36:28.176709 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:28.176683 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ed08a5-b4qwv"] Jul 22 03:36:28.281809 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:28.281769 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7efb9739-c1e0-4214-8b5f-1495d0d39e65-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ed08a5-b4qwv\" (UID: \"7efb9739-c1e0-4214-8b5f-1495d0d39e65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ed08a5-b4qwv" Jul 22 03:36:28.382979 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:28.382943 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7efb9739-c1e0-4214-8b5f-1495d0d39e65-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ed08a5-b4qwv\" (UID: \"7efb9739-c1e0-4214-8b5f-1495d0d39e65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ed08a5-b4qwv" Jul 22 03:36:28.383605 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:28.383585 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7efb9739-c1e0-4214-8b5f-1495d0d39e65-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ed08a5-b4qwv\" (UID: \"7efb9739-c1e0-4214-8b5f-1495d0d39e65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ed08a5-b4qwv" Jul 22 03:36:28.483767 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:28.483731 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ed08a5-b4qwv" Jul 22 03:36:28.629252 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:28.629224 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ed08a5-b4qwv"] Jul 22 03:36:28.631340 ip-10-0-131-67 kubenswrapper[2582]: W0722 03:36:28.631314 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7efb9739_c1e0_4214_8b5f_1495d0d39e65.slice/crio-1e78b5145e2525f8111e9ac7abf41bf0d2c92278f049ff898082d078e47bf97b WatchSource:0}: Error finding container 1e78b5145e2525f8111e9ac7abf41bf0d2c92278f049ff898082d078e47bf97b: Status 404 returned error can't find the container with id 1e78b5145e2525f8111e9ac7abf41bf0d2c92278f049ff898082d078e47bf97b Jul 22 03:36:28.980624 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:28.980585 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:36:29.080572 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:29.080528 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:36:29.093459 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:29.093424 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ed08a5-b4qwv" event={"ID":"7efb9739-c1e0-4214-8b5f-1495d0d39e65","Type":"ContainerStarted","Data":"3b5f9dc5fbedde4c1df270772f67d6f2628098f2ae3223c3e4fe7823ff638f28"} Jul 22 03:36:29.093644 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:29.093487 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ed08a5-b4qwv" event={"ID":"7efb9739-c1e0-4214-8b5f-1495d0d39e65","Type":"ContainerStarted","Data":"1e78b5145e2525f8111e9ac7abf41bf0d2c92278f049ff898082d078e47bf97b"} Jul 22 03:36:29.110096 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:29.110048 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ed08a5-b4qwv" podStartSLOduration=1.110031727 podStartE2EDuration="1.110031727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:36:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:36:29.109197299 +0000 UTC m=+2299.840818146" watchObservedRunningTime="2026-07-22 03:36:29.110031727 +0000 UTC m=+2299.841652564" Jul 22 03:36:30.100981 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:30.100948 2582 generic.go:356] "Generic (PLEG): container finished" podID="7efb9739-c1e0-4214-8b5f-1495d0d39e65" containerID="3b5f9dc5fbedde4c1df270772f67d6f2628098f2ae3223c3e4fe7823ff638f28" exitCode=0 Jul 22 03:36:30.101449 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:30.101049 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ed08a5-b4qwv" event={"ID":"7efb9739-c1e0-4214-8b5f-1495d0d39e65","Type":"ContainerDied","Data":"3b5f9dc5fbedde4c1df270772f67d6f2628098f2ae3223c3e4fe7823ff638f28"} Jul 22 03:36:31.252041 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:31.252000 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ed08a5-b4qwv" Jul 22 03:36:31.408369 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:31.408333 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7efb9739-c1e0-4214-8b5f-1495d0d39e65-maas-api-service-ca\") pod \"7efb9739-c1e0-4214-8b5f-1495d0d39e65\" (UID: \"7efb9739-c1e0-4214-8b5f-1495d0d39e65\") " Jul 22 03:36:31.408698 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:31.408674 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7efb9739-c1e0-4214-8b5f-1495d0d39e65-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7efb9739-c1e0-4214-8b5f-1495d0d39e65" (UID: "7efb9739-c1e0-4214-8b5f-1495d0d39e65"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 03:36:31.509561 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:31.509513 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7efb9739-c1e0-4214-8b5f-1495d0d39e65-maas-api-service-ca\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 03:36:32.115456 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:32.115427 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ed08a5-b4qwv" Jul 22 03:36:32.115649 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:32.115423 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ed08a5-b4qwv" event={"ID":"7efb9739-c1e0-4214-8b5f-1495d0d39e65","Type":"ContainerDied","Data":"1e78b5145e2525f8111e9ac7abf41bf0d2c92278f049ff898082d078e47bf97b"} Jul 22 03:36:32.115649 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:32.115567 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e78b5145e2525f8111e9ac7abf41bf0d2c92278f049ff898082d078e47bf97b" Jul 22 03:36:40.253173 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:40.253137 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:36:43.802155 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:43.802117 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:36:48.255515 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:48.255482 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:36:48.265723 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:48.265675 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ed08a5-b4qwv"] Jul 22 03:36:48.268201 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:48.268180 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ed08a5-b4qwv"] Jul 22 03:36:48.706979 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:48.706944 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:36:48.784822 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:48.784790 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:36:49.901791 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:49.901757 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7efb9739-c1e0-4214-8b5f-1495d0d39e65" path="/var/lib/kubelet/pods/7efb9739-c1e0-4214-8b5f-1495d0d39e65/volumes" Jul 22 03:36:54.871771 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:54.871733 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:36:54.976950 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:54.976920 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:36:55.187499 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:55.187459 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:36:55.417689 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:55.417655 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:36:55.797023 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:55.796978 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:36:56.118638 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:56.118556 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:36:56.646708 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:56.646677 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:36:59.332823 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:59.332780 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:36:59.436823 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:59.436792 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:36:59.510344 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:59.510308 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:36:59.871478 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:36:59.871445 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:37:00.137698 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:37:00.137662 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:37:00.238280 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:37:00.238247 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:37:03.899455 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:37:03.899367 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:37:04.258296 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:37:04.258208 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:37:04.617296 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:37:04.617199 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:37:04.956417 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:37:04.956382 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:37:05.382150 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:37:05.382112 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:37:07.411211 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:37:07.411158 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:37:07.511711 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:37:07.511682 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:37:07.608562 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:37:07.608529 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:37:08.038821 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:37:08.038789 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:37:08.308565 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:37:08.308475 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:37:08.406247 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:37:08.406210 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:37:10.036226 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:37:10.036189 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:37:10.145112 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:37:10.145080 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:37:10.874736 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:37:10.874704 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:37:10.974508 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:37:10.974474 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:37:11.834599 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:37:11.834564 2582 scope.go:117] "RemoveContainer" containerID="a54f21eced5f9ea43edeb3ce0d899562146aed20cd97093d471dbd7606a5dc8a" Jul 22 03:37:11.938610 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:37:11.938576 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:37:17.992537 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:37:17.992504 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:37:21.193836 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:37:21.193800 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:37:21.294711 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:37:21.294673 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:37:29.176802 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:37:29.176763 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:37:29.276292 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:37:29.276260 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:37:31.318090 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:37:31.318057 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:37:31.422991 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:37:31.422953 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:37:34.570165 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:37:34.570131 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:38:12.844914 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:38:12.844879 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:38:16.479862 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:38:16.479828 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:38:30.573256 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:38:30.573219 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:38:30.631313 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:38:30.631275 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:38:30.731447 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:38:30.731409 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:38:39.679255 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:38:39.679216 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:38:39.751710 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:38:39.751675 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:38:47.398572 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:38:47.398536 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:39:05.282207 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:39:05.282170 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:39:24.042825 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:39:24.042791 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:39:42.884341 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:39:42.884301 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:40:01.777060 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:40:01.776962 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:40:13.116430 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:40:13.116396 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:40:13.945433 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:40:13.945403 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:40:18.182623 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:40:18.182589 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:40:35.846100 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:40:35.846005 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:40:38.881057 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:40:38.881002 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:40:52.155197 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:40:52.155159 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:40:52.232568 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:40:52.232533 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:41:03.956031 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:41:03.955988 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:41:07.489521 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:41:07.489482 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:41:11.885771 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:41:11.885737 2582 scope.go:117] "RemoveContainer" containerID="7b79dca1650811ea711fa9b4de3abcfb15bdabf043897cfb8ca225f48835d28d" Jul 22 03:41:23.979425 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:41:23.979385 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:41:41.182209 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:41:41.182173 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:41:47.221901 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:41:47.221866 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:41:48.100886 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:41:48.100849 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:41:59.127236 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:41:59.127204 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:41:59.782792 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:41:59.782752 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:42:09.077922 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:42:09.077883 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:42:11.086966 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:42:11.086925 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:42:11.188942 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:42:11.188906 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:42:11.908429 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:42:11.908401 2582 scope.go:117] "RemoveContainer" containerID="39112ebd161208e64c4e83a790f7522b30e1be057c007a201f1db67f5b29f7ba" Jul 22 03:42:11.917505 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:42:11.917484 2582 scope.go:117] "RemoveContainer" containerID="52d2da316d8e203bc5e486b911d697fdd234321f856c06f4135734efc843663a" Jul 22 03:42:22.006710 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:42:22.006680 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:42:22.108091 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:42:22.108059 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:42:26.345221 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:42:26.345185 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:42:30.033000 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:42:30.032962 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:42:30.128688 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:42:30.128651 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:42:30.586488 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:42:30.586447 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:42:48.280784 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:42:48.280745 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:43:04.082773 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:04.082689 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:43:10.942721 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:10.942687 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:43:11.828315 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:11.828286 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:43:11.946412 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:11.946378 2582 scope.go:117] "RemoveContainer" containerID="3b5f9dc5fbedde4c1df270772f67d6f2628098f2ae3223c3e4fe7823ff638f28" Jul 22 03:43:12.844528 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:12.844492 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7431-140554450338480"] Jul 22 03:43:12.844994 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:12.844929 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7efb9739-c1e0-4214-8b5f-1495d0d39e65" containerName="revoke-keys" Jul 22 03:43:12.844994 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:12.844944 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7efb9739-c1e0-4214-8b5f-1495d0d39e65" containerName="revoke-keys" Jul 22 03:43:12.850617 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:12.845036 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7efb9739-c1e0-4214-8b5f-1495d0d39e65" containerName="revoke-keys" Jul 22 03:43:12.850617 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:12.849613 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7431-140554450338480" Jul 22 03:43:12.852452 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:12.852423 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-55s49\"" Jul 22 03:43:12.855232 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:12.855211 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7431-140554450338480"] Jul 22 03:43:12.972385 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:12.972343 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pb22n\" (UniqueName: \"kubernetes.io/projected/fd278bd5-daf0-4e58-9d13-70d6d2374ab0-kube-api-access-pb22n\") pod \"test-curl-7431-140554450338480\" (UID: \"fd278bd5-daf0-4e58-9d13-70d6d2374ab0\") " pod="opendatahub/test-curl-7431-140554450338480" Jul 22 03:43:13.073194 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:13.073160 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb22n\" (UniqueName: \"kubernetes.io/projected/fd278bd5-daf0-4e58-9d13-70d6d2374ab0-kube-api-access-pb22n\") pod \"test-curl-7431-140554450338480\" (UID: \"fd278bd5-daf0-4e58-9d13-70d6d2374ab0\") " pod="opendatahub/test-curl-7431-140554450338480" Jul 22 03:43:13.082162 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:13.082134 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb22n\" (UniqueName: \"kubernetes.io/projected/fd278bd5-daf0-4e58-9d13-70d6d2374ab0-kube-api-access-pb22n\") pod \"test-curl-7431-140554450338480\" (UID: \"fd278bd5-daf0-4e58-9d13-70d6d2374ab0\") " pod="opendatahub/test-curl-7431-140554450338480" Jul 22 03:43:13.160450 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:13.160414 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7431-140554450338480" Jul 22 03:43:13.297324 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:13.297302 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7431-140554450338480"] Jul 22 03:43:13.299147 ip-10-0-131-67 kubenswrapper[2582]: W0722 03:43:13.299106 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfd278bd5_daf0_4e58_9d13_70d6d2374ab0.slice/crio-4b96731fb2bd12f782f36ab3d7cd8364e1ff8c94ae93e058f53945ab595e8cd0 WatchSource:0}: Error finding container 4b96731fb2bd12f782f36ab3d7cd8364e1ff8c94ae93e058f53945ab595e8cd0: Status 404 returned error can't find the container with id 4b96731fb2bd12f782f36ab3d7cd8364e1ff8c94ae93e058f53945ab595e8cd0 Jul 22 03:43:13.300487 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:13.300466 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 03:43:13.517570 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:13.517486 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7431-140554450338480" event={"ID":"fd278bd5-daf0-4e58-9d13-70d6d2374ab0","Type":"ContainerStarted","Data":"4b96731fb2bd12f782f36ab3d7cd8364e1ff8c94ae93e058f53945ab595e8cd0"} Jul 22 03:43:14.525099 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:14.525061 2582 generic.go:356] "Generic (PLEG): container finished" podID="fd278bd5-daf0-4e58-9d13-70d6d2374ab0" containerID="c2c2cf315a8ff0ed3cf29ae0b663b08296636f286b49f6dd8e92876c67621679" exitCode=0 Jul 22 03:43:14.525476 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:14.525122 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7431-140554450338480" event={"ID":"fd278bd5-daf0-4e58-9d13-70d6d2374ab0","Type":"ContainerDied","Data":"c2c2cf315a8ff0ed3cf29ae0b663b08296636f286b49f6dd8e92876c67621679"} Jul 22 03:43:15.675539 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:15.675513 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7431-140554450338480" Jul 22 03:43:15.696381 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:15.696350 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7431-140554450338480_fd278bd5-daf0-4e58-9d13-70d6d2374ab0/test-curl-7431-140554450338480/0.log" Jul 22 03:43:15.721883 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:15.721854 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7431-140554450338480"] Jul 22 03:43:15.726930 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:15.726903 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7431-140554450338480"] Jul 22 03:43:15.796115 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:15.796006 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pb22n\" (UniqueName: \"kubernetes.io/projected/fd278bd5-daf0-4e58-9d13-70d6d2374ab0-kube-api-access-pb22n\") pod \"fd278bd5-daf0-4e58-9d13-70d6d2374ab0\" (UID: \"fd278bd5-daf0-4e58-9d13-70d6d2374ab0\") " Jul 22 03:43:15.797969 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:15.797938 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd278bd5-daf0-4e58-9d13-70d6d2374ab0-kube-api-access-pb22n" (OuterVolumeSpecName: "kube-api-access-pb22n") pod "fd278bd5-daf0-4e58-9d13-70d6d2374ab0" (UID: "fd278bd5-daf0-4e58-9d13-70d6d2374ab0"). InnerVolumeSpecName "kube-api-access-pb22n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 03:43:15.856892 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:15.856860 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7431-140554450340400"] Jul 22 03:43:15.857309 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:15.857295 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd278bd5-daf0-4e58-9d13-70d6d2374ab0" containerName="test-curl-7431-140554450338480" Jul 22 03:43:15.857360 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:15.857311 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd278bd5-daf0-4e58-9d13-70d6d2374ab0" containerName="test-curl-7431-140554450338480" Jul 22 03:43:15.857396 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:15.857392 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd278bd5-daf0-4e58-9d13-70d6d2374ab0" containerName="test-curl-7431-140554450338480" Jul 22 03:43:15.862624 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:15.862607 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7431-140554450340400" Jul 22 03:43:15.866974 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:15.866949 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7431-140554450340400"] Jul 22 03:43:15.897318 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:15.897286 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pb22n\" (UniqueName: \"kubernetes.io/projected/fd278bd5-daf0-4e58-9d13-70d6d2374ab0-kube-api-access-pb22n\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 03:43:15.902828 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:15.902801 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd278bd5-daf0-4e58-9d13-70d6d2374ab0" path="/var/lib/kubelet/pods/fd278bd5-daf0-4e58-9d13-70d6d2374ab0/volumes" Jul 22 03:43:15.998410 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:15.998376 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58ts4\" (UniqueName: \"kubernetes.io/projected/5ca1559c-3998-42bb-92cb-7c901ab11511-kube-api-access-58ts4\") pod \"test-curl-7431-140554450340400\" (UID: \"5ca1559c-3998-42bb-92cb-7c901ab11511\") " pod="opendatahub/test-curl-7431-140554450340400" Jul 22 03:43:16.099555 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:16.099446 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58ts4\" (UniqueName: \"kubernetes.io/projected/5ca1559c-3998-42bb-92cb-7c901ab11511-kube-api-access-58ts4\") pod \"test-curl-7431-140554450340400\" (UID: \"5ca1559c-3998-42bb-92cb-7c901ab11511\") " pod="opendatahub/test-curl-7431-140554450340400" Jul 22 03:43:16.107319 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:16.107288 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58ts4\" (UniqueName: \"kubernetes.io/projected/5ca1559c-3998-42bb-92cb-7c901ab11511-kube-api-access-58ts4\") pod \"test-curl-7431-140554450340400\" (UID: \"5ca1559c-3998-42bb-92cb-7c901ab11511\") " pod="opendatahub/test-curl-7431-140554450340400" Jul 22 03:43:16.174052 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:16.174003 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7431-140554450340400" Jul 22 03:43:16.316068 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:16.316041 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7431-140554450340400"] Jul 22 03:43:16.318110 ip-10-0-131-67 kubenswrapper[2582]: W0722 03:43:16.318081 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5ca1559c_3998_42bb_92cb_7c901ab11511.slice/crio-1109855f6e74baa404095c2848bbcb278fc231de70da765c76f97bcff75c9534 WatchSource:0}: Error finding container 1109855f6e74baa404095c2848bbcb278fc231de70da765c76f97bcff75c9534: Status 404 returned error can't find the container with id 1109855f6e74baa404095c2848bbcb278fc231de70da765c76f97bcff75c9534 Jul 22 03:43:16.540514 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:16.540461 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7431-140554450340400" event={"ID":"5ca1559c-3998-42bb-92cb-7c901ab11511","Type":"ContainerStarted","Data":"1109855f6e74baa404095c2848bbcb278fc231de70da765c76f97bcff75c9534"} Jul 22 03:43:16.541881 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:16.541859 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7431-140554450338480" Jul 22 03:43:16.542048 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:16.541867 2582 scope.go:117] "RemoveContainer" containerID="c2c2cf315a8ff0ed3cf29ae0b663b08296636f286b49f6dd8e92876c67621679" Jul 22 03:43:17.550581 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:17.550548 2582 generic.go:356] "Generic (PLEG): container finished" podID="5ca1559c-3998-42bb-92cb-7c901ab11511" containerID="a23ea6467d3fd7c56220434166ab00223c31d66bf35f2027dc7e4b05a5cd9cbd" exitCode=0 Jul 22 03:43:17.550996 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:17.550624 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7431-140554450340400" event={"ID":"5ca1559c-3998-42bb-92cb-7c901ab11511","Type":"ContainerDied","Data":"a23ea6467d3fd7c56220434166ab00223c31d66bf35f2027dc7e4b05a5cd9cbd"} Jul 22 03:43:18.702185 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:18.702161 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7431-140554450340400" Jul 22 03:43:18.722427 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:18.722397 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7431-140554450340400_5ca1559c-3998-42bb-92cb-7c901ab11511/test-curl-7431-140554450340400/0.log" Jul 22 03:43:18.724684 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:18.724660 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-58ts4\" (UniqueName: \"kubernetes.io/projected/5ca1559c-3998-42bb-92cb-7c901ab11511-kube-api-access-58ts4\") pod \"5ca1559c-3998-42bb-92cb-7c901ab11511\" (UID: \"5ca1559c-3998-42bb-92cb-7c901ab11511\") " Jul 22 03:43:18.726675 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:18.726649 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ca1559c-3998-42bb-92cb-7c901ab11511-kube-api-access-58ts4" (OuterVolumeSpecName: "kube-api-access-58ts4") pod "5ca1559c-3998-42bb-92cb-7c901ab11511" (UID: "5ca1559c-3998-42bb-92cb-7c901ab11511"). InnerVolumeSpecName "kube-api-access-58ts4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 03:43:18.750663 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:18.750630 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7431-140554450340400"] Jul 22 03:43:18.753002 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:18.752980 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7431-140554450340400"] Jul 22 03:43:18.826002 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:18.825908 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58ts4\" (UniqueName: \"kubernetes.io/projected/5ca1559c-3998-42bb-92cb-7c901ab11511-kube-api-access-58ts4\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 03:43:18.924711 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:18.924674 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7431-140554450375712"] Jul 22 03:43:18.925137 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:18.925123 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ca1559c-3998-42bb-92cb-7c901ab11511" containerName="test-curl-7431-140554450340400" Jul 22 03:43:18.925137 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:18.925138 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ca1559c-3998-42bb-92cb-7c901ab11511" containerName="test-curl-7431-140554450340400" Jul 22 03:43:18.925245 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:18.925203 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ca1559c-3998-42bb-92cb-7c901ab11511" containerName="test-curl-7431-140554450340400" Jul 22 03:43:18.928207 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:18.928188 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7431-140554450375712" Jul 22 03:43:18.934165 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:18.934136 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7431-140554450375712"] Jul 22 03:43:19.027710 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:19.027670 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wt6k\" (UniqueName: \"kubernetes.io/projected/9ad226fa-1e75-4b03-845f-5d7644db7785-kube-api-access-8wt6k\") pod \"test-curl-7431-140554450375712\" (UID: \"9ad226fa-1e75-4b03-845f-5d7644db7785\") " pod="opendatahub/test-curl-7431-140554450375712" Jul 22 03:43:19.128262 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:19.128224 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wt6k\" (UniqueName: \"kubernetes.io/projected/9ad226fa-1e75-4b03-845f-5d7644db7785-kube-api-access-8wt6k\") pod \"test-curl-7431-140554450375712\" (UID: \"9ad226fa-1e75-4b03-845f-5d7644db7785\") " pod="opendatahub/test-curl-7431-140554450375712" Jul 22 03:43:19.137398 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:19.137364 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wt6k\" (UniqueName: \"kubernetes.io/projected/9ad226fa-1e75-4b03-845f-5d7644db7785-kube-api-access-8wt6k\") pod \"test-curl-7431-140554450375712\" (UID: \"9ad226fa-1e75-4b03-845f-5d7644db7785\") " pod="opendatahub/test-curl-7431-140554450375712" Jul 22 03:43:19.238568 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:19.238534 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7431-140554450375712" Jul 22 03:43:19.378612 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:19.378533 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7431-140554450375712"] Jul 22 03:43:19.379024 ip-10-0-131-67 kubenswrapper[2582]: W0722 03:43:19.378986 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9ad226fa_1e75_4b03_845f_5d7644db7785.slice/crio-24cc0c9e1c77888cd077cbaa409ad984ad9d0e69c560b5b3f13def152ef0257b WatchSource:0}: Error finding container 24cc0c9e1c77888cd077cbaa409ad984ad9d0e69c560b5b3f13def152ef0257b: Status 404 returned error can't find the container with id 24cc0c9e1c77888cd077cbaa409ad984ad9d0e69c560b5b3f13def152ef0257b Jul 22 03:43:19.563620 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:19.563590 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7431-140554450340400" Jul 22 03:43:19.563798 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:19.563622 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1109855f6e74baa404095c2848bbcb278fc231de70da765c76f97bcff75c9534" Jul 22 03:43:19.564861 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:19.564832 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7431-140554450375712" event={"ID":"9ad226fa-1e75-4b03-845f-5d7644db7785","Type":"ContainerStarted","Data":"24cc0c9e1c77888cd077cbaa409ad984ad9d0e69c560b5b3f13def152ef0257b"} Jul 22 03:43:19.574176 ip-10-0-131-67 kubenswrapper[2582]: E0722 03:43:19.574146 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7431-140554450340400\" is forbidden: User \"system:node:ip-10-0-131-67.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-67.ec2.internal' and this object" podUID="5ca1559c-3998-42bb-92cb-7c901ab11511" pod="opendatahub/test-curl-7431-140554450340400" Jul 22 03:43:19.901162 ip-10-0-131-67 kubenswrapper[2582]: E0722 03:43:19.901127 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7431-140554450340400\" is forbidden: User \"system:node:ip-10-0-131-67.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-67.ec2.internal' and this object" podUID="5ca1559c-3998-42bb-92cb-7c901ab11511" pod="opendatahub/test-curl-7431-140554450340400" Jul 22 03:43:19.902802 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:19.902782 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ca1559c-3998-42bb-92cb-7c901ab11511" path="/var/lib/kubelet/pods/5ca1559c-3998-42bb-92cb-7c901ab11511/volumes" Jul 22 03:43:20.572162 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:20.572125 2582 generic.go:356] "Generic (PLEG): container finished" podID="9ad226fa-1e75-4b03-845f-5d7644db7785" containerID="3a39e039bc79b701919f80c1358514a5cda95c57b84a1b265e748a4c3ebb3645" exitCode=0 Jul 22 03:43:20.572359 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:20.572207 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7431-140554450375712" event={"ID":"9ad226fa-1e75-4b03-845f-5d7644db7785","Type":"ContainerDied","Data":"3a39e039bc79b701919f80c1358514a5cda95c57b84a1b265e748a4c3ebb3645"} Jul 22 03:43:21.716484 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:21.716459 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7431-140554450375712" Jul 22 03:43:21.740447 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:21.740420 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7431-140554450375712_9ad226fa-1e75-4b03-845f-5d7644db7785/test-curl-7431-140554450375712/0.log" Jul 22 03:43:21.749621 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:21.749597 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8wt6k\" (UniqueName: \"kubernetes.io/projected/9ad226fa-1e75-4b03-845f-5d7644db7785-kube-api-access-8wt6k\") pod \"9ad226fa-1e75-4b03-845f-5d7644db7785\" (UID: \"9ad226fa-1e75-4b03-845f-5d7644db7785\") " Jul 22 03:43:21.751640 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:21.751613 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ad226fa-1e75-4b03-845f-5d7644db7785-kube-api-access-8wt6k" (OuterVolumeSpecName: "kube-api-access-8wt6k") pod "9ad226fa-1e75-4b03-845f-5d7644db7785" (UID: "9ad226fa-1e75-4b03-845f-5d7644db7785"). InnerVolumeSpecName "kube-api-access-8wt6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 03:43:21.766849 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:21.766826 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7431-140554450375712"] Jul 22 03:43:21.770701 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:21.770681 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7431-140554450375712"] Jul 22 03:43:21.851207 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:21.851143 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wt6k\" (UniqueName: \"kubernetes.io/projected/9ad226fa-1e75-4b03-845f-5d7644db7785-kube-api-access-8wt6k\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 03:43:21.901782 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:21.901750 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ad226fa-1e75-4b03-845f-5d7644db7785" path="/var/lib/kubelet/pods/9ad226fa-1e75-4b03-845f-5d7644db7785/volumes" Jul 22 03:43:21.902138 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:21.902125 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7431-140554450375712"] Jul 22 03:43:21.902510 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:21.902496 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ad226fa-1e75-4b03-845f-5d7644db7785" containerName="test-curl-7431-140554450375712" Jul 22 03:43:21.902510 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:21.902508 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ad226fa-1e75-4b03-845f-5d7644db7785" containerName="test-curl-7431-140554450375712" Jul 22 03:43:21.902637 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:21.902582 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ad226fa-1e75-4b03-845f-5d7644db7785" containerName="test-curl-7431-140554450375712" Jul 22 03:43:21.906939 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:21.906911 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7431-140554450375712"] Jul 22 03:43:21.907075 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:21.907031 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7431-140554450375712" Jul 22 03:43:21.952035 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:21.951988 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvfkt\" (UniqueName: \"kubernetes.io/projected/159c4277-43c6-4d2b-bf04-3097a499601b-kube-api-access-jvfkt\") pod \"test-curl-7431-140554450375712\" (UID: \"159c4277-43c6-4d2b-bf04-3097a499601b\") " pod="opendatahub/test-curl-7431-140554450375712" Jul 22 03:43:22.053174 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:22.053142 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvfkt\" (UniqueName: \"kubernetes.io/projected/159c4277-43c6-4d2b-bf04-3097a499601b-kube-api-access-jvfkt\") pod \"test-curl-7431-140554450375712\" (UID: \"159c4277-43c6-4d2b-bf04-3097a499601b\") " pod="opendatahub/test-curl-7431-140554450375712" Jul 22 03:43:22.068493 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:22.068467 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvfkt\" (UniqueName: \"kubernetes.io/projected/159c4277-43c6-4d2b-bf04-3097a499601b-kube-api-access-jvfkt\") pod \"test-curl-7431-140554450375712\" (UID: \"159c4277-43c6-4d2b-bf04-3097a499601b\") " pod="opendatahub/test-curl-7431-140554450375712" Jul 22 03:43:22.218029 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:22.217992 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7431-140554450375712" Jul 22 03:43:22.361253 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:22.361230 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7431-140554450375712"] Jul 22 03:43:22.362984 ip-10-0-131-67 kubenswrapper[2582]: W0722 03:43:22.362959 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod159c4277_43c6_4d2b_bf04_3097a499601b.slice/crio-d04ce02293880f52ff5b29d047a520b1ce951dc0eff65b4b9e4dae10326ce2b9 WatchSource:0}: Error finding container d04ce02293880f52ff5b29d047a520b1ce951dc0eff65b4b9e4dae10326ce2b9: Status 404 returned error can't find the container with id d04ce02293880f52ff5b29d047a520b1ce951dc0eff65b4b9e4dae10326ce2b9 Jul 22 03:43:22.584885 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:22.584796 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7431-140554450375712" Jul 22 03:43:22.584885 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:22.584854 2582 scope.go:117] "RemoveContainer" containerID="3a39e039bc79b701919f80c1358514a5cda95c57b84a1b265e748a4c3ebb3645" Jul 22 03:43:22.586288 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:22.586052 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7431-140554450375712" event={"ID":"159c4277-43c6-4d2b-bf04-3097a499601b","Type":"ContainerStarted","Data":"d04ce02293880f52ff5b29d047a520b1ce951dc0eff65b4b9e4dae10326ce2b9"} Jul 22 03:43:23.074037 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:23.073994 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:43:23.593283 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:23.593249 2582 generic.go:356] "Generic (PLEG): container finished" podID="159c4277-43c6-4d2b-bf04-3097a499601b" containerID="db3ef10f2ad7151bdaa688f3e061433200c2fbe37779a907fba30769bbd8bac6" exitCode=0 Jul 22 03:43:23.593456 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:23.593327 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7431-140554450375712" event={"ID":"159c4277-43c6-4d2b-bf04-3097a499601b","Type":"ContainerDied","Data":"db3ef10f2ad7151bdaa688f3e061433200c2fbe37779a907fba30769bbd8bac6"} Jul 22 03:43:24.756805 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:24.756785 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7431-140554450375712" Jul 22 03:43:24.777669 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:24.777644 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jvfkt\" (UniqueName: \"kubernetes.io/projected/159c4277-43c6-4d2b-bf04-3097a499601b-kube-api-access-jvfkt\") pod \"159c4277-43c6-4d2b-bf04-3097a499601b\" (UID: \"159c4277-43c6-4d2b-bf04-3097a499601b\") " Jul 22 03:43:24.778724 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:24.778683 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7431-140554450375712_159c4277-43c6-4d2b-bf04-3097a499601b/test-curl-7431-140554450375712/0.log" Jul 22 03:43:24.780144 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:24.780120 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/159c4277-43c6-4d2b-bf04-3097a499601b-kube-api-access-jvfkt" (OuterVolumeSpecName: "kube-api-access-jvfkt") pod "159c4277-43c6-4d2b-bf04-3097a499601b" (UID: "159c4277-43c6-4d2b-bf04-3097a499601b"). InnerVolumeSpecName "kube-api-access-jvfkt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 03:43:24.808403 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:24.808376 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7431-140554450375712"] Jul 22 03:43:24.812256 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:24.812234 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7431-140554450375712"] Jul 22 03:43:24.878788 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:24.878761 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvfkt\" (UniqueName: \"kubernetes.io/projected/159c4277-43c6-4d2b-bf04-3097a499601b-kube-api-access-jvfkt\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 03:43:24.940955 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:24.940922 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7431-140554450376720"] Jul 22 03:43:24.941405 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:24.941392 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="159c4277-43c6-4d2b-bf04-3097a499601b" containerName="test-curl-7431-140554450375712" Jul 22 03:43:24.941460 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:24.941408 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="159c4277-43c6-4d2b-bf04-3097a499601b" containerName="test-curl-7431-140554450375712" Jul 22 03:43:24.941491 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:24.941478 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="159c4277-43c6-4d2b-bf04-3097a499601b" containerName="test-curl-7431-140554450375712" Jul 22 03:43:24.944533 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:24.944519 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7431-140554450376720" Jul 22 03:43:24.952920 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:24.952895 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7431-140554450376720"] Jul 22 03:43:24.979596 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:24.979554 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpgz8\" (UniqueName: \"kubernetes.io/projected/449a6e95-c8a9-480e-8d7f-1389980361f6-kube-api-access-jpgz8\") pod \"test-curl-7431-140554450376720\" (UID: \"449a6e95-c8a9-480e-8d7f-1389980361f6\") " pod="opendatahub/test-curl-7431-140554450376720" Jul 22 03:43:25.080941 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:25.080906 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpgz8\" (UniqueName: \"kubernetes.io/projected/449a6e95-c8a9-480e-8d7f-1389980361f6-kube-api-access-jpgz8\") pod \"test-curl-7431-140554450376720\" (UID: \"449a6e95-c8a9-480e-8d7f-1389980361f6\") " pod="opendatahub/test-curl-7431-140554450376720" Jul 22 03:43:25.089017 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:25.088993 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpgz8\" (UniqueName: \"kubernetes.io/projected/449a6e95-c8a9-480e-8d7f-1389980361f6-kube-api-access-jpgz8\") pod \"test-curl-7431-140554450376720\" (UID: \"449a6e95-c8a9-480e-8d7f-1389980361f6\") " pod="opendatahub/test-curl-7431-140554450376720" Jul 22 03:43:25.254815 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:25.254729 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7431-140554450376720" Jul 22 03:43:25.395890 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:25.395862 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7431-140554450376720"] Jul 22 03:43:25.396707 ip-10-0-131-67 kubenswrapper[2582]: W0722 03:43:25.396671 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod449a6e95_c8a9_480e_8d7f_1389980361f6.slice/crio-d5e6e5e2d8538bcf3e457778279fa928e84320de7920f7d3953ff7010926e6e7 WatchSource:0}: Error finding container d5e6e5e2d8538bcf3e457778279fa928e84320de7920f7d3953ff7010926e6e7: Status 404 returned error can't find the container with id d5e6e5e2d8538bcf3e457778279fa928e84320de7920f7d3953ff7010926e6e7 Jul 22 03:43:25.607921 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:25.607834 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7431-140554450376720" event={"ID":"449a6e95-c8a9-480e-8d7f-1389980361f6","Type":"ContainerStarted","Data":"d5e6e5e2d8538bcf3e457778279fa928e84320de7920f7d3953ff7010926e6e7"} Jul 22 03:43:25.609040 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:25.609004 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7431-140554450375712" Jul 22 03:43:25.609197 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:25.609051 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d04ce02293880f52ff5b29d047a520b1ce951dc0eff65b4b9e4dae10326ce2b9" Jul 22 03:43:25.619712 ip-10-0-131-67 kubenswrapper[2582]: E0722 03:43:25.619681 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7431-140554450375712\" is forbidden: User \"system:node:ip-10-0-131-67.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-67.ec2.internal' and this object" podUID="159c4277-43c6-4d2b-bf04-3097a499601b" pod="opendatahub/test-curl-7431-140554450375712" Jul 22 03:43:25.902602 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:25.902572 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="159c4277-43c6-4d2b-bf04-3097a499601b" path="/var/lib/kubelet/pods/159c4277-43c6-4d2b-bf04-3097a499601b/volumes" Jul 22 03:43:26.617286 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:26.617250 2582 generic.go:356] "Generic (PLEG): container finished" podID="449a6e95-c8a9-480e-8d7f-1389980361f6" containerID="c16e81f3b5561b4162a5fa55b1ec3c3989d4ba8494c534ed0632073d7208daee" exitCode=0 Jul 22 03:43:26.617468 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:26.617337 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7431-140554450376720" event={"ID":"449a6e95-c8a9-480e-8d7f-1389980361f6","Type":"ContainerDied","Data":"c16e81f3b5561b4162a5fa55b1ec3c3989d4ba8494c534ed0632073d7208daee"} Jul 22 03:43:27.761452 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:27.761422 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7431-140554450376720" Jul 22 03:43:27.783069 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:27.782986 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7431-140554450376720_449a6e95-c8a9-480e-8d7f-1389980361f6/test-curl-7431-140554450376720/0.log" Jul 22 03:43:27.811971 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:27.811938 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7431-140554450376720"] Jul 22 03:43:27.813584 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:27.813567 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jpgz8\" (UniqueName: \"kubernetes.io/projected/449a6e95-c8a9-480e-8d7f-1389980361f6-kube-api-access-jpgz8\") pod \"449a6e95-c8a9-480e-8d7f-1389980361f6\" (UID: \"449a6e95-c8a9-480e-8d7f-1389980361f6\") " Jul 22 03:43:27.814419 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:27.814400 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7431-140554450376720"] Jul 22 03:43:27.815563 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:27.815539 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/449a6e95-c8a9-480e-8d7f-1389980361f6-kube-api-access-jpgz8" (OuterVolumeSpecName: "kube-api-access-jpgz8") pod "449a6e95-c8a9-480e-8d7f-1389980361f6" (UID: "449a6e95-c8a9-480e-8d7f-1389980361f6"). InnerVolumeSpecName "kube-api-access-jpgz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 03:43:27.902232 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:27.902151 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="449a6e95-c8a9-480e-8d7f-1389980361f6" path="/var/lib/kubelet/pods/449a6e95-c8a9-480e-8d7f-1389980361f6/volumes" Jul 22 03:43:27.914409 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:27.914386 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpgz8\" (UniqueName: \"kubernetes.io/projected/449a6e95-c8a9-480e-8d7f-1389980361f6-kube-api-access-jpgz8\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 03:43:27.944653 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:27.944627 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7431-140554450376720"] Jul 22 03:43:27.945112 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:27.945100 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="449a6e95-c8a9-480e-8d7f-1389980361f6" containerName="test-curl-7431-140554450376720" Jul 22 03:43:27.945168 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:27.945113 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="449a6e95-c8a9-480e-8d7f-1389980361f6" containerName="test-curl-7431-140554450376720" Jul 22 03:43:27.945206 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:27.945198 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="449a6e95-c8a9-480e-8d7f-1389980361f6" containerName="test-curl-7431-140554450376720" Jul 22 03:43:27.948456 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:27.948440 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7431-140554450376720" Jul 22 03:43:27.954654 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:27.954621 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7431-140554450376720"] Jul 22 03:43:28.015697 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:28.015660 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hklr\" (UniqueName: \"kubernetes.io/projected/3c5a008e-0e8d-4189-90d5-5f21f2ded722-kube-api-access-8hklr\") pod \"test-curl-7431-140554450376720\" (UID: \"3c5a008e-0e8d-4189-90d5-5f21f2ded722\") " pod="opendatahub/test-curl-7431-140554450376720" Jul 22 03:43:28.116437 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:28.116398 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hklr\" (UniqueName: \"kubernetes.io/projected/3c5a008e-0e8d-4189-90d5-5f21f2ded722-kube-api-access-8hklr\") pod \"test-curl-7431-140554450376720\" (UID: \"3c5a008e-0e8d-4189-90d5-5f21f2ded722\") " pod="opendatahub/test-curl-7431-140554450376720" Jul 22 03:43:28.124352 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:28.124319 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hklr\" (UniqueName: \"kubernetes.io/projected/3c5a008e-0e8d-4189-90d5-5f21f2ded722-kube-api-access-8hklr\") pod \"test-curl-7431-140554450376720\" (UID: \"3c5a008e-0e8d-4189-90d5-5f21f2ded722\") " pod="opendatahub/test-curl-7431-140554450376720" Jul 22 03:43:28.260327 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:28.260244 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7431-140554450376720" Jul 22 03:43:28.393652 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:28.393627 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7431-140554450376720"] Jul 22 03:43:28.395871 ip-10-0-131-67 kubenswrapper[2582]: W0722 03:43:28.395844 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c5a008e_0e8d_4189_90d5_5f21f2ded722.slice/crio-dea8ba7caf46a33efc339d4f76d9ff4fb196452a8c2113a323f1481c0bc6802d WatchSource:0}: Error finding container dea8ba7caf46a33efc339d4f76d9ff4fb196452a8c2113a323f1481c0bc6802d: Status 404 returned error can't find the container with id dea8ba7caf46a33efc339d4f76d9ff4fb196452a8c2113a323f1481c0bc6802d Jul 22 03:43:28.630972 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:28.630939 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7431-140554450376720" event={"ID":"3c5a008e-0e8d-4189-90d5-5f21f2ded722","Type":"ContainerStarted","Data":"dea8ba7caf46a33efc339d4f76d9ff4fb196452a8c2113a323f1481c0bc6802d"} Jul 22 03:43:28.632146 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:28.632126 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7431-140554450376720" Jul 22 03:43:28.632271 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:28.632173 2582 scope.go:117] "RemoveContainer" containerID="c16e81f3b5561b4162a5fa55b1ec3c3989d4ba8494c534ed0632073d7208daee" Jul 22 03:43:29.639581 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:29.639544 2582 generic.go:356] "Generic (PLEG): container finished" podID="3c5a008e-0e8d-4189-90d5-5f21f2ded722" containerID="ca9647b14c95932b3813c661b73193fed9aba087e7184e3d7d98f1d3ae818eac" exitCode=0 Jul 22 03:43:29.640040 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:29.639621 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7431-140554450376720" event={"ID":"3c5a008e-0e8d-4189-90d5-5f21f2ded722","Type":"ContainerDied","Data":"ca9647b14c95932b3813c661b73193fed9aba087e7184e3d7d98f1d3ae818eac"} Jul 22 03:43:30.788796 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:30.788771 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7431-140554450376720" Jul 22 03:43:30.811368 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:30.811338 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7431-140554450376720_3c5a008e-0e8d-4189-90d5-5f21f2ded722/test-curl-7431-140554450376720/0.log" Jul 22 03:43:30.839411 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:30.839375 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7431-140554450376720"] Jul 22 03:43:30.840668 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:30.840645 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8hklr\" (UniqueName: \"kubernetes.io/projected/3c5a008e-0e8d-4189-90d5-5f21f2ded722-kube-api-access-8hklr\") pod \"3c5a008e-0e8d-4189-90d5-5f21f2ded722\" (UID: \"3c5a008e-0e8d-4189-90d5-5f21f2ded722\") " Jul 22 03:43:30.843160 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:30.843134 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c5a008e-0e8d-4189-90d5-5f21f2ded722-kube-api-access-8hklr" (OuterVolumeSpecName: "kube-api-access-8hklr") pod "3c5a008e-0e8d-4189-90d5-5f21f2ded722" (UID: "3c5a008e-0e8d-4189-90d5-5f21f2ded722"). InnerVolumeSpecName "kube-api-access-8hklr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 03:43:30.844628 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:30.844606 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7431-140554450376720"] Jul 22 03:43:30.941884 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:30.941789 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hklr\" (UniqueName: \"kubernetes.io/projected/3c5a008e-0e8d-4189-90d5-5f21f2ded722-kube-api-access-8hklr\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 03:43:31.050790 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:31.050753 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-c19e4b-dxxbb"] Jul 22 03:43:31.051190 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:31.051171 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c5a008e-0e8d-4189-90d5-5f21f2ded722" containerName="test-curl-7431-140554450376720" Jul 22 03:43:31.051190 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:31.051189 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c5a008e-0e8d-4189-90d5-5f21f2ded722" containerName="test-curl-7431-140554450376720" Jul 22 03:43:31.051336 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:31.051264 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c5a008e-0e8d-4189-90d5-5f21f2ded722" containerName="test-curl-7431-140554450376720" Jul 22 03:43:31.054437 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:31.054417 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-c19e4b-dxxbb" Jul 22 03:43:31.057290 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:31.057269 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 03:43:31.057390 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:31.057349 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cw8l4\"" Jul 22 03:43:31.063722 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:31.063700 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-c19e4b-dxxbb"] Jul 22 03:43:31.143690 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:31.143662 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5ca50fd-572e-406c-9d17-d1c6b1d3e491-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-c19e4b-dxxbb\" (UID: \"f5ca50fd-572e-406c-9d17-d1c6b1d3e491\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-c19e4b-dxxbb" Jul 22 03:43:31.244665 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:31.244564 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5ca50fd-572e-406c-9d17-d1c6b1d3e491-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-c19e4b-dxxbb\" (UID: \"f5ca50fd-572e-406c-9d17-d1c6b1d3e491\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-c19e4b-dxxbb" Jul 22 03:43:31.245246 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:31.245225 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5ca50fd-572e-406c-9d17-d1c6b1d3e491-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-c19e4b-dxxbb\" (UID: \"f5ca50fd-572e-406c-9d17-d1c6b1d3e491\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-c19e4b-dxxbb" Jul 22 03:43:31.366620 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:31.366586 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-c19e4b-dxxbb" Jul 22 03:43:31.521857 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:31.521832 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-c19e4b-dxxbb"] Jul 22 03:43:31.523723 ip-10-0-131-67 kubenswrapper[2582]: W0722 03:43:31.523695 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5ca50fd_572e_406c_9d17_d1c6b1d3e491.slice/crio-7000c432be6db1275c2a700826bb861a2ad320a1008869e50c8c059d043389ed WatchSource:0}: Error finding container 7000c432be6db1275c2a700826bb861a2ad320a1008869e50c8c059d043389ed: Status 404 returned error can't find the container with id 7000c432be6db1275c2a700826bb861a2ad320a1008869e50c8c059d043389ed Jul 22 03:43:31.653973 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:31.653940 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7431-140554450376720" Jul 22 03:43:31.653973 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:31.653964 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dea8ba7caf46a33efc339d4f76d9ff4fb196452a8c2113a323f1481c0bc6802d" Jul 22 03:43:31.655236 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:31.655212 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-c19e4b-dxxbb" event={"ID":"f5ca50fd-572e-406c-9d17-d1c6b1d3e491","Type":"ContainerStarted","Data":"7000c432be6db1275c2a700826bb861a2ad320a1008869e50c8c059d043389ed"} Jul 22 03:43:31.664187 ip-10-0-131-67 kubenswrapper[2582]: E0722 03:43:31.664158 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7431-140554450376720\" is forbidden: User \"system:node:ip-10-0-131-67.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-67.ec2.internal' and this object" podUID="3c5a008e-0e8d-4189-90d5-5f21f2ded722" pod="opendatahub/test-curl-7431-140554450376720" Jul 22 03:43:31.753926 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:31.753893 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:43:31.881215 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:31.881183 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:43:31.902693 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:31.902664 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c5a008e-0e8d-4189-90d5-5f21f2ded722" path="/var/lib/kubelet/pods/3c5a008e-0e8d-4189-90d5-5f21f2ded722/volumes" Jul 22 03:43:32.662118 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:32.662086 2582 generic.go:356] "Generic (PLEG): container finished" podID="f5ca50fd-572e-406c-9d17-d1c6b1d3e491" containerID="e6c3f57d97e0c5abd360393f66532da98f338bc626e6d5c16676fe80b994914b" exitCode=0 Jul 22 03:43:32.662284 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:32.662179 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-c19e4b-dxxbb" event={"ID":"f5ca50fd-572e-406c-9d17-d1c6b1d3e491","Type":"ContainerDied","Data":"e6c3f57d97e0c5abd360393f66532da98f338bc626e6d5c16676fe80b994914b"} Jul 22 03:43:33.807829 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:33.807801 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-c19e4b-dxxbb" Jul 22 03:43:33.868398 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:33.868368 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5ca50fd-572e-406c-9d17-d1c6b1d3e491-maas-api-service-ca\") pod \"f5ca50fd-572e-406c-9d17-d1c6b1d3e491\" (UID: \"f5ca50fd-572e-406c-9d17-d1c6b1d3e491\") " Jul 22 03:43:33.868728 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:33.868703 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5ca50fd-572e-406c-9d17-d1c6b1d3e491-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f5ca50fd-572e-406c-9d17-d1c6b1d3e491" (UID: "f5ca50fd-572e-406c-9d17-d1c6b1d3e491"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 03:43:33.969552 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:33.969475 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5ca50fd-572e-406c-9d17-d1c6b1d3e491-maas-api-service-ca\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 03:43:34.675271 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:34.675242 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-c19e4b-dxxbb" Jul 22 03:43:34.675470 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:34.675274 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-c19e4b-dxxbb" event={"ID":"f5ca50fd-572e-406c-9d17-d1c6b1d3e491","Type":"ContainerDied","Data":"7000c432be6db1275c2a700826bb861a2ad320a1008869e50c8c059d043389ed"} Jul 22 03:43:34.675470 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:34.675316 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7000c432be6db1275c2a700826bb861a2ad320a1008869e50c8c059d043389ed" Jul 22 03:43:41.720062 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:41.720008 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:43:44.587298 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:44.587264 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:43:46.142086 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:46.142052 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:43:46.143156 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:46.143130 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-c19e4b-dxxbb"] Jul 22 03:43:46.145590 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:46.145568 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-c19e4b-dxxbb"] Jul 22 03:43:46.577104 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:46.576999 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:43:47.901746 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:47.901712 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5ca50fd-572e-406c-9d17-d1c6b1d3e491" path="/var/lib/kubelet/pods/f5ca50fd-572e-406c-9d17-d1c6b1d3e491/volumes" Jul 22 03:43:53.032918 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:53.032877 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:43:53.125064 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:53.125005 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:43:53.330639 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:53.330555 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:43:53.585539 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:53.585452 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:43:53.704616 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:53.704577 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-49462a-nb944"] Jul 22 03:43:53.705869 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:53.705840 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5ca50fd-572e-406c-9d17-d1c6b1d3e491" containerName="revoke-keys" Jul 22 03:43:53.706057 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:53.706044 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5ca50fd-572e-406c-9d17-d1c6b1d3e491" containerName="revoke-keys" Jul 22 03:43:53.706347 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:53.706333 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5ca50fd-572e-406c-9d17-d1c6b1d3e491" containerName="revoke-keys" Jul 22 03:43:53.711473 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:53.711007 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-49462a-nb944" Jul 22 03:43:53.712790 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:53.712764 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-49462a-nb944"] Jul 22 03:43:53.714312 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:53.714288 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 03:43:53.715203 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:53.715184 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cw8l4\"" Jul 22 03:43:53.847045 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:53.846924 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b802703-a392-4db0-9186-3ed8413cbf8a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-49462a-nb944\" (UID: \"0b802703-a392-4db0-9186-3ed8413cbf8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-49462a-nb944" Jul 22 03:43:53.948475 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:53.948437 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b802703-a392-4db0-9186-3ed8413cbf8a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-49462a-nb944\" (UID: \"0b802703-a392-4db0-9186-3ed8413cbf8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-49462a-nb944" Jul 22 03:43:53.949064 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:53.949045 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b802703-a392-4db0-9186-3ed8413cbf8a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-49462a-nb944\" (UID: \"0b802703-a392-4db0-9186-3ed8413cbf8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-49462a-nb944" Jul 22 03:43:54.023335 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:54.023296 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-49462a-nb944" Jul 22 03:43:54.173008 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:54.172984 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-49462a-nb944"] Jul 22 03:43:54.175736 ip-10-0-131-67 kubenswrapper[2582]: W0722 03:43:54.175704 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b802703_a392_4db0_9186_3ed8413cbf8a.slice/crio-9b6d1952aef842fcccb63985db38b81f1415242bd3ec5c8524c76e939701d064 WatchSource:0}: Error finding container 9b6d1952aef842fcccb63985db38b81f1415242bd3ec5c8524c76e939701d064: Status 404 returned error can't find the container with id 9b6d1952aef842fcccb63985db38b81f1415242bd3ec5c8524c76e939701d064 Jul 22 03:43:54.257253 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:54.257216 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:43:54.357307 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:54.357279 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:43:54.801478 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:54.801391 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-49462a-nb944" event={"ID":"0b802703-a392-4db0-9186-3ed8413cbf8a","Type":"ContainerStarted","Data":"766fdefae9c108468ffc2d4677039932d9125a25ebf1f3bad9e7b9de0da18a08"} Jul 22 03:43:54.801478 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:54.801451 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-49462a-nb944" event={"ID":"0b802703-a392-4db0-9186-3ed8413cbf8a","Type":"ContainerStarted","Data":"9b6d1952aef842fcccb63985db38b81f1415242bd3ec5c8524c76e939701d064"} Jul 22 03:43:54.821181 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:54.821129 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-49462a-nb944" podStartSLOduration=1.821111935 podStartE2EDuration="1.821111935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:43:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:43:54.818690062 +0000 UTC m=+2745.550310910" watchObservedRunningTime="2026-07-22 03:43:54.821111935 +0000 UTC m=+2745.552732818" Jul 22 03:43:55.808641 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:55.808610 2582 generic.go:356] "Generic (PLEG): container finished" podID="0b802703-a392-4db0-9186-3ed8413cbf8a" containerID="766fdefae9c108468ffc2d4677039932d9125a25ebf1f3bad9e7b9de0da18a08" exitCode=0 Jul 22 03:43:55.809052 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:55.808695 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-49462a-nb944" event={"ID":"0b802703-a392-4db0-9186-3ed8413cbf8a","Type":"ContainerDied","Data":"766fdefae9c108468ffc2d4677039932d9125a25ebf1f3bad9e7b9de0da18a08"} Jul 22 03:43:56.958218 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:56.958191 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-49462a-nb944" Jul 22 03:43:57.079285 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:57.079189 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b802703-a392-4db0-9186-3ed8413cbf8a-maas-api-service-ca\") pod \"0b802703-a392-4db0-9186-3ed8413cbf8a\" (UID: \"0b802703-a392-4db0-9186-3ed8413cbf8a\") " Jul 22 03:43:57.079569 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:57.079545 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b802703-a392-4db0-9186-3ed8413cbf8a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0b802703-a392-4db0-9186-3ed8413cbf8a" (UID: "0b802703-a392-4db0-9186-3ed8413cbf8a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 03:43:57.180873 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:57.180833 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b802703-a392-4db0-9186-3ed8413cbf8a-maas-api-service-ca\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\"" Jul 22 03:43:57.827233 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:57.827199 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-49462a-nb944" Jul 22 03:43:57.827233 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:57.827226 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-49462a-nb944" event={"ID":"0b802703-a392-4db0-9186-3ed8413cbf8a","Type":"ContainerDied","Data":"9b6d1952aef842fcccb63985db38b81f1415242bd3ec5c8524c76e939701d064"} Jul 22 03:43:57.827504 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:43:57.827296 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b6d1952aef842fcccb63985db38b81f1415242bd3ec5c8524c76e939701d064" Jul 22 03:44:04.269272 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:04.269243 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:44:06.981230 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:06.981194 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8k8vz"] Jul 22 03:44:08.786509 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:08.786471 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-49462a-nb944"] Jul 22 03:44:08.791224 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:08.791194 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:44:08.791367 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:08.791290 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-49462a-nb944"] Jul 22 03:44:09.207323 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:09.207287 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:44:09.910205 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:09.910169 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b802703-a392-4db0-9186-3ed8413cbf8a" path="/var/lib/kubelet/pods/0b802703-a392-4db0-9186-3ed8413cbf8a/volumes" Jul 22 03:44:15.614027 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:15.613985 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:44:15.713357 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:15.713327 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:44:15.929734 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:15.929703 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:44:16.030278 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:16.030245 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:44:16.224174 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:16.224087 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:44:16.477166 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:16.477082 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:44:22.671220 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:22.671188 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-lzsb5_6edeec1e-1428-4fe4-b99b-f4348dc39d74/manager/0.log" Jul 22 03:44:22.802163 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:22.802131 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-grbpl_c6393858-4cb7-4730-82ad-eac79608c284/manager/0.log" Jul 22 03:44:22.930907 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:22.930830 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6446b4cb87-mx6mt_ef264272-2474-4344-9d30-5ad28556cae8/manager/0.log" Jul 22 03:44:23.051433 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:23.051402 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-pnl96_20b8fa3f-a5f9-425b-8fd2-e52ee34adf8a/server/0.log" Jul 22 03:44:23.177198 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:23.177171 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-nbpjd_39934543-08e4-4456-b200-27ce181b98e8/manager/0.log" Jul 22 03:44:23.456576 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:23.456542 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-94fc48448-gsdv4_79701dcf-7b01-484f-8cc0-d6ce840733d1/copy-manifests/0.log" Jul 22 03:44:23.464960 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:23.464933 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-94fc48448-gsdv4_79701dcf-7b01-484f-8cc0-d6ce840733d1/manager/0.log" Jul 22 03:44:24.238617 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:24.238586 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29744850-vp62n_33141429-e38e-4dff-89d1-29ec69356888/cleanup/0.log" Jul 22 03:44:25.174054 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:25.173998 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d6k5j_6f7e9741-e996-4fd5-8363-c54e86d3dd05/util/0.log" Jul 22 03:44:25.181355 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:25.181325 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d6k5j_6f7e9741-e996-4fd5-8363-c54e86d3dd05/pull/0.log" Jul 22 03:44:25.188506 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:25.188479 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d6k5j_6f7e9741-e996-4fd5-8363-c54e86d3dd05/extract/0.log" Jul 22 03:44:25.317446 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:25.317417 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kvc54_3f608664-0cb4-47f7-9e41-ca87b2abe2cf/util/0.log" Jul 22 03:44:25.324892 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:25.324864 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kvc54_3f608664-0cb4-47f7-9e41-ca87b2abe2cf/pull/0.log" Jul 22 03:44:25.332190 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:25.332164 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kvc54_3f608664-0cb4-47f7-9e41-ca87b2abe2cf/extract/0.log" Jul 22 03:44:25.460593 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:25.460513 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lv5sb_bce77674-28ac-4fcf-8d29-73c7178ed338/util/0.log" Jul 22 03:44:25.468023 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:25.467994 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lv5sb_bce77674-28ac-4fcf-8d29-73c7178ed338/pull/0.log" Jul 22 03:44:25.474449 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:25.474422 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lv5sb_bce77674-28ac-4fcf-8d29-73c7178ed338/extract/0.log" Jul 22 03:44:25.600172 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:25.600142 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef187sg6_90fe6071-e5e0-4797-8252-17607fbc5def/util/0.log" Jul 22 03:44:25.608260 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:25.608237 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef187sg6_90fe6071-e5e0-4797-8252-17607fbc5def/pull/0.log" Jul 22 03:44:25.615569 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:25.615539 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef187sg6_90fe6071-e5e0-4797-8252-17607fbc5def/extract/0.log" Jul 22 03:44:25.886008 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:25.885982 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-l458v_0577bf79-9074-4ad0-a3f5-5f8bc286831e/manager/0.log" Jul 22 03:44:26.012234 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:26.012207 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-q4rcm_941e3af6-8144-4317-bec3-9e26bf4248f4/manager/0.log" Jul 22 03:44:26.136214 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:26.136132 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-dvrw9_32f52f23-526c-4e4b-a988-9d69c264dca9/kuadrant-console-plugin/0.log" Jul 22 03:44:26.269474 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:26.269445 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-sp78b_dc453083-f6e0-4809-9b8e-92034ab30c43/registry-server/0.log" Jul 22 03:44:26.552101 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:26.551995 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-8k8vz_18482f74-ab9c-4f92-b1f6-ea302e2419a2/limitador/0.log" Jul 22 03:44:26.684823 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:26.684792 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-9mb6v_36a3a681-1af5-4069-bb00-34d6ccdf9842/manager/0.log" Jul 22 03:44:27.300290 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:27.300262 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-684578bfd-2jd6x_d10d14d5-4415-431c-81c3-623c8050e7c8/kube-auth-proxy/0.log" Jul 22 03:44:27.535028 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:27.534986 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-lgdw9_9ca7bc57-768b-42fd-98df-ddddde7d3b08/istio-proxy/1.log" Jul 22 03:44:27.657925 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:27.657877 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-szpl4_56ca64c4-0973-4ed3-acb1-2d2ebfc80814/payload-pre-processing/0.log" Jul 22 03:44:27.782918 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:27.782888 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-tf965_877a033d-62c3-46d6-a0d5-f23139602b32/payload-processing/0.log" Jul 22 03:44:28.407317 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:28.407283 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-mnctl_cfa663d5-84bd-49ec-bdf6-6e964a2acc7c/main/0.log" Jul 22 03:44:28.907714 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:28.907686 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-jcc2f_18577aca-6f29-47a0-ac67-697b298b664d/main/0.log" Jul 22 03:44:35.065614 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:35.065535 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:44:35.227669 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:35.227633 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:44:37.902193 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:37.902159 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-lc5mn_5bd7c5db-509d-4f85-82ac-8e357591ae72/global-pull-secret-syncer/0.log" Jul 22 03:44:38.037769 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:38.037732 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" Jul 22 03:44:38.169406 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:38.169332 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-67.ec2.internal_5d8b9a289a92f6d831f68731b2b2419a/haproxy/0.log" Jul 22 03:44:42.536130 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:42.536101 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d6k5j_6f7e9741-e996-4fd5-8363-c54e86d3dd05/extract/0.log" Jul 22 03:44:42.573659 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:42.573628 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d6k5j_6f7e9741-e996-4fd5-8363-c54e86d3dd05/util/0.log" Jul 22 03:44:42.610526 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:42.610502 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d6k5j_6f7e9741-e996-4fd5-8363-c54e86d3dd05/pull/0.log" Jul 22 03:44:42.647544 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:42.647512 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kvc54_3f608664-0cb4-47f7-9e41-ca87b2abe2cf/extract/0.log" Jul 22 03:44:42.670265 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:42.670237 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kvc54_3f608664-0cb4-47f7-9e41-ca87b2abe2cf/util/0.log" Jul 22 03:44:42.696491 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:42.696467 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kvc54_3f608664-0cb4-47f7-9e41-ca87b2abe2cf/pull/0.log" Jul 22 03:44:42.724685 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:42.724658 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lv5sb_bce77674-28ac-4fcf-8d29-73c7178ed338/extract/0.log" Jul 22 03:44:42.753127 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:42.753102 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lv5sb_bce77674-28ac-4fcf-8d29-73c7178ed338/util/0.log" Jul 22 03:44:42.775932 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:42.775909 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lv5sb_bce77674-28ac-4fcf-8d29-73c7178ed338/pull/0.log" Jul 22 03:44:42.802903 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:42.802827 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef187sg6_90fe6071-e5e0-4797-8252-17607fbc5def/extract/0.log" Jul 22 03:44:42.824418 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:42.824393 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef187sg6_90fe6071-e5e0-4797-8252-17607fbc5def/util/0.log" Jul 22 03:44:42.850493 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:42.850469 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef187sg6_90fe6071-e5e0-4797-8252-17607fbc5def/pull/0.log" Jul 22 03:44:43.356525 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:43.356500 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-l458v_0577bf79-9074-4ad0-a3f5-5f8bc286831e/manager/0.log" Jul 22 03:44:43.387403 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:43.387377 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-q4rcm_941e3af6-8144-4317-bec3-9e26bf4248f4/manager/0.log" Jul 22 03:44:43.420866 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:43.420838 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-dvrw9_32f52f23-526c-4e4b-a988-9d69c264dca9/kuadrant-console-plugin/0.log" Jul 22 03:44:43.469609 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:43.469570 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-sp78b_dc453083-f6e0-4809-9b8e-92034ab30c43/registry-server/0.log" Jul 22 03:44:43.724699 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:43.724672 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-8k8vz_18482f74-ab9c-4f92-b1f6-ea302e2419a2/limitador/0.log" Jul 22 03:44:43.795427 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:43.795392 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-9mb6v_36a3a681-1af5-4069-bb00-34d6ccdf9842/manager/0.log" Jul 22 03:44:45.811954 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:45.811884 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-7b7b7bc8bf-n42kf_1737a1fb-ebbd-4aab-8a32-595697912b60/monitoring-plugin/0.log" Jul 22 03:44:45.931366 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:45.931337 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ffjds_ba8f4a56-ce73-4f6a-9215-13a7f32f9342/node-exporter/0.log" Jul 22 03:44:45.955521 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:45.955496 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ffjds_ba8f4a56-ce73-4f6a-9215-13a7f32f9342/kube-rbac-proxy/0.log" Jul 22 03:44:45.977621 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:45.977601 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ffjds_ba8f4a56-ce73-4f6a-9215-13a7f32f9342/init-textfile/0.log" Jul 22 03:44:46.104155 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:46.104085 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-lsxjz_6d641c24-367d-4e6b-9402-e61e594f8811/kube-rbac-proxy-main/0.log" Jul 22 03:44:46.126509 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:46.126484 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-lsxjz_6d641c24-367d-4e6b-9402-e61e594f8811/kube-rbac-proxy-self/0.log" Jul 22 03:44:46.150748 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:46.150721 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-lsxjz_6d641c24-367d-4e6b-9402-e61e594f8811/openshift-state-metrics/0.log" Jul 22 03:44:46.374174 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:46.374149 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-8dht8_2f209b19-94fe-48ba-9d66-81886ca09b08/prometheus-operator/0.log" Jul 22 03:44:46.397228 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:46.397202 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-8dht8_2f209b19-94fe-48ba-9d66-81886ca09b08/kube-rbac-proxy/0.log" Jul 22 03:44:46.469238 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:46.469212 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7787d7f5b8-5p8w2_184bbaf0-8590-451b-9235-f9c9ab652e55/telemeter-client/0.log" Jul 22 03:44:46.493989 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:46.493960 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7787d7f5b8-5p8w2_184bbaf0-8590-451b-9235-f9c9ab652e55/reload/0.log" Jul 22 03:44:46.517366 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:46.517340 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7787d7f5b8-5p8w2_184bbaf0-8590-451b-9235-f9c9ab652e55/kube-rbac-proxy/0.log" Jul 22 03:44:46.553265 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:46.553237 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5c648fddd6-v4m7c_00475f40-7c5a-49d4-abd5-c4893b108673/thanos-query/0.log" Jul 22 03:44:46.576104 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:46.576078 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5c648fddd6-v4m7c_00475f40-7c5a-49d4-abd5-c4893b108673/kube-rbac-proxy-web/0.log" Jul 22 03:44:46.603519 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:46.603491 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5c648fddd6-v4m7c_00475f40-7c5a-49d4-abd5-c4893b108673/kube-rbac-proxy/0.log" Jul 22 03:44:46.624792 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:46.624720 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5c648fddd6-v4m7c_00475f40-7c5a-49d4-abd5-c4893b108673/prom-label-proxy/0.log" Jul 22 03:44:46.648052 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:46.648022 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5c648fddd6-v4m7c_00475f40-7c5a-49d4-abd5-c4893b108673/kube-rbac-proxy-rules/0.log" Jul 22 03:44:46.671525 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:46.671498 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5c648fddd6-v4m7c_00475f40-7c5a-49d4-abd5-c4893b108673/kube-rbac-proxy-metrics/0.log" Jul 22 03:44:48.124777 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:48.124746 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kwn6d/perf-node-gather-daemonset-5z6pr"] Jul 22 03:44:48.125252 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:48.125236 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b802703-a392-4db0-9186-3ed8413cbf8a" containerName="revoke-keys" Jul 22 03:44:48.125297 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:48.125253 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b802703-a392-4db0-9186-3ed8413cbf8a" containerName="revoke-keys" Jul 22 03:44:48.125332 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:48.125325 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b802703-a392-4db0-9186-3ed8413cbf8a" containerName="revoke-keys" Jul 22 03:44:48.129073 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:48.129046 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kwn6d/perf-node-gather-daemonset-5z6pr" Jul 22 03:44:48.133777 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:48.133739 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kwn6d\"/\"openshift-service-ca.crt\"" Jul 22 03:44:48.134127 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:48.134108 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kwn6d\"/\"default-dockercfg-qqsd7\"" Jul 22 03:44:48.134696 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:48.134676 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kwn6d\"/\"kube-root-ca.crt\"" Jul 22 03:44:48.140097 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:48.140076 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kwn6d/perf-node-gather-daemonset-5z6pr"] Jul 22 03:44:48.241441 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:48.241402 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b075bd4f-8fc2-470c-948b-b8ee7f88ee20-podres\") pod \"perf-node-gather-daemonset-5z6pr\" (UID: \"b075bd4f-8fc2-470c-948b-b8ee7f88ee20\") " pod="openshift-must-gather-kwn6d/perf-node-gather-daemonset-5z6pr" Jul 22 03:44:48.241441 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:48.241447 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdhws\" (UniqueName: \"kubernetes.io/projected/b075bd4f-8fc2-470c-948b-b8ee7f88ee20-kube-api-access-gdhws\") pod \"perf-node-gather-daemonset-5z6pr\" (UID: \"b075bd4f-8fc2-470c-948b-b8ee7f88ee20\") " pod="openshift-must-gather-kwn6d/perf-node-gather-daemonset-5z6pr" Jul 22 03:44:48.241711 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:48.241516 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b075bd4f-8fc2-470c-948b-b8ee7f88ee20-lib-modules\") pod \"perf-node-gather-daemonset-5z6pr\" (UID: \"b075bd4f-8fc2-470c-948b-b8ee7f88ee20\") " pod="openshift-must-gather-kwn6d/perf-node-gather-daemonset-5z6pr" Jul 22 03:44:48.241711 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:48.241582 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b075bd4f-8fc2-470c-948b-b8ee7f88ee20-proc\") pod \"perf-node-gather-daemonset-5z6pr\" (UID: \"b075bd4f-8fc2-470c-948b-b8ee7f88ee20\") " pod="openshift-must-gather-kwn6d/perf-node-gather-daemonset-5z6pr" Jul 22 03:44:48.241711 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:48.241678 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b075bd4f-8fc2-470c-948b-b8ee7f88ee20-sys\") pod \"perf-node-gather-daemonset-5z6pr\" (UID: \"b075bd4f-8fc2-470c-948b-b8ee7f88ee20\") " pod="openshift-must-gather-kwn6d/perf-node-gather-daemonset-5z6pr" Jul 22 03:44:48.342626 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:48.342566 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b075bd4f-8fc2-470c-948b-b8ee7f88ee20-sys\") pod \"perf-node-gather-daemonset-5z6pr\" (UID: \"b075bd4f-8fc2-470c-948b-b8ee7f88ee20\") " pod="openshift-must-gather-kwn6d/perf-node-gather-daemonset-5z6pr" Jul 22 03:44:48.342848 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:48.342640 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b075bd4f-8fc2-470c-948b-b8ee7f88ee20-podres\") pod \"perf-node-gather-daemonset-5z6pr\" (UID: \"b075bd4f-8fc2-470c-948b-b8ee7f88ee20\") " pod="openshift-must-gather-kwn6d/perf-node-gather-daemonset-5z6pr" Jul 22 03:44:48.342848 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:48.342673 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdhws\" (UniqueName: \"kubernetes.io/projected/b075bd4f-8fc2-470c-948b-b8ee7f88ee20-kube-api-access-gdhws\") pod \"perf-node-gather-daemonset-5z6pr\" (UID: \"b075bd4f-8fc2-470c-948b-b8ee7f88ee20\") " pod="openshift-must-gather-kwn6d/perf-node-gather-daemonset-5z6pr" Jul 22 03:44:48.342848 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:48.342705 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b075bd4f-8fc2-470c-948b-b8ee7f88ee20-sys\") pod \"perf-node-gather-daemonset-5z6pr\" (UID: \"b075bd4f-8fc2-470c-948b-b8ee7f88ee20\") " pod="openshift-must-gather-kwn6d/perf-node-gather-daemonset-5z6pr" Jul 22 03:44:48.342848 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:48.342717 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b075bd4f-8fc2-470c-948b-b8ee7f88ee20-lib-modules\") pod \"perf-node-gather-daemonset-5z6pr\" (UID: \"b075bd4f-8fc2-470c-948b-b8ee7f88ee20\") " pod="openshift-must-gather-kwn6d/perf-node-gather-daemonset-5z6pr" Jul 22 03:44:48.342848 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:48.342799 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b075bd4f-8fc2-470c-948b-b8ee7f88ee20-proc\") pod \"perf-node-gather-daemonset-5z6pr\" (UID: \"b075bd4f-8fc2-470c-948b-b8ee7f88ee20\") " pod="openshift-must-gather-kwn6d/perf-node-gather-daemonset-5z6pr" Jul 22 03:44:48.342848 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:48.342802 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b075bd4f-8fc2-470c-948b-b8ee7f88ee20-podres\") pod \"perf-node-gather-daemonset-5z6pr\" (UID: \"b075bd4f-8fc2-470c-948b-b8ee7f88ee20\") " pod="openshift-must-gather-kwn6d/perf-node-gather-daemonset-5z6pr" Jul 22 03:44:48.342848 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:48.342831 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b075bd4f-8fc2-470c-948b-b8ee7f88ee20-lib-modules\") pod \"perf-node-gather-daemonset-5z6pr\" (UID: \"b075bd4f-8fc2-470c-948b-b8ee7f88ee20\") " pod="openshift-must-gather-kwn6d/perf-node-gather-daemonset-5z6pr" Jul 22 03:44:48.342848 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:48.342846 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b075bd4f-8fc2-470c-948b-b8ee7f88ee20-proc\") pod \"perf-node-gather-daemonset-5z6pr\" (UID: \"b075bd4f-8fc2-470c-948b-b8ee7f88ee20\") " pod="openshift-must-gather-kwn6d/perf-node-gather-daemonset-5z6pr" Jul 22 03:44:48.353626 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:48.353595 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdhws\" (UniqueName: \"kubernetes.io/projected/b075bd4f-8fc2-470c-948b-b8ee7f88ee20-kube-api-access-gdhws\") pod \"perf-node-gather-daemonset-5z6pr\" (UID: \"b075bd4f-8fc2-470c-948b-b8ee7f88ee20\") " pod="openshift-must-gather-kwn6d/perf-node-gather-daemonset-5z6pr" Jul 22 03:44:48.446218 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:48.446120 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kwn6d/perf-node-gather-daemonset-5z6pr" Jul 22 03:44:48.608549 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:48.608522 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kwn6d/perf-node-gather-daemonset-5z6pr"] Jul 22 03:44:48.610075 ip-10-0-131-67 kubenswrapper[2582]: W0722 03:44:48.610047 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb075bd4f_8fc2_470c_948b_b8ee7f88ee20.slice/crio-2133b8b4e15a2a9c718b141f7de50f483dacff652df8a3865f18e96ed12eca7e WatchSource:0}: Error finding container 2133b8b4e15a2a9c718b141f7de50f483dacff652df8a3865f18e96ed12eca7e: Status 404 returned error can't find the container with id 2133b8b4e15a2a9c718b141f7de50f483dacff652df8a3865f18e96ed12eca7e Jul 22 03:44:49.147043 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:49.146986 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kwn6d/perf-node-gather-daemonset-5z6pr" event={"ID":"b075bd4f-8fc2-470c-948b-b8ee7f88ee20","Type":"ContainerStarted","Data":"ccfb20242684e1c6c6261ae98e8efac125d5f0169b9a6bfc8c17e833d386855d"} Jul 22 03:44:49.147451 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:49.147064 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kwn6d/perf-node-gather-daemonset-5z6pr" event={"ID":"b075bd4f-8fc2-470c-948b-b8ee7f88ee20","Type":"ContainerStarted","Data":"2133b8b4e15a2a9c718b141f7de50f483dacff652df8a3865f18e96ed12eca7e"} Jul 22 03:44:49.147451 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:49.147195 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-kwn6d/perf-node-gather-daemonset-5z6pr" Jul 22 03:44:49.164270 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:49.164221 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-kwn6d/perf-node-gather-daemonset-5z6pr" podStartSLOduration=1.164204984 podStartE2EDuration="1.164204984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 03:44:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:44:49.163687733 +0000 UTC m=+2799.895308580" watchObservedRunningTime="2026-07-22 03:44:49.164204984 +0000 UTC m=+2799.895825917" Jul 22 03:44:50.847378 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:50.847356 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-67vqv_2835f18b-f325-418a-bf5c-5600606720f5/dns/0.log" Jul 22 03:44:50.869821 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:50.869794 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-67vqv_2835f18b-f325-418a-bf5c-5600606720f5/kube-rbac-proxy/0.log" Jul 22 03:44:50.946809 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:50.946782 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5gftd_8c3db93b-85e3-4f29-a0cc-1b6b3d847a94/dns-node-resolver/0.log" Jul 22 03:44:51.523844 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:51.523817 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-67748fc486-qc6lw_4efe03aa-d6c0-4702-9f2d-517a51158c53/registry/0.log" Jul 22 03:44:51.592494 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:51.592468 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vssht_5ee42689-5bff-46fe-8047-adbe6cb34273/node-ca/0.log" Jul 22 03:44:52.650937 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:52.650904 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:44:52.831979 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:52.831951 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-684578bfd-2jd6x_d10d14d5-4415-431c-81c3-623c8050e7c8/kube-auth-proxy/0.log" Jul 22 03:44:52.899856 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:52.899827 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-lgdw9_9ca7bc57-768b-42fd-98df-ddddde7d3b08/istio-proxy/0.log" Jul 22 03:44:52.925328 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:52.925244 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-lgdw9_9ca7bc57-768b-42fd-98df-ddddde7d3b08/istio-proxy/1.log" Jul 22 03:44:52.958667 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:52.958641 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-szpl4_56ca64c4-0973-4ed3-acb1-2d2ebfc80814/payload-pre-processing/0.log" Jul 22 03:44:52.988651 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:52.988624 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-tf965_877a033d-62c3-46d6-a0d5-f23139602b32/payload-processing/0.log" Jul 22 03:44:53.684337 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:53.684289 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-czh64_a31582c5-7c16-40ce-818e-a45d99137e90/serve-healthcheck-canary/0.log" Jul 22 03:44:54.416724 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:54.416693 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zhrpq_c9f901fc-712c-4bc6-a5c8-2471837fcc18/kube-rbac-proxy/0.log" Jul 22 03:44:54.439435 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:54.439400 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zhrpq_c9f901fc-712c-4bc6-a5c8-2471837fcc18/exporter/0.log" Jul 22 03:44:54.465392 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:54.465355 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zhrpq_c9f901fc-712c-4bc6-a5c8-2471837fcc18/extractor/0.log" Jul 22 03:44:55.163548 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:55.163517 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-kwn6d/perf-node-gather-daemonset-5z6pr" Jul 22 03:44:55.587121 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:55.586984 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28hhd_e33233aa-5e20-483c-8bb0-0cd5a577da98/konnectivity-agent/0.log" limit=1024 Jul 22 03:44:56.801024 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:56.800995 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-lzsb5_6edeec1e-1428-4fe4-b99b-f4348dc39d74/manager/0.log" Jul 22 03:44:56.979241 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:56.979208 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-grbpl_c6393858-4cb7-4730-82ad-eac79608c284/manager/0.log" Jul 22 03:44:57.100907 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:57.100880 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6446b4cb87-mx6mt_ef264272-2474-4344-9d30-5ad28556cae8/manager/0.log" Jul 22 03:44:57.121318 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:57.121296 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-pnl96_20b8fa3f-a5f9-425b-8fd2-e52ee34adf8a/server/0.log" Jul 22 03:44:57.199071 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:57.199030 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-nbpjd_39934543-08e4-4456-b200-27ce181b98e8/manager/0.log" Jul 22 03:44:57.279050 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:57.279000 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-94fc48448-gsdv4_79701dcf-7b01-484f-8cc0-d6ce840733d1/manager/0.log" Jul 22 03:44:57.304150 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:57.304083 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-94fc48448-gsdv4_79701dcf-7b01-484f-8cc0-d6ce840733d1/copy-manifests/0.log" Jul 22 03:44:58.860380 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:58.860347 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-684578bfd-2jd6x_d10d14d5-4415-431c-81c3-623c8050e7c8/kube-auth-proxy/0.log" Jul 22 03:44:58.897952 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:58.897925 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-lgdw9_9ca7bc57-768b-42fd-98df-ddddde7d3b08/istio-proxy/0.log" Jul 22 03:44:58.926858 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:58.926814 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-lgdw9_9ca7bc57-768b-42fd-98df-ddddde7d3b08/istio-proxy/1.log" Jul 22 03:44:58.953032 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:58.952987 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-szpl4_56ca64c4-0973-4ed3-acb1-2d2ebfc80814/payload-pre-processing/0.log" Jul 22 03:44:58.976590 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:44:58.976563 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-tf965_877a033d-62c3-46d6-a0d5-f23139602b32/payload-processing/0.log" Jul 22 03:45:00.146230 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:45:00.146198 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29744865-85vxt"] Jul 22 03:45:00.150132 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:45:00.150113 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744865-85vxt" Jul 22 03:45:00.152647 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:45:00.152625 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cw8l4\"" Jul 22 03:45:00.156923 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:45:00.156896 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29744865-85vxt"] Jul 22 03:45:00.162070 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:45:00.162052 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744865-85vxt" Jul 22 03:45:00.304299 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:45:00.304276 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29744865-85vxt"] Jul 22 03:45:00.306214 ip-10-0-131-67 kubenswrapper[2582]: W0722 03:45:00.306183 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeba6139_16a9_4f9d_b207_fb71b0cddf5a.slice/crio-9ac60c4f068736705cba08891fe3e58c791846b071eedcc5befe9820f9dc4ea2 WatchSource:0}: Error finding container 9ac60c4f068736705cba08891fe3e58c791846b071eedcc5befe9820f9dc4ea2: Status 404 returned error can't find the container with id 9ac60c4f068736705cba08891fe3e58c791846b071eedcc5befe9820f9dc4ea2 Jul 22 03:45:00.489576 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:45:00.489493 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-684578bfd-2jd6x_d10d14d5-4415-431c-81c3-623c8050e7c8/kube-auth-proxy/0.log" Jul 22 03:45:00.536176 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:45:00.536145 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-lgdw9_9ca7bc57-768b-42fd-98df-ddddde7d3b08/istio-proxy/0.log" Jul 22 03:45:00.567278 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:45:00.567248 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-lgdw9_9ca7bc57-768b-42fd-98df-ddddde7d3b08/istio-proxy/1.log" Jul 22 03:45:00.588422 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:45:00.588385 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-szpl4_56ca64c4-0973-4ed3-acb1-2d2ebfc80814/payload-pre-processing/0.log" Jul 22 03:45:00.607008 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:45:00.606986 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-tf965_877a033d-62c3-46d6-a0d5-f23139602b32/payload-processing/0.log" Jul 22 03:45:01.225170 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:45:01.225130 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744865-85vxt" event={"ID":"beba6139-16a9-4f9d-b207-fb71b0cddf5a","Type":"ContainerStarted","Data":"a0e5a2421c787935f82957e4bd0c5b449253daf70ae15336653dc2854334514c"} Jul 22 03:45:01.225559 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:45:01.225203 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744865-85vxt" event={"ID":"beba6139-16a9-4f9d-b207-fb71b0cddf5a","Type":"ContainerStarted","Data":"9ac60c4f068736705cba08891fe3e58c791846b071eedcc5befe9820f9dc4ea2"} Jul 22 03:45:01.245893 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:45:01.245820 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744865-85vxt" podStartSLOduration=1.245798392 podStartE2EDuration="1.245798392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:45:01.241482649 +0000 UTC m=+2811.973103513" watchObservedRunningTime="2026-07-22 03:45:01.245798392 +0000 UTC m=+2811.977419242" Jul 22 03:45:02.233166 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:45:02.233133 2582 generic.go:356] "Generic (PLEG): container finished" podID="beba6139-16a9-4f9d-b207-fb71b0cddf5a" containerID="a0e5a2421c787935f82957e4bd0c5b449253daf70ae15336653dc2854334514c" exitCode=0 Jul 22 03:45:02.233572 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:45:02.233177 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744865-85vxt" event={"ID":"beba6139-16a9-4f9d-b207-fb71b0cddf5a","Type":"ContainerDied","Data":"a0e5a2421c787935f82957e4bd0c5b449253daf70ae15336653dc2854334514c"} Jul 22 03:45:03.378753 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:45:03.378726 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744865-85vxt" Jul 22 03:45:04.183962 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:45:04.183880 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-h5q8g_46945b18-460c-4da9-b5d3-eeeaf93b3a67/migrator/0.log" Jul 22 03:45:04.206425 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:45:04.206402 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-h5q8g_46945b18-460c-4da9-b5d3-eeeaf93b3a67/graceful-termination/0.log" Jul 22 03:45:04.246959 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:45:04.246932 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744865-85vxt" Jul 22 03:45:04.247184 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:45:04.246931 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744865-85vxt" event={"ID":"beba6139-16a9-4f9d-b207-fb71b0cddf5a","Type":"ContainerDied","Data":"9ac60c4f068736705cba08891fe3e58c791846b071eedcc5befe9820f9dc4ea2"} Jul 22 03:45:04.247184 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:45:04.247066 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ac60c4f068736705cba08891fe3e58c791846b071eedcc5befe9820f9dc4ea2" Jul 22 03:45:04.606330 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:45:04.606249 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-2nk8j_34cca172-635b-4437-a6e0-fa79ada1fd74/kube-storage-version-migrator-operator/1.log" Jul 22 03:45:04.607112 ip-10-0-131-67 kubenswrapper[2582]: I0722 03:45:04.607091 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-2nk8j_34cca172-635b-4437-a6e0-fa79ada1fd74/kube-storage-version-migrator-operator/0.log"