Jul 23 09:14:45.258301 ip-10-0-135-145 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 23 09:14:45.258316 ip-10-0-135-145 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 23 09:14:45.258326 ip-10-0-135-145 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 23 09:14:45.258656 ip-10-0-135-145 systemd[1]: Failed to start Kubernetes Kubelet. Jul 23 09:14:56.921640 ip-10-0-135-145 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 23 09:14:56.921655 ip-10-0-135-145 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot e6d873144d3141eba5f21c373e823f00 -- Jul 23 09:16:51.366551 ip-10-0-135-145 systemd[1]: Starting Kubernetes Kubelet... Jul 23 09:16:51.836876 ip-10-0-135-145 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 23 09:16:51.836876 ip-10-0-135-145 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 23 09:16:51.836876 ip-10-0-135-145 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 23 09:16:51.836876 ip-10-0-135-145 kubenswrapper[2576]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 23 09:16:51.836876 ip-10-0-135-145 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 23 09:16:51.836876 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.720146 2576 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 23 09:16:51.836876 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726034 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 09:16:51.836876 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726049 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 09:16:51.836876 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726057 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 09:16:51.836876 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726062 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 09:16:51.836876 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726068 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 09:16:51.836876 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726074 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 09:16:51.836876 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726078 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 09:16:51.786017 ip-10-0-135-145 systemd[1]: Started Kubernetes Kubelet. Jul 23 09:16:51.840706 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726083 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 09:16:51.840706 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726088 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 09:16:51.840706 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726093 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 09:16:51.840706 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726098 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 09:16:51.840706 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726102 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 09:16:51.840706 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726107 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 09:16:51.840706 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726110 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 09:16:51.840706 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726114 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 09:16:51.840706 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726118 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 09:16:51.840706 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726122 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 09:16:51.840706 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726126 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 09:16:51.840706 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726129 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 09:16:51.840706 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726133 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 09:16:51.840706 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726138 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 09:16:51.840706 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726142 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 09:16:51.840706 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726146 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 09:16:51.840706 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726150 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 09:16:51.840706 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726154 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 09:16:51.840706 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726158 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 09:16:51.840706 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726162 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 09:16:51.843006 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726166 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 09:16:51.843006 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726171 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 09:16:51.843006 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726175 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 09:16:51.843006 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726179 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 09:16:51.843006 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726183 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 09:16:51.843006 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726187 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 09:16:51.843006 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726191 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 09:16:51.843006 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726195 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 09:16:51.843006 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726199 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 09:16:51.843006 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726203 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 09:16:51.843006 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726207 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 09:16:51.843006 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726211 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 09:16:51.843006 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726214 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 09:16:51.843006 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726218 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 09:16:51.843006 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726222 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 09:16:51.843006 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726226 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 09:16:51.843006 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726230 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 09:16:51.843006 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726234 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 09:16:51.843006 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726256 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 09:16:51.843006 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726260 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 09:16:51.849707 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726264 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 09:16:51.849707 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726268 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 09:16:51.849707 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726272 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 09:16:51.849707 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726276 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 09:16:51.849707 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726281 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 09:16:51.849707 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726285 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 09:16:51.849707 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726289 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 09:16:51.849707 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726293 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 09:16:51.849707 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726297 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 09:16:51.849707 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726301 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 09:16:51.849707 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726305 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 09:16:51.849707 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726309 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 09:16:51.849707 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726314 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 09:16:51.849707 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726317 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 09:16:51.849707 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726321 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 09:16:51.849707 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726325 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 09:16:51.849707 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726329 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 09:16:51.849707 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726333 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 09:16:51.849707 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726336 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 09:16:51.849707 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726340 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 09:16:51.851130 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726343 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 09:16:51.851130 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726347 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 09:16:51.851130 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726351 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 09:16:51.851130 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726356 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 09:16:51.851130 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726360 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 09:16:51.851130 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726364 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 09:16:51.851130 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726368 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 09:16:51.851130 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726372 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 09:16:51.851130 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726376 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 23 09:16:51.851130 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726381 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 09:16:51.851130 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726385 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 09:16:51.851130 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726389 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 09:16:51.851130 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726393 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 09:16:51.851130 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726397 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 09:16:51.851130 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726401 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 09:16:51.851130 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726405 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 09:16:51.851130 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726411 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 09:16:51.851130 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726417 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 09:16:51.851130 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726421 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 09:16:51.851130 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726425 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 09:16:51.852127 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726429 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 09:16:51.852127 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726433 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 09:16:51.852127 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726436 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 09:16:51.852127 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726441 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 09:16:51.852127 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726445 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 09:16:51.852127 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726450 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 09:16:51.852127 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726454 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 09:16:51.852127 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726458 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 09:16:51.852127 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726462 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 09:16:51.852127 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726466 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 09:16:51.852127 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726471 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 09:16:51.852127 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.726475 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 09:16:51.852127 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727499 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 09:16:51.852127 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727509 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 09:16:51.852127 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727513 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 09:16:51.852127 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727518 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 09:16:51.852127 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727522 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 09:16:51.852127 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727527 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 09:16:51.852127 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727531 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 09:16:51.852127 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727535 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 23 09:16:51.853160 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727541 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 09:16:51.853160 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727545 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 09:16:51.853160 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727550 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 09:16:51.853160 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727554 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 09:16:51.853160 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727557 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 09:16:51.853160 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727561 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 09:16:51.853160 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727565 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 09:16:51.853160 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727568 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 09:16:51.853160 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727572 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 09:16:51.853160 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727576 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 09:16:51.853160 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727580 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 09:16:51.853160 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727583 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 09:16:51.853160 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727587 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 09:16:51.853160 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727591 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 09:16:51.853160 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727596 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 09:16:51.853160 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727599 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 09:16:51.853160 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727604 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 09:16:51.853160 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727607 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 09:16:51.853160 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727610 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 09:16:51.853160 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727614 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 09:16:51.854230 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727619 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 09:16:51.854230 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727624 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 09:16:51.854230 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727628 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 09:16:51.854230 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727632 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 09:16:51.854230 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727636 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 09:16:51.854230 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727640 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 09:16:51.854230 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727644 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 09:16:51.854230 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727649 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 09:16:51.854230 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727652 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 09:16:51.854230 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727656 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 09:16:51.854230 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727660 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 09:16:51.854230 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727663 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 09:16:51.854230 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727667 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 09:16:51.854230 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727671 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 09:16:51.854230 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727677 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 09:16:51.854230 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727683 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 09:16:51.854230 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727687 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 09:16:51.854230 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727691 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 09:16:51.854230 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727695 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 09:16:51.855252 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727699 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 09:16:51.855252 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727702 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 09:16:51.855252 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727706 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 09:16:51.855252 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727710 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 09:16:51.855252 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727716 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 09:16:51.855252 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727720 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 09:16:51.855252 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727724 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 09:16:51.855252 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727728 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 09:16:51.855252 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727732 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 09:16:51.855252 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727735 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 09:16:51.855252 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727739 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 09:16:51.855252 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727743 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 09:16:51.855252 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727747 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 09:16:51.855252 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727751 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 09:16:51.855252 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727755 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 09:16:51.855252 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727759 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 09:16:51.855252 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727763 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 09:16:51.855252 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727767 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 09:16:51.855252 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727771 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 09:16:51.855252 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727775 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 09:16:51.856300 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727781 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 09:16:51.856300 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727785 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 09:16:51.856300 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727790 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 09:16:51.856300 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727794 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 09:16:51.856300 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727798 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 09:16:51.856300 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727801 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 09:16:51.856300 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727805 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 09:16:51.856300 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727809 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 09:16:51.856300 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727812 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 09:16:51.856300 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727822 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 09:16:51.856300 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727826 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 09:16:51.856300 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727830 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 09:16:51.856300 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727834 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 09:16:51.856300 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727838 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 09:16:51.856300 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727841 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 09:16:51.856300 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727845 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 09:16:51.856300 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727849 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 09:16:51.856300 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727853 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 09:16:51.856300 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727856 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 09:16:51.857290 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727860 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 09:16:51.857290 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727866 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 09:16:51.857290 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727870 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 09:16:51.857290 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727874 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 09:16:51.857290 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727877 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 09:16:51.857290 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727881 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 09:16:51.857290 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727884 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 09:16:51.857290 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727888 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 09:16:51.857290 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727892 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 09:16:51.857290 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727896 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 09:16:51.857290 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727901 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 09:16:51.857290 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727905 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 09:16:51.857290 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.727909 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 09:16:51.857290 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728414 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 09:16:51.857290 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728421 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 09:16:51.857290 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728425 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 09:16:51.857290 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728431 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 09:16:51.857290 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728437 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 09:16:51.857290 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728441 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 09:16:51.858334 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728445 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 09:16:51.858334 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728449 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 09:16:51.858334 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728453 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 09:16:51.858334 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728458 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 09:16:51.858334 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728462 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 09:16:51.858334 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728466 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 09:16:51.858334 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728470 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 09:16:51.858334 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728474 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 09:16:51.858334 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728478 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 09:16:51.858334 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728483 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 09:16:51.858334 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728486 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 09:16:51.858334 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728490 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 09:16:51.858334 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728495 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 09:16:51.858334 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728500 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 09:16:51.858334 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728505 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 09:16:51.858334 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728509 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 09:16:51.858334 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728512 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 09:16:51.858334 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728516 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 09:16:51.858334 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728520 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 09:16:51.858334 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728523 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 09:16:51.859365 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728527 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 09:16:51.859365 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728531 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 09:16:51.859365 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728536 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 09:16:51.859365 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728540 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 09:16:51.859365 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728543 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 09:16:51.859365 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728547 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 09:16:51.859365 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728550 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 09:16:51.859365 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728554 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 09:16:51.859365 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728578 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 23 09:16:51.859365 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728585 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 09:16:51.859365 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728589 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 09:16:51.859365 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728593 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 09:16:51.859365 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728601 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 09:16:51.859365 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728605 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 09:16:51.859365 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728609 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 09:16:51.859365 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728614 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 09:16:51.859365 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728618 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 09:16:51.859365 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728623 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 09:16:51.859365 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728629 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 09:16:51.859365 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728633 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 09:16:51.860400 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728637 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 09:16:51.860400 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728641 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 09:16:51.860400 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728645 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 09:16:51.860400 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728649 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 09:16:51.860400 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728652 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 09:16:51.860400 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728657 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 09:16:51.860400 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728660 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 09:16:51.860400 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728664 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 09:16:51.860400 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728668 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 09:16:51.860400 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728671 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 09:16:51.860400 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728675 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 09:16:51.860400 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728679 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 09:16:51.860400 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728683 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 09:16:51.860400 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728687 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 09:16:51.860400 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728690 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 09:16:51.860400 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728694 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 09:16:51.860400 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728698 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 09:16:51.860400 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728702 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 09:16:51.860400 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728706 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 09:16:51.861416 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728710 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 09:16:51.861416 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728714 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 09:16:51.861416 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728717 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 09:16:51.861416 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728721 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 09:16:51.861416 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728724 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 09:16:51.861416 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728732 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 09:16:51.861416 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728735 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 09:16:51.861416 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728739 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 09:16:51.861416 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728744 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 09:16:51.861416 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728747 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 09:16:51.861416 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728752 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 09:16:51.861416 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728757 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 09:16:51.861416 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728761 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 09:16:51.861416 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728764 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 09:16:51.861416 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728768 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 09:16:51.861416 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728773 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 09:16:51.861416 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728776 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 09:16:51.861416 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728780 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 09:16:51.861416 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728784 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 09:16:51.861416 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728788 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 09:16:51.863814 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728791 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 09:16:51.863814 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728795 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 09:16:51.863814 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728798 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 09:16:51.863814 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728802 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 09:16:51.863814 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728806 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 09:16:51.863814 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728809 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 09:16:51.863814 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728813 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 09:16:51.863814 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728817 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 09:16:51.863814 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728821 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 09:16:51.863814 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728825 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 09:16:51.863814 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728829 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 09:16:51.863814 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728833 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 09:16:51.863814 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728836 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 09:16:51.863814 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.728839 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 09:16:51.863814 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.728917 2576 flags.go:64] FLAG: --address="0.0.0.0" Jul 23 09:16:51.863814 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.728926 2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 23 09:16:51.863814 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.728934 2576 flags.go:64] FLAG: --anonymous-auth="true" Jul 23 09:16:51.863814 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.728941 2576 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 23 09:16:51.863814 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.728946 2576 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 23 09:16:51.863814 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.728951 2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 23 09:16:51.863814 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.728957 2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 23 09:16:51.864843 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.728963 2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 23 09:16:51.864843 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.728968 2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 23 09:16:51.864843 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.728972 2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 23 09:16:51.864843 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.728978 2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 23 09:16:51.864843 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.728983 2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 23 09:16:51.864843 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.728987 2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 23 09:16:51.864843 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.728991 2576 flags.go:64] FLAG: --cgroup-root="" Jul 23 09:16:51.864843 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.728995 2576 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 23 09:16:51.864843 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729000 2576 flags.go:64] FLAG: --client-ca-file="" Jul 23 09:16:51.864843 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729004 2576 flags.go:64] FLAG: --cloud-provider="external" Jul 23 09:16:51.864843 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729008 2576 flags.go:64] FLAG: --cluster-dns="[]" Jul 23 09:16:51.864843 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729014 2576 flags.go:64] FLAG: --cluster-domain="" Jul 23 09:16:51.864843 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729019 2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 23 09:16:51.864843 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729023 2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 23 09:16:51.864843 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729028 2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 23 09:16:51.864843 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729033 2576 flags.go:64] FLAG: --container-log-max-files="5" Jul 23 09:16:51.864843 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729037 2576 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 23 09:16:51.864843 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729041 2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 23 09:16:51.864843 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729046 2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 23 09:16:51.864843 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729050 2576 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 23 09:16:51.864843 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729055 2576 flags.go:64] FLAG: --contention-profiling="false" Jul 23 09:16:51.864843 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729059 2576 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 23 09:16:51.864843 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729063 2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 23 09:16:51.864843 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729067 2576 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 23 09:16:51.867680 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729072 2576 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 23 09:16:51.867680 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729077 2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 23 09:16:51.867680 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729082 2576 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 23 09:16:51.867680 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729085 2576 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 23 09:16:51.867680 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729092 2576 flags.go:64] FLAG: --enable-load-reader="false" Jul 23 09:16:51.867680 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729096 2576 flags.go:64] FLAG: --enable-server="true" Jul 23 09:16:51.867680 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729100 2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 23 09:16:51.867680 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729106 2576 flags.go:64] FLAG: --event-burst="100" Jul 23 09:16:51.867680 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729110 2576 flags.go:64] FLAG: --event-qps="50" Jul 23 09:16:51.867680 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729114 2576 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 23 09:16:51.867680 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729118 2576 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 23 09:16:51.867680 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729123 2576 flags.go:64] FLAG: --eviction-hard="" Jul 23 09:16:51.867680 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729129 2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 23 09:16:51.867680 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729133 2576 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 23 09:16:51.867680 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729137 2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 23 09:16:51.867680 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729142 2576 flags.go:64] FLAG: --eviction-soft="" Jul 23 09:16:51.867680 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729146 2576 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 23 09:16:51.867680 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729150 2576 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 23 09:16:51.867680 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729154 2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 23 09:16:51.867680 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729159 2576 flags.go:64] FLAG: --experimental-mounter-path="" Jul 23 09:16:51.867680 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729166 2576 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 23 09:16:51.867680 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729169 2576 flags.go:64] FLAG: --fail-swap-on="true" Jul 23 09:16:51.867680 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729174 2576 flags.go:64] FLAG: --feature-gates="" Jul 23 09:16:51.867680 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729179 2576 flags.go:64] FLAG: --file-check-frequency="20s" Jul 23 09:16:51.867680 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729184 2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 23 09:16:51.869179 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729188 2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 23 09:16:51.869179 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729193 2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 23 09:16:51.869179 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729197 2576 flags.go:64] FLAG: --healthz-port="10248" Jul 23 09:16:51.869179 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729201 2576 flags.go:64] FLAG: --help="false" Jul 23 09:16:51.869179 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729206 2576 flags.go:64] FLAG: --hostname-override="ip-10-0-135-145.ec2.internal" Jul 23 09:16:51.869179 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729211 2576 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 23 09:16:51.869179 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729214 2576 flags.go:64] FLAG: --http-check-frequency="20s" Jul 23 09:16:51.869179 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729218 2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 23 09:16:51.869179 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729223 2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 23 09:16:51.869179 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729228 2576 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 23 09:16:51.869179 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729232 2576 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 23 09:16:51.869179 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729255 2576 flags.go:64] FLAG: --image-service-endpoint="" Jul 23 09:16:51.869179 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729260 2576 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 23 09:16:51.869179 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729265 2576 flags.go:64] FLAG: --kube-api-burst="100" Jul 23 09:16:51.869179 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729269 2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 23 09:16:51.869179 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729274 2576 flags.go:64] FLAG: --kube-api-qps="50" Jul 23 09:16:51.869179 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729278 2576 flags.go:64] FLAG: --kube-reserved="" Jul 23 09:16:51.869179 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729282 2576 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 23 09:16:51.869179 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729287 2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 23 09:16:51.869179 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729292 2576 flags.go:64] FLAG: --kubelet-cgroups="" Jul 23 09:16:51.869179 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729297 2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 23 09:16:51.869179 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729301 2576 flags.go:64] FLAG: --lock-file="" Jul 23 09:16:51.869179 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729305 2576 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 23 09:16:51.869179 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729309 2576 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 23 09:16:51.871159 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729313 2576 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 23 09:16:51.871159 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729322 2576 flags.go:64] FLAG: --log-json-split-stream="false" Jul 23 09:16:51.871159 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729326 2576 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 23 09:16:51.871159 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729332 2576 flags.go:64] FLAG: --log-text-split-stream="false" Jul 23 09:16:51.871159 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729337 2576 flags.go:64] FLAG: --logging-format="text" Jul 23 09:16:51.871159 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729341 2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 23 09:16:51.871159 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729346 2576 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 23 09:16:51.871159 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729350 2576 flags.go:64] FLAG: --manifest-url="" Jul 23 09:16:51.871159 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729353 2576 flags.go:64] FLAG: --manifest-url-header="" Jul 23 09:16:51.871159 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729359 2576 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 23 09:16:51.871159 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729363 2576 flags.go:64] FLAG: --max-open-files="1000000" Jul 23 09:16:51.871159 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729369 2576 flags.go:64] FLAG: --max-pods="110" Jul 23 09:16:51.871159 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729373 2576 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 23 09:16:51.871159 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729378 2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 23 09:16:51.871159 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729381 2576 flags.go:64] FLAG: --memory-manager-policy="None" Jul 23 09:16:51.871159 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729385 2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 23 09:16:51.871159 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729390 2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 23 09:16:51.871159 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729394 2576 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 23 09:16:51.871159 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729398 2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 23 09:16:51.871159 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729408 2576 flags.go:64] FLAG: --node-status-max-images="50" Jul 23 09:16:51.871159 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729413 2576 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 23 09:16:51.871159 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729417 2576 flags.go:64] FLAG: --oom-score-adj="-999" Jul 23 09:16:51.871159 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729421 2576 flags.go:64] FLAG: --pod-cidr="" Jul 23 09:16:51.872202 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729425 2576 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 23 09:16:51.872202 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729432 2576 flags.go:64] FLAG: --pod-manifest-path="" Jul 23 09:16:51.872202 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729437 2576 flags.go:64] FLAG: --pod-max-pids="-1" Jul 23 09:16:51.872202 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729441 2576 flags.go:64] FLAG: --pods-per-core="0" Jul 23 09:16:51.872202 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729446 2576 flags.go:64] FLAG: --port="10250" Jul 23 09:16:51.872202 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729450 2576 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 23 09:16:51.872202 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729454 2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-057f960c7ccc27aeb" Jul 23 09:16:51.872202 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729458 2576 flags.go:64] FLAG: --qos-reserved="" Jul 23 09:16:51.872202 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729463 2576 flags.go:64] FLAG: --read-only-port="10255" Jul 23 09:16:51.872202 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729467 2576 flags.go:64] FLAG: --register-node="true" Jul 23 09:16:51.872202 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729471 2576 flags.go:64] FLAG: --register-with-taints="" Jul 23 09:16:51.872202 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729475 2576 flags.go:64] FLAG: --registry-burst="10" Jul 23 09:16:51.872202 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729481 2576 flags.go:64] FLAG: --registry-qps="5" Jul 23 09:16:51.872202 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729485 2576 flags.go:64] FLAG: --reserved-cpus="" Jul 23 09:16:51.872202 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729490 2576 flags.go:64] FLAG: --reserved-memory="" Jul 23 09:16:51.872202 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729501 2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 23 09:16:51.872202 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729505 2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 23 09:16:51.872202 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729509 2576 flags.go:64] FLAG: --rotate-certificates="false" Jul 23 09:16:51.872202 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729513 2576 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 23 09:16:51.872202 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729518 2576 flags.go:64] FLAG: --runonce="false" Jul 23 09:16:51.872202 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729522 2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 23 09:16:51.872202 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729526 2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 23 09:16:51.872202 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729530 2576 flags.go:64] FLAG: --seccomp-default="false" Jul 23 09:16:51.872202 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729535 2576 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 23 09:16:51.872202 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729539 2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 23 09:16:51.873282 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729544 2576 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 23 09:16:51.873282 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729547 2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 23 09:16:51.873282 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729551 2576 flags.go:64] FLAG: --storage-driver-password="root" Jul 23 09:16:51.873282 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729555 2576 flags.go:64] FLAG: --storage-driver-secure="false" Jul 23 09:16:51.873282 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729561 2576 flags.go:64] FLAG: --storage-driver-table="stats" Jul 23 09:16:51.873282 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729565 2576 flags.go:64] FLAG: --storage-driver-user="root" Jul 23 09:16:51.873282 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729569 2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 23 09:16:51.873282 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729574 2576 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 23 09:16:51.873282 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729578 2576 flags.go:64] FLAG: --system-cgroups="" Jul 23 09:16:51.873282 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729582 2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 23 09:16:51.873282 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729589 2576 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 23 09:16:51.873282 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729593 2576 flags.go:64] FLAG: --tls-cert-file="" Jul 23 09:16:51.873282 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729597 2576 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 23 09:16:51.873282 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729605 2576 flags.go:64] FLAG: --tls-min-version="" Jul 23 09:16:51.873282 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729609 2576 flags.go:64] FLAG: --tls-private-key-file="" Jul 23 09:16:51.873282 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729614 2576 flags.go:64] FLAG: --topology-manager-policy="none" Jul 23 09:16:51.873282 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729618 2576 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 23 09:16:51.873282 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729622 2576 flags.go:64] FLAG: --topology-manager-scope="container" Jul 23 09:16:51.873282 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729626 2576 flags.go:64] FLAG: --v="2" Jul 23 09:16:51.873282 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729633 2576 flags.go:64] FLAG: --version="false" Jul 23 09:16:51.873282 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729639 2576 flags.go:64] FLAG: --vmodule="" Jul 23 09:16:51.873282 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729644 2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 23 09:16:51.873282 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.729648 2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 23 09:16:51.873282 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.729803 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 09:16:51.873282 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.729810 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 09:16:51.874357 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.729814 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 09:16:51.874357 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.729818 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 09:16:51.874357 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.729821 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 09:16:51.874357 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.729827 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 09:16:51.874357 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.729832 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 09:16:51.874357 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.729837 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 09:16:51.874357 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.729841 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 09:16:51.874357 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.729845 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 09:16:51.874357 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.729849 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 09:16:51.874357 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.729853 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 09:16:51.874357 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.729858 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 09:16:51.874357 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.729863 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 09:16:51.874357 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.729867 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 09:16:51.874357 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.729871 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 09:16:51.874357 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.729876 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 09:16:51.874357 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.729881 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 09:16:51.874357 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.729886 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 09:16:51.874357 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.729891 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 09:16:51.874357 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.729895 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 09:16:51.875366 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.729898 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 09:16:51.875366 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.729902 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 09:16:51.875366 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.729907 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 09:16:51.875366 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.729911 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 09:16:51.875366 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.729914 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 09:16:51.875366 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.729918 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 09:16:51.875366 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.729922 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 09:16:51.875366 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.729926 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 09:16:51.875366 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.729930 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 09:16:51.875366 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.729934 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 09:16:51.875366 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.729937 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 09:16:51.875366 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.729941 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 09:16:51.875366 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.729945 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 09:16:51.875366 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.729957 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 09:16:51.875366 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.729961 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 09:16:51.875366 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.729966 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 09:16:51.875366 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.729969 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 09:16:51.875366 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.729973 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 09:16:51.875366 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.729977 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 09:16:51.875366 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.729980 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 09:16:51.877739 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.729984 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 09:16:51.877739 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.729988 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 09:16:51.877739 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.729991 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 09:16:51.877739 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.729995 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 09:16:51.877739 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.730000 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 09:16:51.877739 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.730003 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 09:16:51.877739 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.730007 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 09:16:51.877739 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.730011 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 09:16:51.877739 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.730016 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 09:16:51.877739 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.730019 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 09:16:51.877739 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.730023 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 09:16:51.877739 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.730027 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 09:16:51.877739 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.730030 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 09:16:51.877739 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.730034 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 09:16:51.877739 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.730038 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 09:16:51.877739 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.730042 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 09:16:51.877739 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.730046 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 09:16:51.877739 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.730049 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 09:16:51.877739 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.730053 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 09:16:51.877739 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.730059 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 09:16:51.878776 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.730063 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 09:16:51.878776 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.730066 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 09:16:51.878776 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.730070 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 09:16:51.878776 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.730074 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 09:16:51.878776 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.730077 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 09:16:51.878776 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.730081 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 09:16:51.878776 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.730085 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 09:16:51.878776 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.730089 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 09:16:51.878776 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.730094 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 09:16:51.878776 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.730100 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 09:16:51.878776 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.730103 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 09:16:51.878776 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.730107 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 09:16:51.878776 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.730112 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 09:16:51.878776 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.730115 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 09:16:51.878776 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.730119 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 09:16:51.878776 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.730123 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 09:16:51.878776 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.730128 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 09:16:51.878776 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.730131 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 09:16:51.878776 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.730135 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 09:16:51.878776 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.730139 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 09:16:51.879805 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.730143 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 09:16:51.879805 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.730146 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 09:16:51.879805 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.730150 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 09:16:51.879805 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.730154 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 09:16:51.879805 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.730157 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 09:16:51.879805 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.730162 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 09:16:51.879805 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.730166 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 09:16:51.879805 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.730169 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 23 09:16:51.879805 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.730173 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 09:16:51.879805 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.730177 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 09:16:51.879805 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.730180 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 09:16:51.879805 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.730185 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 09:16:51.879805 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.730189 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 09:16:51.879805 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.730192 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 09:16:51.879805 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.730196 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 09:16:51.879805 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.730200 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 09:16:51.879805 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.730204 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 09:16:51.879805 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.730208 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 09:16:51.880847 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.730671 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 23 09:16:51.880847 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.738679 2576 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 23 09:16:51.880847 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.738694 2576 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 23 09:16:51.880847 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.738714 2576 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 23 09:16:51.880847 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.738723 2576 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 23 09:16:51.880847 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.738777 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 09:16:51.880847 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.738781 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 09:16:51.880847 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.738785 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 09:16:51.880847 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.738788 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 09:16:51.880847 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.738791 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 09:16:51.880847 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.738794 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 09:16:51.880847 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.738797 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 09:16:51.880847 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.738800 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 09:16:51.880847 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.738804 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 09:16:51.880847 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.738806 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 09:16:51.882716 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.738809 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 09:16:51.882716 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.738812 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 09:16:51.882716 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.738816 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 09:16:51.882716 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.738820 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 09:16:51.882716 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.738823 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 09:16:51.882716 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.738826 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 09:16:51.882716 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.738828 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 09:16:51.882716 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.738831 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 09:16:51.882716 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.738834 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 09:16:51.882716 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.738837 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 09:16:51.882716 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.738840 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 09:16:51.882716 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.738842 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 09:16:51.882716 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.738845 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 09:16:51.882716 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.738848 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 09:16:51.882716 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.738852 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 09:16:51.882716 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.738855 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 09:16:51.882716 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.738858 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 09:16:51.882716 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.738860 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 09:16:51.882716 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.738863 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 09:16:51.981219 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.738866 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 09:16:51.981219 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.738869 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 09:16:51.981219 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.738871 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 09:16:51.981219 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.738874 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 09:16:51.981219 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.738876 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 09:16:51.981219 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.738879 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 09:16:51.981219 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.738882 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 09:16:51.981219 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.738884 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 09:16:51.981219 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.738887 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 09:16:51.981219 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.738889 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 09:16:51.981219 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.738892 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 09:16:51.981219 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.738894 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 09:16:51.981219 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.738897 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 09:16:51.981219 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.738900 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 09:16:51.981219 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.738904 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 09:16:51.981219 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.738907 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 09:16:51.981219 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.738911 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 09:16:51.981219 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.738913 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 09:16:51.981219 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.738916 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 09:16:51.983072 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.738918 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 09:16:51.983072 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.738921 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 09:16:51.983072 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.738923 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 09:16:51.983072 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.738926 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 09:16:51.983072 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.738928 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 09:16:51.983072 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.738931 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 09:16:51.983072 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.738933 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 09:16:51.983072 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.738939 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 09:16:51.983072 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.738942 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 09:16:51.983072 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.738945 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 09:16:51.983072 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.738947 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 09:16:51.983072 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.738950 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 09:16:51.983072 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.738953 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 09:16:51.983072 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.738955 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 09:16:51.983072 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.738958 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 09:16:51.983072 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.738961 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 09:16:51.983072 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.738964 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 09:16:51.983072 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.738966 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 09:16:51.983072 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.738969 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 09:16:51.983072 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.738972 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 09:16:51.984104 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.738976 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 09:16:51.984104 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.738979 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 09:16:51.984104 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.738982 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 09:16:51.984104 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.738985 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 09:16:51.984104 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.738987 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 09:16:51.984104 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.738990 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 09:16:51.984104 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.738993 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 09:16:51.984104 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.738996 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 09:16:51.984104 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.738999 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 09:16:51.984104 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739001 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 09:16:51.984104 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739004 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 09:16:51.984104 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739007 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 09:16:51.984104 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739010 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 09:16:51.984104 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739013 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 09:16:51.984104 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739016 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 09:16:51.984104 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739018 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 09:16:51.984104 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739021 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 09:16:51.984104 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739023 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 09:16:51.984104 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739026 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 09:16:51.984104 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739030 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 09:16:51.986003 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739032 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 23 09:16:51.986003 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739035 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 09:16:51.986003 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739038 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 09:16:51.986003 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739040 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 09:16:51.986003 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739043 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 09:16:51.986003 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739045 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 09:16:51.986003 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739047 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 09:16:51.986003 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739050 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 09:16:51.986003 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739053 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 09:16:51.986003 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739055 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 09:16:51.986003 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739058 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 09:16:51.986003 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.739063 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 23 09:16:51.986003 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739191 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 09:16:51.986003 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739196 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 09:16:51.986003 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739199 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 09:16:51.986986 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739202 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 09:16:51.986986 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739205 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 09:16:51.986986 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739207 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 09:16:51.986986 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739210 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 09:16:51.986986 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739213 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 09:16:51.986986 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739215 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 09:16:51.986986 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739220 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 09:16:51.986986 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739223 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 09:16:51.986986 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739225 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 09:16:51.986986 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739228 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 09:16:51.986986 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739232 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 09:16:51.986986 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739252 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 09:16:51.986986 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739255 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 09:16:51.986986 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739258 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 09:16:51.986986 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739261 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 09:16:51.986986 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739264 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 09:16:51.986986 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739271 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 09:16:51.986986 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739274 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 09:16:51.986986 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739277 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 09:16:51.986986 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739279 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 09:16:51.988887 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739282 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 09:16:51.988887 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739284 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 09:16:51.988887 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739287 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 09:16:51.988887 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739290 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 09:16:51.988887 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739292 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 09:16:51.988887 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739295 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 09:16:51.988887 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739298 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 09:16:51.988887 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739300 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 09:16:51.988887 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739303 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 09:16:51.988887 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739307 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 09:16:51.988887 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739310 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 09:16:51.988887 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739313 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 09:16:51.988887 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739316 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 09:16:51.988887 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739319 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 09:16:51.988887 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739321 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 09:16:51.988887 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739323 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 09:16:51.988887 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739326 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 09:16:51.988887 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739329 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 09:16:51.988887 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739332 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 09:16:51.989885 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739335 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 09:16:51.989885 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739338 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 09:16:51.989885 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739340 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 09:16:51.989885 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739343 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 09:16:51.989885 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739346 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 09:16:51.989885 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739349 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 09:16:51.989885 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739351 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 09:16:51.989885 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739354 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 09:16:51.989885 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739356 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 09:16:51.989885 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739360 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 09:16:51.989885 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739362 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 09:16:51.989885 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739365 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 09:16:51.989885 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739367 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 09:16:51.989885 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739370 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 09:16:51.989885 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739372 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 09:16:51.989885 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739375 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 09:16:51.989885 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739377 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 09:16:51.989885 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739380 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 09:16:51.989885 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739383 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 09:16:51.989885 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739385 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 09:16:51.991715 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739388 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 09:16:51.991715 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739390 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 09:16:51.991715 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739393 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 09:16:51.991715 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739395 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 09:16:51.991715 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739398 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 09:16:51.991715 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739400 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 09:16:51.991715 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739403 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 09:16:51.991715 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739405 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 09:16:51.991715 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739408 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 09:16:51.991715 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739410 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 09:16:51.991715 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739413 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 09:16:51.991715 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739416 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 09:16:51.991715 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739419 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 09:16:51.991715 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739422 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 09:16:51.991715 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739424 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 09:16:51.991715 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739427 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 09:16:51.991715 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739430 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 09:16:51.991715 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739433 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 09:16:51.991715 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739436 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 09:16:51.991715 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739438 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 09:16:51.992707 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739441 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 09:16:51.992707 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739445 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 09:16:51.992707 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739449 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 09:16:51.992707 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739453 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 09:16:51.992707 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739456 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 23 09:16:51.992707 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739458 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 09:16:51.992707 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739461 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 09:16:51.992707 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739463 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 09:16:51.992707 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739466 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 09:16:51.992707 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739469 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 09:16:51.992707 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739471 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 09:16:51.992707 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739474 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 09:16:51.992707 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739476 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 09:16:51.992707 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739479 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 09:16:51.992707 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739481 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 09:16:51.992707 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739484 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 09:16:51.992707 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:51.739487 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 09:16:51.995445 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.739492 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 23 09:16:51.995445 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.739590 2576 server.go:962] "Client rotation is on, will bootstrap in background" Jul 23 09:16:51.995445 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.741510 2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 23 09:16:51.995445 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.742573 2576 server.go:1019] "Starting client certificate rotation" Jul 23 09:16:51.995445 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.742675 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 23 09:16:51.995445 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.742706 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 23 09:16:51.995445 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.745198 2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 23 09:16:51.995445 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.746798 2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 23 09:16:51.995445 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.754781 2576 log.go:25] "Validated CRI v1 runtime API" Jul 23 09:16:51.995445 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.758518 2576 log.go:25] "Validated CRI v1 image API" Jul 23 09:16:51.995445 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.759868 2576 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 23 09:16:51.995445 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.763338 2576 fs.go:135] Filesystem UUIDs: map[07e47ce9-79ce-487d-a3e8-5852bf866718:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 db0ae381-c23c-46a9-a29d-701ba8337a92:/dev/nvme0n1p3] Jul 23 09:16:51.996419 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.763360 2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 23 09:16:51.996889 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.768225 2576 manager.go:216] Machine: {Timestamp:2026-07-23 09:16:51.767703737 +0000 UTC m=+0.306982305 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3115330 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22679756360c78025f8ccd9929257c SystemUUID:ec226797-5636-0c78-025f-8ccd9929257c BootID:e6d87314-4d31-41eb-a5f2-1c373e823f00 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:65:5c:ca:da:2f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:65:5c:ca:da:2f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:d2:b7:02:a3:bc:b8 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 23 09:16:51.996889 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.768335 2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 23 09:16:51.996889 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.768402 2576 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 23 09:16:51.996889 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.769269 2576 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 23 09:16:51.996889 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.769292 2576 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-145.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 23 09:16:51.996889 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.769901 2576 topology_manager.go:138] "Creating topology manager with none policy" Jul 23 09:16:51.996889 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.769909 2576 container_manager_linux.go:309] "Creating device plugin manager" Jul 23 09:16:51.996889 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.769917 2576 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 23 09:16:51.997392 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.769930 2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 23 09:16:51.997392 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.770611 2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 23 09:16:51.997392 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.771095 2576 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 23 09:16:51.997392 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.774439 2576 state_mem.go:36] "Initialized new in-memory state store" Jul 23 09:16:51.997392 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.774554 2576 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 23 09:16:51.997392 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.774580 2576 kubelet.go:486] "Attempting to sync node with API server" Jul 23 09:16:51.997392 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.774590 2576 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 23 09:16:51.997392 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.774600 2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 23 09:16:51.997392 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.774609 2576 kubelet.go:398] "Adding apiserver pod source" Jul 23 09:16:51.997392 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.774618 2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 23 09:16:51.997392 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.775947 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 23 09:16:51.997392 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.775964 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 23 09:16:51.997392 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.778088 2576 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 23 09:16:51.997392 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.780057 2576 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 23 09:16:51.997392 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.780085 2576 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 23 09:16:51.997392 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.781032 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 23 09:16:51.997392 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.781057 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 23 09:16:51.997392 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.781071 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 23 09:16:51.997392 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.781085 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 23 09:16:51.997392 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.781101 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 23 09:16:51.997392 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.781111 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 23 09:16:51.997392 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.781127 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 23 09:16:51.997392 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.781141 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 23 09:16:51.997392 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.781153 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 23 09:16:51.997392 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.781160 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 23 09:16:51.997392 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.781169 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 23 09:16:51.997392 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.781174 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 23 09:16:51.997392 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.781827 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 23 09:16:51.997392 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.781836 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 23 09:16:51.997392 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.785228 2576 server.go:1268] "Started kubelet" Jul 23 09:16:51.997392 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.785318 2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 23 09:16:51.997392 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.785376 2576 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 23 09:16:51.997392 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.785895 2576 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 23 09:16:51.997392 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.786759 2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-145.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 23 09:16:51.997392 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:16:51.786755 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 23 09:16:51.997392 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:16:51.786755 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-145.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 23 09:16:51.997392 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.787815 2576 server.go:310] "Adding debug handlers to kubelet server" Jul 23 09:16:51.997392 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:16:51.789889 2576 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 23 09:16:51.997392 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.789971 2576 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 23 09:16:51.999879 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:16:51.791106 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-145.ec2.internal.18c4e0847132f004 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-145.ec2.internal,UID:ip-10-0-135-145.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-145.ec2.internal,},FirstTimestamp:2026-07-23 09:16:51.785027588 +0000 UTC m=+0.324306155,LastTimestamp:2026-07-23 09:16:51.785027588 +0000 UTC m=+0.324306155,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-145.ec2.internal,}" Jul 23 09:16:51.999879 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.795312 2576 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 23 09:16:51.999879 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.795328 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 23 09:16:51.999879 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:16:51.795408 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-145.ec2.internal\" not found" Jul 23 09:16:51.999879 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.795907 2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 23 09:16:51.999879 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.795954 2576 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 23 09:16:51.999879 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.795974 2576 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 23 09:16:51.999879 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.796163 2576 reconstruct.go:97] "Volume reconstruction finished" Jul 23 09:16:51.999879 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.796171 2576 reconciler.go:29] "Reconciler: start to sync state" Jul 23 09:16:51.999879 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.796831 2576 factory.go:55] Registering systemd factory Jul 23 09:16:51.999879 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.796849 2576 factory.go:223] Registration of the systemd container factory successfully Jul 23 09:16:51.999879 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.797061 2576 factory.go:153] Registering CRI-O factory Jul 23 09:16:51.999879 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.797075 2576 factory.go:223] Registration of the crio container factory successfully Jul 23 09:16:51.999879 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.797124 2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 23 09:16:51.999879 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.797145 2576 factory.go:103] Registering Raw factory Jul 23 09:16:51.999879 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.797157 2576 manager.go:1196] Started watching for new ooms in manager Jul 23 09:16:51.999879 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.797566 2576 manager.go:349] Starting recovery of all containers Jul 23 09:16:51.999879 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:16:51.799828 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 23 09:16:51.999879 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:16:51.799894 2576 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-145.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 23 09:16:51.999879 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.802720 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 23 09:16:51.999879 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.813400 2576 manager.go:354] Recovery completed Jul 23 09:16:51.999879 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.819549 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 09:16:51.999879 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.821464 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-145.ec2.internal" event="NodeHasSufficientMemory" Jul 23 09:16:51.999879 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.821493 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-145.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 09:16:51.999879 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.821504 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-145.ec2.internal" event="NodeHasSufficientPID" Jul 23 09:16:51.999879 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.821969 2576 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 23 09:16:51.999879 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.821980 2576 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 23 09:16:51.999879 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.821994 2576 state_mem.go:36] "Initialized new in-memory state store" Jul 23 09:16:51.999879 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.822048 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 09:16:51.999879 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.824955 2576 policy_none.go:49] "None policy: Start" Jul 23 09:16:51.999879 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.824969 2576 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 23 09:16:51.999879 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.824980 2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 23 09:16:51.999879 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.826729 2576 policy_none.go:47] "Start" Jul 23 09:16:51.999879 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.827418 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-145.ec2.internal" event="NodeHasSufficientMemory" Jul 23 09:16:51.999879 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.827449 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-145.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 09:16:52.001453 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.827465 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-145.ec2.internal" event="NodeHasSufficientPID" Jul 23 09:16:52.001453 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.832852 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cfwz7" Jul 23 09:16:52.001453 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:16:51.832974 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-145.ec2.internal.18c4e084735f1efa default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-145.ec2.internal,UID:ip-10-0-135-145.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-145.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-145.ec2.internal,},FirstTimestamp:2026-07-23 09:16:51.821477626 +0000 UTC m=+0.360756202,LastTimestamp:2026-07-23 09:16:51.821477626 +0000 UTC m=+0.360756202,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-145.ec2.internal,}" Jul 23 09:16:52.001453 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:51.843397 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cfwz7" Jul 23 09:16:52.001453 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:16:51.848823 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-145.ec2.internal.18c4e084735f7186 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-145.ec2.internal,UID:ip-10-0-135-145.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-145.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-145.ec2.internal,},FirstTimestamp:2026-07-23 09:16:51.821498758 +0000 UTC m=+0.360777326,LastTimestamp:2026-07-23 09:16:51.821498758 +0000 UTC m=+0.360777326,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-145.ec2.internal,}" Jul 23 09:16:52.001453 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:16:51.896176 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-145.ec2.internal\" not found" Jul 23 09:16:52.001453 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:16:51.996354 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-145.ec2.internal\" not found" Jul 23 09:16:52.006940 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:16:52.006077 2576 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-135-145.ec2.internal\" not found" node="ip-10-0-135-145.ec2.internal" Jul 23 09:16:52.109568 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:16:52.097278 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-145.ec2.internal\" not found" Jul 23 09:16:52.230857 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:16:52.197580 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-145.ec2.internal\" not found" Jul 23 09:16:52.365576 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:16:52.298173 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-145.ec2.internal\" not found" Jul 23 09:16:52.497700 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:16:52.398495 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-145.ec2.internal\" not found" Jul 23 09:16:52.500881 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:16:52.499218 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-145.ec2.internal\" not found" Jul 23 09:16:52.631541 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:16:52.599636 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-145.ec2.internal\" not found" Jul 23 09:16:52.749223 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:52.680977 2576 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 09:16:52.749223 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:16:52.700671 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-145.ec2.internal\" not found" Jul 23 09:16:52.749223 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:52.742897 2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 23 09:16:52.749223 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:52.743016 2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 23 09:16:52.801170 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:16:52.801130 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-145.ec2.internal\" not found" Jul 23 09:16:52.811103 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:52.811087 2576 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-135-145.ec2.internal" not found Jul 23 09:16:52.830253 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:52.830239 2576 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-135-145.ec2.internal" not found Jul 23 09:16:52.847777 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:52.847638 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-22 09:11:51 +0000 UTC" deadline="2028-05-03 18:34:12.16040165 +0000 UTC" Jul 23 09:16:52.847777 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:52.847775 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15609h17m19.312629722s" Jul 23 09:16:52.871975 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:52.871960 2576 manager.go:337] "Starting Device Plugin manager" Jul 23 09:16:52.872075 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:16:52.871998 2576 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 23 09:16:52.872075 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:52.872012 2576 server.go:87] "Starting device plugin registration server" Jul 23 09:16:52.872300 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:52.872280 2576 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 23 09:16:52.872389 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:52.872302 2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 23 09:16:52.872437 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:52.872398 2576 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 23 09:16:52.872501 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:52.872489 2576 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 23 09:16:52.872501 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:52.872499 2576 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 23 09:16:52.873013 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:16:52.872995 2576 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 23 09:16:52.873103 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:16:52.873047 2576 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-145.ec2.internal\" not found" Jul 23 09:16:52.895886 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:52.895841 2576 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-135-145.ec2.internal" not found Jul 23 09:16:52.977251 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:52.977229 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 09:16:52.979370 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:52.978185 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-145.ec2.internal" event="NodeHasSufficientMemory" Jul 23 09:16:52.979370 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:52.978214 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-145.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 09:16:52.979370 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:52.978227 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-145.ec2.internal" event="NodeHasSufficientPID" Jul 23 09:16:52.979370 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:52.978262 2576 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-145.ec2.internal" Jul 23 09:16:52.979370 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:52.979013 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 23 09:16:52.979370 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:52.979052 2576 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 23 09:16:52.979370 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:52.979071 2576 kubelet.go:2448] "Starting kubelet main sync loop" Jul 23 09:16:52.979370 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:16:52.979103 2576 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 23 09:16:52.981672 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:52.981655 2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 09:16:52.986934 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:52.986916 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 09:16:52.987665 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:52.987648 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-145.ec2.internal" event="NodeHasSufficientMemory" Jul 23 09:16:52.987766 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:52.987676 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-145.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 09:16:52.987766 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:52.987686 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-145.ec2.internal" event="NodeHasSufficientPID" Jul 23 09:16:52.989621 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:52.989606 2576 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-145.ec2.internal" Jul 23 09:16:52.989678 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:16:52.989627 2576 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-145.ec2.internal\": node \"ip-10-0-135-145.ec2.internal\" not found" Jul 23 09:16:53.006576 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.006554 2576 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 09:16:53.033365 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:16:53.033345 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-145.ec2.internal\" not found" Jul 23 09:16:53.079512 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.079486 2576 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-145.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-145.ec2.internal"] Jul 23 09:16:53.079576 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.079552 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 09:16:53.080356 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.080340 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-145.ec2.internal" event="NodeHasSufficientMemory" Jul 23 09:16:53.080423 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.080367 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-145.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 09:16:53.080423 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.080382 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-145.ec2.internal" event="NodeHasSufficientPID" Jul 23 09:16:53.082826 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.082812 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 09:16:53.082970 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.082954 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-145.ec2.internal" Jul 23 09:16:53.083005 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.082986 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 09:16:53.083542 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.083528 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-145.ec2.internal" event="NodeHasSufficientMemory" Jul 23 09:16:53.083609 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.083550 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-145.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 09:16:53.083609 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.083559 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-145.ec2.internal" event="NodeHasSufficientPID" Jul 23 09:16:53.083609 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.083584 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-145.ec2.internal" event="NodeHasSufficientMemory" Jul 23 09:16:53.083609 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.083602 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-145.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 09:16:53.083771 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.083615 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-145.ec2.internal" event="NodeHasSufficientPID" Jul 23 09:16:53.085881 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.085865 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-145.ec2.internal" Jul 23 09:16:53.085961 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.085892 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 09:16:53.086560 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.086542 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-145.ec2.internal" event="NodeHasSufficientMemory" Jul 23 09:16:53.086632 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.086572 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-145.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 09:16:53.086632 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.086585 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-145.ec2.internal" event="NodeHasSufficientPID" Jul 23 09:16:53.088224 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.088210 2576 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 09:16:53.096046 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.096028 2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-145.ec2.internal" Jul 23 09:16:53.102740 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.102725 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/00d27bbe0e5e208cc7ea070ef5013896-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-145.ec2.internal\" (UID: \"00d27bbe0e5e208cc7ea070ef5013896\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-145.ec2.internal" Jul 23 09:16:53.102808 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.102748 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/00d27bbe0e5e208cc7ea070ef5013896-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-145.ec2.internal\" (UID: \"00d27bbe0e5e208cc7ea070ef5013896\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-145.ec2.internal" Jul 23 09:16:53.102808 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.102766 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6e4c6298602a592029f63da426ffeb44-config\") pod \"kube-apiserver-proxy-ip-10-0-135-145.ec2.internal\" (UID: \"6e4c6298602a592029f63da426ffeb44\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-145.ec2.internal" Jul 23 09:16:53.109904 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.109884 2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-145.ec2.internal" Jul 23 09:16:53.114465 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.114451 2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-145.ec2.internal" Jul 23 09:16:53.130185 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.130166 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 23 09:16:53.131120 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.131108 2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-145.ec2.internal" Jul 23 09:16:53.136239 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.136223 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 23 09:16:53.156877 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.156825 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 23 09:16:53.156877 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.156843 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 23 09:16:53.156877 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:16:53.156858 2576 kubelet.go:3250] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-135-145.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-145.ec2.internal" Jul 23 09:16:53.157043 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:16:53.156889 2576 kubelet.go:3250] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-135-145.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-145.ec2.internal" Jul 23 09:16:53.203134 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.203110 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/00d27bbe0e5e208cc7ea070ef5013896-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-145.ec2.internal\" (UID: \"00d27bbe0e5e208cc7ea070ef5013896\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-145.ec2.internal" Jul 23 09:16:53.203225 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.203137 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/00d27bbe0e5e208cc7ea070ef5013896-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-145.ec2.internal\" (UID: \"00d27bbe0e5e208cc7ea070ef5013896\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-145.ec2.internal" Jul 23 09:16:53.203225 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.203154 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6e4c6298602a592029f63da426ffeb44-config\") pod \"kube-apiserver-proxy-ip-10-0-135-145.ec2.internal\" (UID: \"6e4c6298602a592029f63da426ffeb44\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-145.ec2.internal" Jul 23 09:16:53.203225 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.203192 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6e4c6298602a592029f63da426ffeb44-config\") pod \"kube-apiserver-proxy-ip-10-0-135-145.ec2.internal\" (UID: \"6e4c6298602a592029f63da426ffeb44\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-145.ec2.internal" Jul 23 09:16:53.203225 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.203214 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/00d27bbe0e5e208cc7ea070ef5013896-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-145.ec2.internal\" (UID: \"00d27bbe0e5e208cc7ea070ef5013896\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-145.ec2.internal" Jul 23 09:16:53.203405 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.203261 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/00d27bbe0e5e208cc7ea070ef5013896-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-145.ec2.internal\" (UID: \"00d27bbe0e5e208cc7ea070ef5013896\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-145.ec2.internal" Jul 23 09:16:53.440918 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.440829 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-145.ec2.internal" Jul 23 09:16:53.460235 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.460204 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-145.ec2.internal" Jul 23 09:16:53.777052 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.776971 2576 apiserver.go:52] "Watching apiserver" Jul 23 09:16:53.786914 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.786892 2576 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 23 09:16:53.788368 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.788346 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-l6fgq","kube-system/konnectivity-agent-czm9w","kube-system/kube-apiserver-proxy-ip-10-0-135-145.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drzcv","openshift-cluster-node-tuning-operator/tuned-8zd5l","openshift-image-registry/node-ca-9r4c5","openshift-multus/network-metrics-daemon-b7fmg","openshift-network-diagnostics/network-check-target-b24sn","openshift-dns/node-resolver-gf7qk","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-145.ec2.internal","openshift-multus/multus-4gw2w","openshift-multus/multus-additional-cni-plugins-xgtzd","openshift-network-operator/iptables-alerter-z2nbl","openshift-ovn-kubernetes/ovnkube-node-gvwsp"] Jul 23 09:16:53.790846 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.790829 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6fgq" Jul 23 09:16:53.790930 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:16:53.790909 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6fgq" podUID="33072424-e286-4580-a3b5-0ccf6b4cbcf8" Jul 23 09:16:53.793498 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.793478 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-czm9w" Jul 23 09:16:53.795703 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.795680 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drzcv" Jul 23 09:16:53.795866 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.795845 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 23 09:16:53.797535 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.797520 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 23 09:16:53.797750 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.797735 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-8pqxc\"" Jul 23 09:16:53.797843 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.797824 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8zd5l" Jul 23 09:16:53.798015 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.797999 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 23 09:16:53.798614 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.798598 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 23 09:16:53.799327 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.799302 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-pdb2j\"" Jul 23 09:16:53.799489 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.799475 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 23 09:16:53.799637 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.799619 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 23 09:16:53.800072 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.800059 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9r4c5" Jul 23 09:16:53.802339 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.802320 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b7fmg" Jul 23 09:16:53.802440 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:16:53.802419 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b7fmg" podUID="381423db-deea-45d8-8cff-408186294f3e" Jul 23 09:16:53.804279 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.804265 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 23 09:16:53.804524 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.804509 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 23 09:16:53.804565 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.804521 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b24sn" Jul 23 09:16:53.804619 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:16:53.804593 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b24sn" podUID="82e553d7-cc99-4f29-99ba-1816621870c6" Jul 23 09:16:53.804683 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.804673 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-p46nh\"" Jul 23 09:16:53.804860 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.804845 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vx2ms\"" Jul 23 09:16:53.804928 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.804884 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 23 09:16:53.805410 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.805394 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 23 09:16:53.805603 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.805590 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 23 09:16:53.805773 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.805761 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba46e6f4-7695-4578-89a5-c95d6a27890a-etc-kubernetes\") pod \"tuned-8zd5l\" (UID: \"ba46e6f4-7695-4578-89a5-c95d6a27890a\") " pod="openshift-cluster-node-tuning-operator/tuned-8zd5l" Jul 23 09:16:53.805811 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.805782 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ba46e6f4-7695-4578-89a5-c95d6a27890a-etc-sysconfig\") pod \"tuned-8zd5l\" (UID: \"ba46e6f4-7695-4578-89a5-c95d6a27890a\") " pod="openshift-cluster-node-tuning-operator/tuned-8zd5l" Jul 23 09:16:53.805811 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.805797 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ba46e6f4-7695-4578-89a5-c95d6a27890a-etc-tuned\") pod \"tuned-8zd5l\" (UID: \"ba46e6f4-7695-4578-89a5-c95d6a27890a\") " pod="openshift-cluster-node-tuning-operator/tuned-8zd5l" Jul 23 09:16:53.805871 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.805815 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ba46e6f4-7695-4578-89a5-c95d6a27890a-etc-sysctl-d\") pod \"tuned-8zd5l\" (UID: \"ba46e6f4-7695-4578-89a5-c95d6a27890a\") " pod="openshift-cluster-node-tuning-operator/tuned-8zd5l" Jul 23 09:16:53.805871 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.805841 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7bccf2b8-bb0e-4371-bd49-0c889d86206f-socket-dir\") pod \"aws-ebs-csi-driver-node-drzcv\" (UID: \"7bccf2b8-bb0e-4371-bd49-0c889d86206f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drzcv" Jul 23 09:16:53.805949 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.805876 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7bccf2b8-bb0e-4371-bd49-0c889d86206f-device-dir\") pod \"aws-ebs-csi-driver-node-drzcv\" (UID: \"7bccf2b8-bb0e-4371-bd49-0c889d86206f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drzcv" Jul 23 09:16:53.805949 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.805900 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ba46e6f4-7695-4578-89a5-c95d6a27890a-run\") pod \"tuned-8zd5l\" (UID: \"ba46e6f4-7695-4578-89a5-c95d6a27890a\") " pod="openshift-cluster-node-tuning-operator/tuned-8zd5l" Jul 23 09:16:53.805949 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.805920 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c4122a2d-66ca-4ec5-969b-14b89809eaa0-konnectivity-ca\") pod \"konnectivity-agent-czm9w\" (UID: \"c4122a2d-66ca-4ec5-969b-14b89809eaa0\") " pod="kube-system/konnectivity-agent-czm9w" Jul 23 09:16:53.805949 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.805938 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ba46e6f4-7695-4578-89a5-c95d6a27890a-lib-modules\") pod \"tuned-8zd5l\" (UID: \"ba46e6f4-7695-4578-89a5-c95d6a27890a\") " pod="openshift-cluster-node-tuning-operator/tuned-8zd5l" Jul 23 09:16:53.806085 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.805964 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77v7h\" (UniqueName: \"kubernetes.io/projected/ba46e6f4-7695-4578-89a5-c95d6a27890a-kube-api-access-77v7h\") pod \"tuned-8zd5l\" (UID: \"ba46e6f4-7695-4578-89a5-c95d6a27890a\") " pod="openshift-cluster-node-tuning-operator/tuned-8zd5l" Jul 23 09:16:53.806085 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.805986 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/33072424-e286-4580-a3b5-0ccf6b4cbcf8-kubelet-config\") pod \"global-pull-secret-syncer-l6fgq\" (UID: \"33072424-e286-4580-a3b5-0ccf6b4cbcf8\") " pod="kube-system/global-pull-secret-syncer-l6fgq" Jul 23 09:16:53.806085 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.806002 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/33072424-e286-4580-a3b5-0ccf6b4cbcf8-original-pull-secret\") pod \"global-pull-secret-syncer-l6fgq\" (UID: \"33072424-e286-4580-a3b5-0ccf6b4cbcf8\") " pod="kube-system/global-pull-secret-syncer-l6fgq" Jul 23 09:16:53.806085 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.806039 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c4122a2d-66ca-4ec5-969b-14b89809eaa0-agent-certs\") pod \"konnectivity-agent-czm9w\" (UID: \"c4122a2d-66ca-4ec5-969b-14b89809eaa0\") " pod="kube-system/konnectivity-agent-czm9w" Jul 23 09:16:53.806085 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.806064 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7bccf2b8-bb0e-4371-bd49-0c889d86206f-registration-dir\") pod \"aws-ebs-csi-driver-node-drzcv\" (UID: \"7bccf2b8-bb0e-4371-bd49-0c889d86206f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drzcv" Jul 23 09:16:53.806085 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.806082 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7bccf2b8-bb0e-4371-bd49-0c889d86206f-sys-fs\") pod \"aws-ebs-csi-driver-node-drzcv\" (UID: \"7bccf2b8-bb0e-4371-bd49-0c889d86206f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drzcv" Jul 23 09:16:53.806284 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.806098 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ba46e6f4-7695-4578-89a5-c95d6a27890a-etc-sysctl-conf\") pod \"tuned-8zd5l\" (UID: \"ba46e6f4-7695-4578-89a5-c95d6a27890a\") " pod="openshift-cluster-node-tuning-operator/tuned-8zd5l" Jul 23 09:16:53.806284 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.806115 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ba46e6f4-7695-4578-89a5-c95d6a27890a-etc-systemd\") pod \"tuned-8zd5l\" (UID: \"ba46e6f4-7695-4578-89a5-c95d6a27890a\") " pod="openshift-cluster-node-tuning-operator/tuned-8zd5l" Jul 23 09:16:53.806284 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.806130 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7bccf2b8-bb0e-4371-bd49-0c889d86206f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-drzcv\" (UID: \"7bccf2b8-bb0e-4371-bd49-0c889d86206f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drzcv" Jul 23 09:16:53.806284 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.806149 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba46e6f4-7695-4578-89a5-c95d6a27890a-var-lib-kubelet\") pod \"tuned-8zd5l\" (UID: \"ba46e6f4-7695-4578-89a5-c95d6a27890a\") " pod="openshift-cluster-node-tuning-operator/tuned-8zd5l" Jul 23 09:16:53.806284 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.806163 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ba46e6f4-7695-4578-89a5-c95d6a27890a-sys\") pod \"tuned-8zd5l\" (UID: \"ba46e6f4-7695-4578-89a5-c95d6a27890a\") " pod="openshift-cluster-node-tuning-operator/tuned-8zd5l" Jul 23 09:16:53.806284 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.806177 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ba46e6f4-7695-4578-89a5-c95d6a27890a-host\") pod \"tuned-8zd5l\" (UID: \"ba46e6f4-7695-4578-89a5-c95d6a27890a\") " pod="openshift-cluster-node-tuning-operator/tuned-8zd5l" Jul 23 09:16:53.806284 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.806197 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/33072424-e286-4580-a3b5-0ccf6b4cbcf8-dbus\") pod \"global-pull-secret-syncer-l6fgq\" (UID: \"33072424-e286-4580-a3b5-0ccf6b4cbcf8\") " pod="kube-system/global-pull-secret-syncer-l6fgq" Jul 23 09:16:53.806284 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.806214 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7bccf2b8-bb0e-4371-bd49-0c889d86206f-etc-selinux\") pod \"aws-ebs-csi-driver-node-drzcv\" (UID: \"7bccf2b8-bb0e-4371-bd49-0c889d86206f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drzcv" Jul 23 09:16:53.806284 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.806231 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm8m6\" (UniqueName: \"kubernetes.io/projected/7bccf2b8-bb0e-4371-bd49-0c889d86206f-kube-api-access-mm8m6\") pod \"aws-ebs-csi-driver-node-drzcv\" (UID: \"7bccf2b8-bb0e-4371-bd49-0c889d86206f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drzcv" Jul 23 09:16:53.806284 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.806246 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ba46e6f4-7695-4578-89a5-c95d6a27890a-etc-modprobe-d\") pod \"tuned-8zd5l\" (UID: \"ba46e6f4-7695-4578-89a5-c95d6a27890a\") " pod="openshift-cluster-node-tuning-operator/tuned-8zd5l" Jul 23 09:16:53.806617 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.806288 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ba46e6f4-7695-4578-89a5-c95d6a27890a-tmp\") pod \"tuned-8zd5l\" (UID: \"ba46e6f4-7695-4578-89a5-c95d6a27890a\") " pod="openshift-cluster-node-tuning-operator/tuned-8zd5l" Jul 23 09:16:53.806968 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.806953 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gf7qk" Jul 23 09:16:53.809406 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.809387 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4gw2w" Jul 23 09:16:53.810404 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.810384 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 23 09:16:53.810619 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.810604 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-r92zm\"" Jul 23 09:16:53.811304 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.811286 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 23 09:16:53.811860 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.811843 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 23 09:16:53.811957 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.811925 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xgtzd" Jul 23 09:16:53.812020 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.811996 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-974zc\"" Jul 23 09:16:53.812570 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.812554 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 23 09:16:53.812885 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.812870 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 23 09:16:53.813020 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.812998 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 23 09:16:53.814101 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.814086 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zjsk6\"" Jul 23 09:16:53.814329 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.814306 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-z2nbl" Jul 23 09:16:53.814440 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.814423 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 23 09:16:53.814753 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.814739 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 23 09:16:53.816879 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.816862 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gvwsp" Jul 23 09:16:53.817085 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.817068 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 23 09:16:53.817203 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.817185 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-xcctd\"" Jul 23 09:16:53.817294 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.817185 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 23 09:16:53.817652 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.817189 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 23 09:16:53.821796 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.821773 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 23 09:16:53.821917 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.821799 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 23 09:16:53.822007 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.821989 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-k8hnw\"" Jul 23 09:16:53.822225 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.822210 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 23 09:16:53.822297 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.822208 2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 23 09:16:53.822363 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.822349 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 23 09:16:53.824592 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.824565 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 23 09:16:53.824592 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.824586 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 23 09:16:53.854548 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.854528 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-pz2kp" Jul 23 09:16:53.866523 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.866502 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-pz2kp" Jul 23 09:16:53.897708 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.897692 2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 23 09:16:53.906749 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.906727 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7bccf2b8-bb0e-4371-bd49-0c889d86206f-etc-selinux\") pod \"aws-ebs-csi-driver-node-drzcv\" (UID: \"7bccf2b8-bb0e-4371-bd49-0c889d86206f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drzcv" Jul 23 09:16:53.906847 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.906763 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ba46e6f4-7695-4578-89a5-c95d6a27890a-etc-modprobe-d\") pod \"tuned-8zd5l\" (UID: \"ba46e6f4-7695-4578-89a5-c95d6a27890a\") " pod="openshift-cluster-node-tuning-operator/tuned-8zd5l" Jul 23 09:16:53.906847 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.906792 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5362d831-0050-47f8-acb1-fe06ddf24832-systemd-units\") pod \"ovnkube-node-gvwsp\" (UID: \"5362d831-0050-47f8-acb1-fe06ddf24832\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvwsp" Jul 23 09:16:53.906847 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.906822 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5362d831-0050-47f8-acb1-fe06ddf24832-etc-openvswitch\") pod \"ovnkube-node-gvwsp\" (UID: \"5362d831-0050-47f8-acb1-fe06ddf24832\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvwsp" Jul 23 09:16:53.906991 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.906864 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5362d831-0050-47f8-acb1-fe06ddf24832-host-run-ovn-kubernetes\") pod \"ovnkube-node-gvwsp\" (UID: \"5362d831-0050-47f8-acb1-fe06ddf24832\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvwsp" Jul 23 09:16:53.906991 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.906885 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ba46e6f4-7695-4578-89a5-c95d6a27890a-etc-modprobe-d\") pod \"tuned-8zd5l\" (UID: \"ba46e6f4-7695-4578-89a5-c95d6a27890a\") " pod="openshift-cluster-node-tuning-operator/tuned-8zd5l" Jul 23 09:16:53.906991 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.906886 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7bccf2b8-bb0e-4371-bd49-0c889d86206f-etc-selinux\") pod \"aws-ebs-csi-driver-node-drzcv\" (UID: \"7bccf2b8-bb0e-4371-bd49-0c889d86206f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drzcv" Jul 23 09:16:53.906991 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.906924 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5fdaa22f-7fb2-42dd-8c10-2e422fa19df2-cni-binary-copy\") pod \"multus-4gw2w\" (UID: \"5fdaa22f-7fb2-42dd-8c10-2e422fa19df2\") " pod="openshift-multus/multus-4gw2w" Jul 23 09:16:53.906991 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.906950 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/95ff7554-5e90-41c1-9ab5-93d6120788fa-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xgtzd\" (UID: \"95ff7554-5e90-41c1-9ab5-93d6120788fa\") " pod="openshift-multus/multus-additional-cni-plugins-xgtzd" Jul 23 09:16:53.906991 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.906969 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t5w8\" (UniqueName: \"kubernetes.io/projected/72bcfd4f-fce8-46b6-b3f1-52e2ba6306fa-kube-api-access-7t5w8\") pod \"node-resolver-gf7qk\" (UID: \"72bcfd4f-fce8-46b6-b3f1-52e2ba6306fa\") " pod="openshift-dns/node-resolver-gf7qk" Jul 23 09:16:53.906991 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.906991 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5362d831-0050-47f8-acb1-fe06ddf24832-ovn-node-metrics-cert\") pod \"ovnkube-node-gvwsp\" (UID: \"5362d831-0050-47f8-acb1-fe06ddf24832\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvwsp" Jul 23 09:16:53.907251 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.907020 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5fdaa22f-7fb2-42dd-8c10-2e422fa19df2-host-var-lib-kubelet\") pod \"multus-4gw2w\" (UID: \"5fdaa22f-7fb2-42dd-8c10-2e422fa19df2\") " pod="openshift-multus/multus-4gw2w" Jul 23 09:16:53.907251 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.907039 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5fdaa22f-7fb2-42dd-8c10-2e422fa19df2-hostroot\") pod \"multus-4gw2w\" (UID: \"5fdaa22f-7fb2-42dd-8c10-2e422fa19df2\") " pod="openshift-multus/multus-4gw2w" Jul 23 09:16:53.907251 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.907059 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ba46e6f4-7695-4578-89a5-c95d6a27890a-etc-sysconfig\") pod \"tuned-8zd5l\" (UID: \"ba46e6f4-7695-4578-89a5-c95d6a27890a\") " pod="openshift-cluster-node-tuning-operator/tuned-8zd5l" Jul 23 09:16:53.907251 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.907102 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ba46e6f4-7695-4578-89a5-c95d6a27890a-etc-sysconfig\") pod \"tuned-8zd5l\" (UID: \"ba46e6f4-7695-4578-89a5-c95d6a27890a\") " pod="openshift-cluster-node-tuning-operator/tuned-8zd5l" Jul 23 09:16:53.907251 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.907132 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7bccf2b8-bb0e-4371-bd49-0c889d86206f-socket-dir\") pod \"aws-ebs-csi-driver-node-drzcv\" (UID: \"7bccf2b8-bb0e-4371-bd49-0c889d86206f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drzcv" Jul 23 09:16:53.907251 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.907169 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7bccf2b8-bb0e-4371-bd49-0c889d86206f-device-dir\") pod \"aws-ebs-csi-driver-node-drzcv\" (UID: \"7bccf2b8-bb0e-4371-bd49-0c889d86206f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drzcv" Jul 23 09:16:53.907251 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.907193 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ba46e6f4-7695-4578-89a5-c95d6a27890a-run\") pod \"tuned-8zd5l\" (UID: \"ba46e6f4-7695-4578-89a5-c95d6a27890a\") " pod="openshift-cluster-node-tuning-operator/tuned-8zd5l" Jul 23 09:16:53.907251 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.907219 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5362d831-0050-47f8-acb1-fe06ddf24832-env-overrides\") pod \"ovnkube-node-gvwsp\" (UID: \"5362d831-0050-47f8-acb1-fe06ddf24832\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvwsp" Jul 23 09:16:53.907251 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.907247 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5fdaa22f-7fb2-42dd-8c10-2e422fa19df2-etc-kubernetes\") pod \"multus-4gw2w\" (UID: \"5fdaa22f-7fb2-42dd-8c10-2e422fa19df2\") " pod="openshift-multus/multus-4gw2w" Jul 23 09:16:53.907819 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.907285 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ba46e6f4-7695-4578-89a5-c95d6a27890a-lib-modules\") pod \"tuned-8zd5l\" (UID: \"ba46e6f4-7695-4578-89a5-c95d6a27890a\") " pod="openshift-cluster-node-tuning-operator/tuned-8zd5l" Jul 23 09:16:53.907819 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.907311 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7bccf2b8-bb0e-4371-bd49-0c889d86206f-socket-dir\") pod \"aws-ebs-csi-driver-node-drzcv\" (UID: \"7bccf2b8-bb0e-4371-bd49-0c889d86206f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drzcv" Jul 23 09:16:53.907819 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.907338 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7bccf2b8-bb0e-4371-bd49-0c889d86206f-device-dir\") pod \"aws-ebs-csi-driver-node-drzcv\" (UID: \"7bccf2b8-bb0e-4371-bd49-0c889d86206f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drzcv" Jul 23 09:16:53.907819 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.907348 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/33072424-e286-4580-a3b5-0ccf6b4cbcf8-kubelet-config\") pod \"global-pull-secret-syncer-l6fgq\" (UID: \"33072424-e286-4580-a3b5-0ccf6b4cbcf8\") " pod="kube-system/global-pull-secret-syncer-l6fgq" Jul 23 09:16:53.907819 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.907386 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ba46e6f4-7695-4578-89a5-c95d6a27890a-run\") pod \"tuned-8zd5l\" (UID: \"ba46e6f4-7695-4578-89a5-c95d6a27890a\") " pod="openshift-cluster-node-tuning-operator/tuned-8zd5l" Jul 23 09:16:53.907819 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.907392 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/33072424-e286-4580-a3b5-0ccf6b4cbcf8-original-pull-secret\") pod \"global-pull-secret-syncer-l6fgq\" (UID: \"33072424-e286-4580-a3b5-0ccf6b4cbcf8\") " pod="kube-system/global-pull-secret-syncer-l6fgq" Jul 23 09:16:53.907819 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.907401 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/33072424-e286-4580-a3b5-0ccf6b4cbcf8-kubelet-config\") pod \"global-pull-secret-syncer-l6fgq\" (UID: \"33072424-e286-4580-a3b5-0ccf6b4cbcf8\") " pod="kube-system/global-pull-secret-syncer-l6fgq" Jul 23 09:16:53.907819 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.907429 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ba46e6f4-7695-4578-89a5-c95d6a27890a-lib-modules\") pod \"tuned-8zd5l\" (UID: \"ba46e6f4-7695-4578-89a5-c95d6a27890a\") " pod="openshift-cluster-node-tuning-operator/tuned-8zd5l" Jul 23 09:16:53.907819 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.907431 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c4122a2d-66ca-4ec5-969b-14b89809eaa0-agent-certs\") pod \"konnectivity-agent-czm9w\" (UID: \"c4122a2d-66ca-4ec5-969b-14b89809eaa0\") " pod="kube-system/konnectivity-agent-czm9w" Jul 23 09:16:53.907819 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:16:53.907488 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 09:16:53.907819 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.907525 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmxq5\" (UniqueName: \"kubernetes.io/projected/381423db-deea-45d8-8cff-408186294f3e-kube-api-access-pmxq5\") pod \"network-metrics-daemon-b7fmg\" (UID: \"381423db-deea-45d8-8cff-408186294f3e\") " pod="openshift-multus/network-metrics-daemon-b7fmg" Jul 23 09:16:53.907819 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.907584 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/72bcfd4f-fce8-46b6-b3f1-52e2ba6306fa-hosts-file\") pod \"node-resolver-gf7qk\" (UID: \"72bcfd4f-fce8-46b6-b3f1-52e2ba6306fa\") " pod="openshift-dns/node-resolver-gf7qk" Jul 23 09:16:53.907819 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:16:53.907647 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33072424-e286-4580-a3b5-0ccf6b4cbcf8-original-pull-secret podName:33072424-e286-4580-a3b5-0ccf6b4cbcf8 nodeName:}" failed. No retries permitted until 2026-07-23 09:16:54.407600134 +0000 UTC m=+2.946878696 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/33072424-e286-4580-a3b5-0ccf6b4cbcf8-original-pull-secret") pod "global-pull-secret-syncer-l6fgq" (UID: "33072424-e286-4580-a3b5-0ccf6b4cbcf8") : object "kube-system"/"original-pull-secret" not registered Jul 23 09:16:53.907819 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.907675 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5fdaa22f-7fb2-42dd-8c10-2e422fa19df2-cnibin\") pod \"multus-4gw2w\" (UID: \"5fdaa22f-7fb2-42dd-8c10-2e422fa19df2\") " pod="openshift-multus/multus-4gw2w" Jul 23 09:16:53.908444 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.907709 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5fdaa22f-7fb2-42dd-8c10-2e422fa19df2-multus-conf-dir\") pod \"multus-4gw2w\" (UID: \"5fdaa22f-7fb2-42dd-8c10-2e422fa19df2\") " pod="openshift-multus/multus-4gw2w" Jul 23 09:16:53.908444 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.907782 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ba46e6f4-7695-4578-89a5-c95d6a27890a-etc-sysctl-conf\") pod \"tuned-8zd5l\" (UID: \"ba46e6f4-7695-4578-89a5-c95d6a27890a\") " pod="openshift-cluster-node-tuning-operator/tuned-8zd5l" Jul 23 09:16:53.908444 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.907810 2576 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 23 09:16:53.908444 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.907838 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ba46e6f4-7695-4578-89a5-c95d6a27890a-etc-systemd\") pod \"tuned-8zd5l\" (UID: \"ba46e6f4-7695-4578-89a5-c95d6a27890a\") " pod="openshift-cluster-node-tuning-operator/tuned-8zd5l" Jul 23 09:16:53.908444 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.907889 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ba46e6f4-7695-4578-89a5-c95d6a27890a-etc-systemd\") pod \"tuned-8zd5l\" (UID: \"ba46e6f4-7695-4578-89a5-c95d6a27890a\") " pod="openshift-cluster-node-tuning-operator/tuned-8zd5l" Jul 23 09:16:53.908444 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.907916 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/34d12a5c-a8ee-40d7-a365-14d8ef2d94ef-iptables-alerter-script\") pod \"iptables-alerter-z2nbl\" (UID: \"34d12a5c-a8ee-40d7-a365-14d8ef2d94ef\") " pod="openshift-network-operator/iptables-alerter-z2nbl" Jul 23 09:16:53.908444 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.907939 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ba46e6f4-7695-4578-89a5-c95d6a27890a-etc-sysctl-conf\") pod \"tuned-8zd5l\" (UID: \"ba46e6f4-7695-4578-89a5-c95d6a27890a\") " pod="openshift-cluster-node-tuning-operator/tuned-8zd5l" Jul 23 09:16:53.908444 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.907956 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5362d831-0050-47f8-acb1-fe06ddf24832-run-ovn\") pod \"ovnkube-node-gvwsp\" (UID: \"5362d831-0050-47f8-acb1-fe06ddf24832\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvwsp" Jul 23 09:16:53.908444 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.907984 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5362d831-0050-47f8-acb1-fe06ddf24832-host-cni-bin\") pod \"ovnkube-node-gvwsp\" (UID: \"5362d831-0050-47f8-acb1-fe06ddf24832\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvwsp" Jul 23 09:16:53.908444 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.908004 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5362d831-0050-47f8-acb1-fe06ddf24832-ovnkube-config\") pod \"ovnkube-node-gvwsp\" (UID: \"5362d831-0050-47f8-acb1-fe06ddf24832\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvwsp" Jul 23 09:16:53.908444 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.908019 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrwmp\" (UniqueName: \"kubernetes.io/projected/d676a11c-2214-4008-8801-efeb43be1ded-kube-api-access-qrwmp\") pod \"node-ca-9r4c5\" (UID: \"d676a11c-2214-4008-8801-efeb43be1ded\") " pod="openshift-image-registry/node-ca-9r4c5" Jul 23 09:16:53.908444 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.908041 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/95ff7554-5e90-41c1-9ab5-93d6120788fa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xgtzd\" (UID: \"95ff7554-5e90-41c1-9ab5-93d6120788fa\") " pod="openshift-multus/multus-additional-cni-plugins-xgtzd" Jul 23 09:16:53.908444 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.908062 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba46e6f4-7695-4578-89a5-c95d6a27890a-var-lib-kubelet\") pod \"tuned-8zd5l\" (UID: \"ba46e6f4-7695-4578-89a5-c95d6a27890a\") " pod="openshift-cluster-node-tuning-operator/tuned-8zd5l" Jul 23 09:16:53.908444 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.908096 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5362d831-0050-47f8-acb1-fe06ddf24832-host-slash\") pod \"ovnkube-node-gvwsp\" (UID: \"5362d831-0050-47f8-acb1-fe06ddf24832\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvwsp" Jul 23 09:16:53.909267 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.908119 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5362d831-0050-47f8-acb1-fe06ddf24832-run-openvswitch\") pod \"ovnkube-node-gvwsp\" (UID: \"5362d831-0050-47f8-acb1-fe06ddf24832\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvwsp" Jul 23 09:16:53.909267 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.908142 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5362d831-0050-47f8-acb1-fe06ddf24832-log-socket\") pod \"ovnkube-node-gvwsp\" (UID: \"5362d831-0050-47f8-acb1-fe06ddf24832\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvwsp" Jul 23 09:16:53.909267 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.908148 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba46e6f4-7695-4578-89a5-c95d6a27890a-var-lib-kubelet\") pod \"tuned-8zd5l\" (UID: \"ba46e6f4-7695-4578-89a5-c95d6a27890a\") " pod="openshift-cluster-node-tuning-operator/tuned-8zd5l" Jul 23 09:16:53.909267 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.908163 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwp5n\" (UniqueName: \"kubernetes.io/projected/5362d831-0050-47f8-acb1-fe06ddf24832-kube-api-access-vwp5n\") pod \"ovnkube-node-gvwsp\" (UID: \"5362d831-0050-47f8-acb1-fe06ddf24832\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvwsp" Jul 23 09:16:53.909267 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.908185 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5fdaa22f-7fb2-42dd-8c10-2e422fa19df2-system-cni-dir\") pod \"multus-4gw2w\" (UID: \"5fdaa22f-7fb2-42dd-8c10-2e422fa19df2\") " pod="openshift-multus/multus-4gw2w" Jul 23 09:16:53.909267 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.908205 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/95ff7554-5e90-41c1-9ab5-93d6120788fa-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xgtzd\" (UID: \"95ff7554-5e90-41c1-9ab5-93d6120788fa\") " pod="openshift-multus/multus-additional-cni-plugins-xgtzd" Jul 23 09:16:53.909267 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.908227 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ba46e6f4-7695-4578-89a5-c95d6a27890a-sys\") pod \"tuned-8zd5l\" (UID: \"ba46e6f4-7695-4578-89a5-c95d6a27890a\") " pod="openshift-cluster-node-tuning-operator/tuned-8zd5l" Jul 23 09:16:53.909267 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.908288 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ba46e6f4-7695-4578-89a5-c95d6a27890a-sys\") pod \"tuned-8zd5l\" (UID: \"ba46e6f4-7695-4578-89a5-c95d6a27890a\") " pod="openshift-cluster-node-tuning-operator/tuned-8zd5l" Jul 23 09:16:53.909267 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.908289 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ba46e6f4-7695-4578-89a5-c95d6a27890a-host\") pod \"tuned-8zd5l\" (UID: \"ba46e6f4-7695-4578-89a5-c95d6a27890a\") " pod="openshift-cluster-node-tuning-operator/tuned-8zd5l" Jul 23 09:16:53.909267 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.908320 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5fdaa22f-7fb2-42dd-8c10-2e422fa19df2-multus-cni-dir\") pod \"multus-4gw2w\" (UID: \"5fdaa22f-7fb2-42dd-8c10-2e422fa19df2\") " pod="openshift-multus/multus-4gw2w" Jul 23 09:16:53.909267 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.908336 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ba46e6f4-7695-4578-89a5-c95d6a27890a-host\") pod \"tuned-8zd5l\" (UID: \"ba46e6f4-7695-4578-89a5-c95d6a27890a\") " pod="openshift-cluster-node-tuning-operator/tuned-8zd5l" Jul 23 09:16:53.909267 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.908344 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm59d\" (UniqueName: \"kubernetes.io/projected/5fdaa22f-7fb2-42dd-8c10-2e422fa19df2-kube-api-access-nm59d\") pod \"multus-4gw2w\" (UID: \"5fdaa22f-7fb2-42dd-8c10-2e422fa19df2\") " pod="openshift-multus/multus-4gw2w" Jul 23 09:16:53.909267 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.908386 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm8m6\" (UniqueName: \"kubernetes.io/projected/7bccf2b8-bb0e-4371-bd49-0c889d86206f-kube-api-access-mm8m6\") pod \"aws-ebs-csi-driver-node-drzcv\" (UID: \"7bccf2b8-bb0e-4371-bd49-0c889d86206f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drzcv" Jul 23 09:16:53.909267 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.908407 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ba46e6f4-7695-4578-89a5-c95d6a27890a-tmp\") pod \"tuned-8zd5l\" (UID: \"ba46e6f4-7695-4578-89a5-c95d6a27890a\") " pod="openshift-cluster-node-tuning-operator/tuned-8zd5l" Jul 23 09:16:53.910027 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.908431 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/381423db-deea-45d8-8cff-408186294f3e-metrics-certs\") pod \"network-metrics-daemon-b7fmg\" (UID: \"381423db-deea-45d8-8cff-408186294f3e\") " pod="openshift-multus/network-metrics-daemon-b7fmg" Jul 23 09:16:53.910027 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.908455 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5362d831-0050-47f8-acb1-fe06ddf24832-host-run-netns\") pod \"ovnkube-node-gvwsp\" (UID: \"5362d831-0050-47f8-acb1-fe06ddf24832\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvwsp" Jul 23 09:16:53.910027 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.908475 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d676a11c-2214-4008-8801-efeb43be1ded-serviceca\") pod \"node-ca-9r4c5\" (UID: \"d676a11c-2214-4008-8801-efeb43be1ded\") " pod="openshift-image-registry/node-ca-9r4c5" Jul 23 09:16:53.910027 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.908498 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5fdaa22f-7fb2-42dd-8c10-2e422fa19df2-host-run-multus-certs\") pod \"multus-4gw2w\" (UID: \"5fdaa22f-7fb2-42dd-8c10-2e422fa19df2\") " pod="openshift-multus/multus-4gw2w" Jul 23 09:16:53.910027 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.908522 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba46e6f4-7695-4578-89a5-c95d6a27890a-etc-kubernetes\") pod \"tuned-8zd5l\" (UID: \"ba46e6f4-7695-4578-89a5-c95d6a27890a\") " pod="openshift-cluster-node-tuning-operator/tuned-8zd5l" Jul 23 09:16:53.910027 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.908545 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99lgf\" (UniqueName: \"kubernetes.io/projected/82e553d7-cc99-4f29-99ba-1816621870c6-kube-api-access-99lgf\") pod \"network-check-target-b24sn\" (UID: \"82e553d7-cc99-4f29-99ba-1816621870c6\") " pod="openshift-network-diagnostics/network-check-target-b24sn" Jul 23 09:16:53.910027 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.908568 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xptqf\" (UniqueName: \"kubernetes.io/projected/34d12a5c-a8ee-40d7-a365-14d8ef2d94ef-kube-api-access-xptqf\") pod \"iptables-alerter-z2nbl\" (UID: \"34d12a5c-a8ee-40d7-a365-14d8ef2d94ef\") " pod="openshift-network-operator/iptables-alerter-z2nbl" Jul 23 09:16:53.910027 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.908589 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5362d831-0050-47f8-acb1-fe06ddf24832-node-log\") pod \"ovnkube-node-gvwsp\" (UID: \"5362d831-0050-47f8-acb1-fe06ddf24832\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvwsp" Jul 23 09:16:53.910027 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.908609 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5362d831-0050-47f8-acb1-fe06ddf24832-host-cni-netd\") pod \"ovnkube-node-gvwsp\" (UID: \"5362d831-0050-47f8-acb1-fe06ddf24832\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvwsp" Jul 23 09:16:53.910027 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.908638 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5362d831-0050-47f8-acb1-fe06ddf24832-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gvwsp\" (UID: \"5362d831-0050-47f8-acb1-fe06ddf24832\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvwsp" Jul 23 09:16:53.910027 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.908661 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5362d831-0050-47f8-acb1-fe06ddf24832-ovnkube-script-lib\") pod \"ovnkube-node-gvwsp\" (UID: \"5362d831-0050-47f8-acb1-fe06ddf24832\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvwsp" Jul 23 09:16:53.910027 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.908683 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ba46e6f4-7695-4578-89a5-c95d6a27890a-etc-tuned\") pod \"tuned-8zd5l\" (UID: \"ba46e6f4-7695-4578-89a5-c95d6a27890a\") " pod="openshift-cluster-node-tuning-operator/tuned-8zd5l" Jul 23 09:16:53.910027 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.908702 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72bcfd4f-fce8-46b6-b3f1-52e2ba6306fa-tmp-dir\") pod \"node-resolver-gf7qk\" (UID: \"72bcfd4f-fce8-46b6-b3f1-52e2ba6306fa\") " pod="openshift-dns/node-resolver-gf7qk" Jul 23 09:16:53.910027 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.908728 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/34d12a5c-a8ee-40d7-a365-14d8ef2d94ef-host-slash\") pod \"iptables-alerter-z2nbl\" (UID: \"34d12a5c-a8ee-40d7-a365-14d8ef2d94ef\") " pod="openshift-network-operator/iptables-alerter-z2nbl" Jul 23 09:16:53.910736 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.908768 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5362d831-0050-47f8-acb1-fe06ddf24832-run-systemd\") pod \"ovnkube-node-gvwsp\" (UID: \"5362d831-0050-47f8-acb1-fe06ddf24832\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvwsp" Jul 23 09:16:53.910736 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.908791 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5362d831-0050-47f8-acb1-fe06ddf24832-var-lib-openvswitch\") pod \"ovnkube-node-gvwsp\" (UID: \"5362d831-0050-47f8-acb1-fe06ddf24832\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvwsp" Jul 23 09:16:53.910736 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.908812 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5fdaa22f-7fb2-42dd-8c10-2e422fa19df2-os-release\") pod \"multus-4gw2w\" (UID: \"5fdaa22f-7fb2-42dd-8c10-2e422fa19df2\") " pod="openshift-multus/multus-4gw2w" Jul 23 09:16:53.910736 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.908833 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5fdaa22f-7fb2-42dd-8c10-2e422fa19df2-host-var-lib-cni-bin\") pod \"multus-4gw2w\" (UID: \"5fdaa22f-7fb2-42dd-8c10-2e422fa19df2\") " pod="openshift-multus/multus-4gw2w" Jul 23 09:16:53.910736 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.908858 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5fdaa22f-7fb2-42dd-8c10-2e422fa19df2-host-var-lib-cni-multus\") pod \"multus-4gw2w\" (UID: \"5fdaa22f-7fb2-42dd-8c10-2e422fa19df2\") " pod="openshift-multus/multus-4gw2w" Jul 23 09:16:53.910736 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.908927 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ba46e6f4-7695-4578-89a5-c95d6a27890a-etc-sysctl-d\") pod \"tuned-8zd5l\" (UID: \"ba46e6f4-7695-4578-89a5-c95d6a27890a\") " pod="openshift-cluster-node-tuning-operator/tuned-8zd5l" Jul 23 09:16:53.910736 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.908962 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c4122a2d-66ca-4ec5-969b-14b89809eaa0-konnectivity-ca\") pod \"konnectivity-agent-czm9w\" (UID: \"c4122a2d-66ca-4ec5-969b-14b89809eaa0\") " pod="kube-system/konnectivity-agent-czm9w" Jul 23 09:16:53.910736 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.908991 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5fdaa22f-7fb2-42dd-8c10-2e422fa19df2-host-run-netns\") pod \"multus-4gw2w\" (UID: \"5fdaa22f-7fb2-42dd-8c10-2e422fa19df2\") " pod="openshift-multus/multus-4gw2w" Jul 23 09:16:53.910736 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.909017 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5fdaa22f-7fb2-42dd-8c10-2e422fa19df2-multus-daemon-config\") pod \"multus-4gw2w\" (UID: \"5fdaa22f-7fb2-42dd-8c10-2e422fa19df2\") " pod="openshift-multus/multus-4gw2w" Jul 23 09:16:53.910736 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.909043 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/95ff7554-5e90-41c1-9ab5-93d6120788fa-cni-binary-copy\") pod \"multus-additional-cni-plugins-xgtzd\" (UID: \"95ff7554-5e90-41c1-9ab5-93d6120788fa\") " pod="openshift-multus/multus-additional-cni-plugins-xgtzd" Jul 23 09:16:53.910736 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.909075 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77v7h\" (UniqueName: \"kubernetes.io/projected/ba46e6f4-7695-4578-89a5-c95d6a27890a-kube-api-access-77v7h\") pod \"tuned-8zd5l\" (UID: \"ba46e6f4-7695-4578-89a5-c95d6a27890a\") " pod="openshift-cluster-node-tuning-operator/tuned-8zd5l" Jul 23 09:16:53.910736 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.909101 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/95ff7554-5e90-41c1-9ab5-93d6120788fa-cnibin\") pod \"multus-additional-cni-plugins-xgtzd\" (UID: \"95ff7554-5e90-41c1-9ab5-93d6120788fa\") " pod="openshift-multus/multus-additional-cni-plugins-xgtzd" Jul 23 09:16:53.910736 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.909125 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/95ff7554-5e90-41c1-9ab5-93d6120788fa-os-release\") pod \"multus-additional-cni-plugins-xgtzd\" (UID: \"95ff7554-5e90-41c1-9ab5-93d6120788fa\") " pod="openshift-multus/multus-additional-cni-plugins-xgtzd" Jul 23 09:16:53.910736 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.909162 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2qkt\" (UniqueName: \"kubernetes.io/projected/95ff7554-5e90-41c1-9ab5-93d6120788fa-kube-api-access-w2qkt\") pod \"multus-additional-cni-plugins-xgtzd\" (UID: \"95ff7554-5e90-41c1-9ab5-93d6120788fa\") " pod="openshift-multus/multus-additional-cni-plugins-xgtzd" Jul 23 09:16:53.911468 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.909220 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba46e6f4-7695-4578-89a5-c95d6a27890a-etc-kubernetes\") pod \"tuned-8zd5l\" (UID: \"ba46e6f4-7695-4578-89a5-c95d6a27890a\") " pod="openshift-cluster-node-tuning-operator/tuned-8zd5l" Jul 23 09:16:53.911468 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.909262 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7bccf2b8-bb0e-4371-bd49-0c889d86206f-registration-dir\") pod \"aws-ebs-csi-driver-node-drzcv\" (UID: \"7bccf2b8-bb0e-4371-bd49-0c889d86206f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drzcv" Jul 23 09:16:53.911468 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.909286 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7bccf2b8-bb0e-4371-bd49-0c889d86206f-sys-fs\") pod \"aws-ebs-csi-driver-node-drzcv\" (UID: \"7bccf2b8-bb0e-4371-bd49-0c889d86206f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drzcv" Jul 23 09:16:53.911468 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.909312 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5fdaa22f-7fb2-42dd-8c10-2e422fa19df2-multus-socket-dir-parent\") pod \"multus-4gw2w\" (UID: \"5fdaa22f-7fb2-42dd-8c10-2e422fa19df2\") " pod="openshift-multus/multus-4gw2w" Jul 23 09:16:53.911468 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.909382 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7bccf2b8-bb0e-4371-bd49-0c889d86206f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-drzcv\" (UID: \"7bccf2b8-bb0e-4371-bd49-0c889d86206f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drzcv" Jul 23 09:16:53.911468 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.909410 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d676a11c-2214-4008-8801-efeb43be1ded-host\") pod \"node-ca-9r4c5\" (UID: \"d676a11c-2214-4008-8801-efeb43be1ded\") " pod="openshift-image-registry/node-ca-9r4c5" Jul 23 09:16:53.911468 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.909434 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5fdaa22f-7fb2-42dd-8c10-2e422fa19df2-host-run-k8s-cni-cncf-io\") pod \"multus-4gw2w\" (UID: \"5fdaa22f-7fb2-42dd-8c10-2e422fa19df2\") " pod="openshift-multus/multus-4gw2w" Jul 23 09:16:53.911468 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.909463 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/33072424-e286-4580-a3b5-0ccf6b4cbcf8-dbus\") pod \"global-pull-secret-syncer-l6fgq\" (UID: \"33072424-e286-4580-a3b5-0ccf6b4cbcf8\") " pod="kube-system/global-pull-secret-syncer-l6fgq" Jul 23 09:16:53.911468 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.909491 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5362d831-0050-47f8-acb1-fe06ddf24832-host-kubelet\") pod \"ovnkube-node-gvwsp\" (UID: \"5362d831-0050-47f8-acb1-fe06ddf24832\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvwsp" Jul 23 09:16:53.911468 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.909519 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/95ff7554-5e90-41c1-9ab5-93d6120788fa-system-cni-dir\") pod \"multus-additional-cni-plugins-xgtzd\" (UID: \"95ff7554-5e90-41c1-9ab5-93d6120788fa\") " pod="openshift-multus/multus-additional-cni-plugins-xgtzd" Jul 23 09:16:53.911468 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.909618 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7bccf2b8-bb0e-4371-bd49-0c889d86206f-sys-fs\") pod \"aws-ebs-csi-driver-node-drzcv\" (UID: \"7bccf2b8-bb0e-4371-bd49-0c889d86206f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drzcv" Jul 23 09:16:53.911468 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.909350 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ba46e6f4-7695-4578-89a5-c95d6a27890a-etc-sysctl-d\") pod \"tuned-8zd5l\" (UID: \"ba46e6f4-7695-4578-89a5-c95d6a27890a\") " pod="openshift-cluster-node-tuning-operator/tuned-8zd5l" Jul 23 09:16:53.911468 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.909694 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/33072424-e286-4580-a3b5-0ccf6b4cbcf8-dbus\") pod \"global-pull-secret-syncer-l6fgq\" (UID: \"33072424-e286-4580-a3b5-0ccf6b4cbcf8\") " pod="kube-system/global-pull-secret-syncer-l6fgq" Jul 23 09:16:53.911468 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.909748 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7bccf2b8-bb0e-4371-bd49-0c889d86206f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-drzcv\" (UID: \"7bccf2b8-bb0e-4371-bd49-0c889d86206f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drzcv" Jul 23 09:16:53.912191 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.909797 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7bccf2b8-bb0e-4371-bd49-0c889d86206f-registration-dir\") pod \"aws-ebs-csi-driver-node-drzcv\" (UID: \"7bccf2b8-bb0e-4371-bd49-0c889d86206f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drzcv" Jul 23 09:16:53.912191 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.909901 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c4122a2d-66ca-4ec5-969b-14b89809eaa0-konnectivity-ca\") pod \"konnectivity-agent-czm9w\" (UID: \"c4122a2d-66ca-4ec5-969b-14b89809eaa0\") " pod="kube-system/konnectivity-agent-czm9w" Jul 23 09:16:53.912191 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.912045 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c4122a2d-66ca-4ec5-969b-14b89809eaa0-agent-certs\") pod \"konnectivity-agent-czm9w\" (UID: \"c4122a2d-66ca-4ec5-969b-14b89809eaa0\") " pod="kube-system/konnectivity-agent-czm9w" Jul 23 09:16:53.914068 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.914035 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ba46e6f4-7695-4578-89a5-c95d6a27890a-tmp\") pod \"tuned-8zd5l\" (UID: \"ba46e6f4-7695-4578-89a5-c95d6a27890a\") " pod="openshift-cluster-node-tuning-operator/tuned-8zd5l" Jul 23 09:16:53.914068 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.914061 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ba46e6f4-7695-4578-89a5-c95d6a27890a-etc-tuned\") pod \"tuned-8zd5l\" (UID: \"ba46e6f4-7695-4578-89a5-c95d6a27890a\") " pod="openshift-cluster-node-tuning-operator/tuned-8zd5l" Jul 23 09:16:53.918650 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.918618 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77v7h\" (UniqueName: \"kubernetes.io/projected/ba46e6f4-7695-4578-89a5-c95d6a27890a-kube-api-access-77v7h\") pod \"tuned-8zd5l\" (UID: \"ba46e6f4-7695-4578-89a5-c95d6a27890a\") " pod="openshift-cluster-node-tuning-operator/tuned-8zd5l" Jul 23 09:16:53.919682 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:53.919663 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm8m6\" (UniqueName: \"kubernetes.io/projected/7bccf2b8-bb0e-4371-bd49-0c889d86206f-kube-api-access-mm8m6\") pod \"aws-ebs-csi-driver-node-drzcv\" (UID: \"7bccf2b8-bb0e-4371-bd49-0c889d86206f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drzcv" Jul 23 09:16:54.010155 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.010129 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99lgf\" (UniqueName: \"kubernetes.io/projected/82e553d7-cc99-4f29-99ba-1816621870c6-kube-api-access-99lgf\") pod \"network-check-target-b24sn\" (UID: \"82e553d7-cc99-4f29-99ba-1816621870c6\") " pod="openshift-network-diagnostics/network-check-target-b24sn" Jul 23 09:16:54.010284 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.010160 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xptqf\" (UniqueName: \"kubernetes.io/projected/34d12a5c-a8ee-40d7-a365-14d8ef2d94ef-kube-api-access-xptqf\") pod \"iptables-alerter-z2nbl\" (UID: \"34d12a5c-a8ee-40d7-a365-14d8ef2d94ef\") " pod="openshift-network-operator/iptables-alerter-z2nbl" Jul 23 09:16:54.010284 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.010178 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5362d831-0050-47f8-acb1-fe06ddf24832-node-log\") pod \"ovnkube-node-gvwsp\" (UID: \"5362d831-0050-47f8-acb1-fe06ddf24832\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvwsp" Jul 23 09:16:54.010284 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.010194 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5362d831-0050-47f8-acb1-fe06ddf24832-host-cni-netd\") pod \"ovnkube-node-gvwsp\" (UID: \"5362d831-0050-47f8-acb1-fe06ddf24832\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvwsp" Jul 23 09:16:54.010284 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.010210 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5362d831-0050-47f8-acb1-fe06ddf24832-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gvwsp\" (UID: \"5362d831-0050-47f8-acb1-fe06ddf24832\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvwsp" Jul 23 09:16:54.010284 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.010253 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5362d831-0050-47f8-acb1-fe06ddf24832-host-cni-netd\") pod \"ovnkube-node-gvwsp\" (UID: \"5362d831-0050-47f8-acb1-fe06ddf24832\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvwsp" Jul 23 09:16:54.010546 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.010282 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5362d831-0050-47f8-acb1-fe06ddf24832-node-log\") pod \"ovnkube-node-gvwsp\" (UID: \"5362d831-0050-47f8-acb1-fe06ddf24832\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvwsp" Jul 23 09:16:54.010546 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.010293 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5362d831-0050-47f8-acb1-fe06ddf24832-ovnkube-script-lib\") pod \"ovnkube-node-gvwsp\" (UID: \"5362d831-0050-47f8-acb1-fe06ddf24832\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvwsp" Jul 23 09:16:54.010546 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.010390 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72bcfd4f-fce8-46b6-b3f1-52e2ba6306fa-tmp-dir\") pod \"node-resolver-gf7qk\" (UID: \"72bcfd4f-fce8-46b6-b3f1-52e2ba6306fa\") " pod="openshift-dns/node-resolver-gf7qk" Jul 23 09:16:54.010546 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.010397 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5362d831-0050-47f8-acb1-fe06ddf24832-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gvwsp\" (UID: \"5362d831-0050-47f8-acb1-fe06ddf24832\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvwsp" Jul 23 09:16:54.010546 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.010419 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/34d12a5c-a8ee-40d7-a365-14d8ef2d94ef-host-slash\") pod \"iptables-alerter-z2nbl\" (UID: \"34d12a5c-a8ee-40d7-a365-14d8ef2d94ef\") " pod="openshift-network-operator/iptables-alerter-z2nbl" Jul 23 09:16:54.010546 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.010448 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5362d831-0050-47f8-acb1-fe06ddf24832-run-systemd\") pod \"ovnkube-node-gvwsp\" (UID: \"5362d831-0050-47f8-acb1-fe06ddf24832\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvwsp" Jul 23 09:16:54.010546 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.010471 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5362d831-0050-47f8-acb1-fe06ddf24832-var-lib-openvswitch\") pod \"ovnkube-node-gvwsp\" (UID: \"5362d831-0050-47f8-acb1-fe06ddf24832\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvwsp" Jul 23 09:16:54.010546 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.010499 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5fdaa22f-7fb2-42dd-8c10-2e422fa19df2-os-release\") pod \"multus-4gw2w\" (UID: \"5fdaa22f-7fb2-42dd-8c10-2e422fa19df2\") " pod="openshift-multus/multus-4gw2w" Jul 23 09:16:54.010546 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.010506 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5362d831-0050-47f8-acb1-fe06ddf24832-run-systemd\") pod \"ovnkube-node-gvwsp\" (UID: \"5362d831-0050-47f8-acb1-fe06ddf24832\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvwsp" Jul 23 09:16:54.010546 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.010511 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5362d831-0050-47f8-acb1-fe06ddf24832-var-lib-openvswitch\") pod \"ovnkube-node-gvwsp\" (UID: \"5362d831-0050-47f8-acb1-fe06ddf24832\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvwsp" Jul 23 09:16:54.010546 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.010473 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/34d12a5c-a8ee-40d7-a365-14d8ef2d94ef-host-slash\") pod \"iptables-alerter-z2nbl\" (UID: \"34d12a5c-a8ee-40d7-a365-14d8ef2d94ef\") " pod="openshift-network-operator/iptables-alerter-z2nbl" Jul 23 09:16:54.010546 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.010516 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5fdaa22f-7fb2-42dd-8c10-2e422fa19df2-host-var-lib-cni-bin\") pod \"multus-4gw2w\" (UID: \"5fdaa22f-7fb2-42dd-8c10-2e422fa19df2\") " pod="openshift-multus/multus-4gw2w" Jul 23 09:16:54.010546 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.010552 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5fdaa22f-7fb2-42dd-8c10-2e422fa19df2-host-var-lib-cni-bin\") pod \"multus-4gw2w\" (UID: \"5fdaa22f-7fb2-42dd-8c10-2e422fa19df2\") " pod="openshift-multus/multus-4gw2w" Jul 23 09:16:54.011218 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.010572 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5fdaa22f-7fb2-42dd-8c10-2e422fa19df2-os-release\") pod \"multus-4gw2w\" (UID: \"5fdaa22f-7fb2-42dd-8c10-2e422fa19df2\") " pod="openshift-multus/multus-4gw2w" Jul 23 09:16:54.011218 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.010580 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5fdaa22f-7fb2-42dd-8c10-2e422fa19df2-host-var-lib-cni-multus\") pod \"multus-4gw2w\" (UID: \"5fdaa22f-7fb2-42dd-8c10-2e422fa19df2\") " pod="openshift-multus/multus-4gw2w" Jul 23 09:16:54.011218 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.010604 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5fdaa22f-7fb2-42dd-8c10-2e422fa19df2-host-run-netns\") pod \"multus-4gw2w\" (UID: \"5fdaa22f-7fb2-42dd-8c10-2e422fa19df2\") " pod="openshift-multus/multus-4gw2w" Jul 23 09:16:54.011218 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.010621 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5fdaa22f-7fb2-42dd-8c10-2e422fa19df2-multus-daemon-config\") pod \"multus-4gw2w\" (UID: \"5fdaa22f-7fb2-42dd-8c10-2e422fa19df2\") " pod="openshift-multus/multus-4gw2w" Jul 23 09:16:54.011218 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.010638 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/95ff7554-5e90-41c1-9ab5-93d6120788fa-cni-binary-copy\") pod \"multus-additional-cni-plugins-xgtzd\" (UID: \"95ff7554-5e90-41c1-9ab5-93d6120788fa\") " pod="openshift-multus/multus-additional-cni-plugins-xgtzd" Jul 23 09:16:54.011218 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.010660 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5fdaa22f-7fb2-42dd-8c10-2e422fa19df2-host-var-lib-cni-multus\") pod \"multus-4gw2w\" (UID: \"5fdaa22f-7fb2-42dd-8c10-2e422fa19df2\") " pod="openshift-multus/multus-4gw2w" Jul 23 09:16:54.011218 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.010655 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/95ff7554-5e90-41c1-9ab5-93d6120788fa-cnibin\") pod \"multus-additional-cni-plugins-xgtzd\" (UID: \"95ff7554-5e90-41c1-9ab5-93d6120788fa\") " pod="openshift-multus/multus-additional-cni-plugins-xgtzd" Jul 23 09:16:54.011218 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.010706 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/95ff7554-5e90-41c1-9ab5-93d6120788fa-os-release\") pod \"multus-additional-cni-plugins-xgtzd\" (UID: \"95ff7554-5e90-41c1-9ab5-93d6120788fa\") " pod="openshift-multus/multus-additional-cni-plugins-xgtzd" Jul 23 09:16:54.011218 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.010713 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/95ff7554-5e90-41c1-9ab5-93d6120788fa-cnibin\") pod \"multus-additional-cni-plugins-xgtzd\" (UID: \"95ff7554-5e90-41c1-9ab5-93d6120788fa\") " pod="openshift-multus/multus-additional-cni-plugins-xgtzd" Jul 23 09:16:54.011218 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.010724 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72bcfd4f-fce8-46b6-b3f1-52e2ba6306fa-tmp-dir\") pod \"node-resolver-gf7qk\" (UID: \"72bcfd4f-fce8-46b6-b3f1-52e2ba6306fa\") " pod="openshift-dns/node-resolver-gf7qk" Jul 23 09:16:54.011218 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.010736 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2qkt\" (UniqueName: \"kubernetes.io/projected/95ff7554-5e90-41c1-9ab5-93d6120788fa-kube-api-access-w2qkt\") pod \"multus-additional-cni-plugins-xgtzd\" (UID: \"95ff7554-5e90-41c1-9ab5-93d6120788fa\") " pod="openshift-multus/multus-additional-cni-plugins-xgtzd" Jul 23 09:16:54.011218 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.010765 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5fdaa22f-7fb2-42dd-8c10-2e422fa19df2-multus-socket-dir-parent\") pod \"multus-4gw2w\" (UID: \"5fdaa22f-7fb2-42dd-8c10-2e422fa19df2\") " pod="openshift-multus/multus-4gw2w" Jul 23 09:16:54.011218 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.010763 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5fdaa22f-7fb2-42dd-8c10-2e422fa19df2-host-run-netns\") pod \"multus-4gw2w\" (UID: \"5fdaa22f-7fb2-42dd-8c10-2e422fa19df2\") " pod="openshift-multus/multus-4gw2w" Jul 23 09:16:54.011218 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.010807 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/95ff7554-5e90-41c1-9ab5-93d6120788fa-os-release\") pod \"multus-additional-cni-plugins-xgtzd\" (UID: \"95ff7554-5e90-41c1-9ab5-93d6120788fa\") " pod="openshift-multus/multus-additional-cni-plugins-xgtzd" Jul 23 09:16:54.011218 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.010819 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5fdaa22f-7fb2-42dd-8c10-2e422fa19df2-multus-socket-dir-parent\") pod \"multus-4gw2w\" (UID: \"5fdaa22f-7fb2-42dd-8c10-2e422fa19df2\") " pod="openshift-multus/multus-4gw2w" Jul 23 09:16:54.011943 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.010866 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5362d831-0050-47f8-acb1-fe06ddf24832-ovnkube-script-lib\") pod \"ovnkube-node-gvwsp\" (UID: \"5362d831-0050-47f8-acb1-fe06ddf24832\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvwsp" Jul 23 09:16:54.011943 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.010874 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d676a11c-2214-4008-8801-efeb43be1ded-host\") pod \"node-ca-9r4c5\" (UID: \"d676a11c-2214-4008-8801-efeb43be1ded\") " pod="openshift-image-registry/node-ca-9r4c5" Jul 23 09:16:54.011943 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.010900 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5fdaa22f-7fb2-42dd-8c10-2e422fa19df2-host-run-k8s-cni-cncf-io\") pod \"multus-4gw2w\" (UID: \"5fdaa22f-7fb2-42dd-8c10-2e422fa19df2\") " pod="openshift-multus/multus-4gw2w" Jul 23 09:16:54.011943 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.010905 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d676a11c-2214-4008-8801-efeb43be1ded-host\") pod \"node-ca-9r4c5\" (UID: \"d676a11c-2214-4008-8801-efeb43be1ded\") " pod="openshift-image-registry/node-ca-9r4c5" Jul 23 09:16:54.011943 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.010928 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5362d831-0050-47f8-acb1-fe06ddf24832-host-kubelet\") pod \"ovnkube-node-gvwsp\" (UID: \"5362d831-0050-47f8-acb1-fe06ddf24832\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvwsp" Jul 23 09:16:54.011943 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.010931 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5fdaa22f-7fb2-42dd-8c10-2e422fa19df2-host-run-k8s-cni-cncf-io\") pod \"multus-4gw2w\" (UID: \"5fdaa22f-7fb2-42dd-8c10-2e422fa19df2\") " pod="openshift-multus/multus-4gw2w" Jul 23 09:16:54.011943 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.010954 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5362d831-0050-47f8-acb1-fe06ddf24832-host-kubelet\") pod \"ovnkube-node-gvwsp\" (UID: \"5362d831-0050-47f8-acb1-fe06ddf24832\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvwsp" Jul 23 09:16:54.011943 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.010970 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/95ff7554-5e90-41c1-9ab5-93d6120788fa-system-cni-dir\") pod \"multus-additional-cni-plugins-xgtzd\" (UID: \"95ff7554-5e90-41c1-9ab5-93d6120788fa\") " pod="openshift-multus/multus-additional-cni-plugins-xgtzd" Jul 23 09:16:54.011943 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.010990 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5362d831-0050-47f8-acb1-fe06ddf24832-systemd-units\") pod \"ovnkube-node-gvwsp\" (UID: \"5362d831-0050-47f8-acb1-fe06ddf24832\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvwsp" Jul 23 09:16:54.011943 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.011006 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5362d831-0050-47f8-acb1-fe06ddf24832-etc-openvswitch\") pod \"ovnkube-node-gvwsp\" (UID: \"5362d831-0050-47f8-acb1-fe06ddf24832\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvwsp" Jul 23 09:16:54.011943 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.011028 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5362d831-0050-47f8-acb1-fe06ddf24832-host-run-ovn-kubernetes\") pod \"ovnkube-node-gvwsp\" (UID: \"5362d831-0050-47f8-acb1-fe06ddf24832\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvwsp" Jul 23 09:16:54.011943 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.011047 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5fdaa22f-7fb2-42dd-8c10-2e422fa19df2-cni-binary-copy\") pod \"multus-4gw2w\" (UID: \"5fdaa22f-7fb2-42dd-8c10-2e422fa19df2\") " pod="openshift-multus/multus-4gw2w" Jul 23 09:16:54.011943 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.011057 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5362d831-0050-47f8-acb1-fe06ddf24832-systemd-units\") pod \"ovnkube-node-gvwsp\" (UID: \"5362d831-0050-47f8-acb1-fe06ddf24832\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvwsp" Jul 23 09:16:54.011943 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.011055 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/95ff7554-5e90-41c1-9ab5-93d6120788fa-system-cni-dir\") pod \"multus-additional-cni-plugins-xgtzd\" (UID: \"95ff7554-5e90-41c1-9ab5-93d6120788fa\") " pod="openshift-multus/multus-additional-cni-plugins-xgtzd" Jul 23 09:16:54.012661 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.011075 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/95ff7554-5e90-41c1-9ab5-93d6120788fa-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xgtzd\" (UID: \"95ff7554-5e90-41c1-9ab5-93d6120788fa\") " pod="openshift-multus/multus-additional-cni-plugins-xgtzd" Jul 23 09:16:54.012661 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.011094 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5362d831-0050-47f8-acb1-fe06ddf24832-etc-openvswitch\") pod \"ovnkube-node-gvwsp\" (UID: \"5362d831-0050-47f8-acb1-fe06ddf24832\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvwsp" Jul 23 09:16:54.012661 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.011105 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5362d831-0050-47f8-acb1-fe06ddf24832-host-run-ovn-kubernetes\") pod \"ovnkube-node-gvwsp\" (UID: \"5362d831-0050-47f8-acb1-fe06ddf24832\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvwsp" Jul 23 09:16:54.012661 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.011106 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t5w8\" (UniqueName: \"kubernetes.io/projected/72bcfd4f-fce8-46b6-b3f1-52e2ba6306fa-kube-api-access-7t5w8\") pod \"node-resolver-gf7qk\" (UID: \"72bcfd4f-fce8-46b6-b3f1-52e2ba6306fa\") " pod="openshift-dns/node-resolver-gf7qk" Jul 23 09:16:54.012661 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.011150 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5362d831-0050-47f8-acb1-fe06ddf24832-ovn-node-metrics-cert\") pod \"ovnkube-node-gvwsp\" (UID: \"5362d831-0050-47f8-acb1-fe06ddf24832\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvwsp" Jul 23 09:16:54.012661 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.011174 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5fdaa22f-7fb2-42dd-8c10-2e422fa19df2-host-var-lib-kubelet\") pod \"multus-4gw2w\" (UID: \"5fdaa22f-7fb2-42dd-8c10-2e422fa19df2\") " pod="openshift-multus/multus-4gw2w" Jul 23 09:16:54.012661 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.011198 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5fdaa22f-7fb2-42dd-8c10-2e422fa19df2-hostroot\") pod \"multus-4gw2w\" (UID: \"5fdaa22f-7fb2-42dd-8c10-2e422fa19df2\") " pod="openshift-multus/multus-4gw2w" Jul 23 09:16:54.012661 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.011229 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5362d831-0050-47f8-acb1-fe06ddf24832-env-overrides\") pod \"ovnkube-node-gvwsp\" (UID: \"5362d831-0050-47f8-acb1-fe06ddf24832\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvwsp" Jul 23 09:16:54.012661 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.011240 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5fdaa22f-7fb2-42dd-8c10-2e422fa19df2-multus-daemon-config\") pod \"multus-4gw2w\" (UID: \"5fdaa22f-7fb2-42dd-8c10-2e422fa19df2\") " pod="openshift-multus/multus-4gw2w" Jul 23 09:16:54.012661 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.011271 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5fdaa22f-7fb2-42dd-8c10-2e422fa19df2-etc-kubernetes\") pod \"multus-4gw2w\" (UID: \"5fdaa22f-7fb2-42dd-8c10-2e422fa19df2\") " pod="openshift-multus/multus-4gw2w" Jul 23 09:16:54.012661 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.011287 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5fdaa22f-7fb2-42dd-8c10-2e422fa19df2-host-var-lib-kubelet\") pod \"multus-4gw2w\" (UID: \"5fdaa22f-7fb2-42dd-8c10-2e422fa19df2\") " pod="openshift-multus/multus-4gw2w" Jul 23 09:16:54.012661 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.011290 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5fdaa22f-7fb2-42dd-8c10-2e422fa19df2-hostroot\") pod \"multus-4gw2w\" (UID: \"5fdaa22f-7fb2-42dd-8c10-2e422fa19df2\") " pod="openshift-multus/multus-4gw2w" Jul 23 09:16:54.012661 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.011319 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmxq5\" (UniqueName: \"kubernetes.io/projected/381423db-deea-45d8-8cff-408186294f3e-kube-api-access-pmxq5\") pod \"network-metrics-daemon-b7fmg\" (UID: \"381423db-deea-45d8-8cff-408186294f3e\") " pod="openshift-multus/network-metrics-daemon-b7fmg" Jul 23 09:16:54.012661 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.011240 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/95ff7554-5e90-41c1-9ab5-93d6120788fa-cni-binary-copy\") pod \"multus-additional-cni-plugins-xgtzd\" (UID: \"95ff7554-5e90-41c1-9ab5-93d6120788fa\") " pod="openshift-multus/multus-additional-cni-plugins-xgtzd" Jul 23 09:16:54.012661 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.011346 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/72bcfd4f-fce8-46b6-b3f1-52e2ba6306fa-hosts-file\") pod \"node-resolver-gf7qk\" (UID: \"72bcfd4f-fce8-46b6-b3f1-52e2ba6306fa\") " pod="openshift-dns/node-resolver-gf7qk" Jul 23 09:16:54.013138 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.011365 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5fdaa22f-7fb2-42dd-8c10-2e422fa19df2-etc-kubernetes\") pod \"multus-4gw2w\" (UID: \"5fdaa22f-7fb2-42dd-8c10-2e422fa19df2\") " pod="openshift-multus/multus-4gw2w" Jul 23 09:16:54.013138 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.011417 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/72bcfd4f-fce8-46b6-b3f1-52e2ba6306fa-hosts-file\") pod \"node-resolver-gf7qk\" (UID: \"72bcfd4f-fce8-46b6-b3f1-52e2ba6306fa\") " pod="openshift-dns/node-resolver-gf7qk" Jul 23 09:16:54.013138 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.011426 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5fdaa22f-7fb2-42dd-8c10-2e422fa19df2-cnibin\") pod \"multus-4gw2w\" (UID: \"5fdaa22f-7fb2-42dd-8c10-2e422fa19df2\") " pod="openshift-multus/multus-4gw2w" Jul 23 09:16:54.013138 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.011370 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5fdaa22f-7fb2-42dd-8c10-2e422fa19df2-cnibin\") pod \"multus-4gw2w\" (UID: \"5fdaa22f-7fb2-42dd-8c10-2e422fa19df2\") " pod="openshift-multus/multus-4gw2w" Jul 23 09:16:54.013138 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.011463 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5fdaa22f-7fb2-42dd-8c10-2e422fa19df2-multus-conf-dir\") pod \"multus-4gw2w\" (UID: \"5fdaa22f-7fb2-42dd-8c10-2e422fa19df2\") " pod="openshift-multus/multus-4gw2w" Jul 23 09:16:54.013138 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.011491 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/34d12a5c-a8ee-40d7-a365-14d8ef2d94ef-iptables-alerter-script\") pod \"iptables-alerter-z2nbl\" (UID: \"34d12a5c-a8ee-40d7-a365-14d8ef2d94ef\") " pod="openshift-network-operator/iptables-alerter-z2nbl" Jul 23 09:16:54.013138 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.011517 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5362d831-0050-47f8-acb1-fe06ddf24832-run-ovn\") pod \"ovnkube-node-gvwsp\" (UID: \"5362d831-0050-47f8-acb1-fe06ddf24832\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvwsp" Jul 23 09:16:54.013138 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.011577 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5362d831-0050-47f8-acb1-fe06ddf24832-run-ovn\") pod \"ovnkube-node-gvwsp\" (UID: \"5362d831-0050-47f8-acb1-fe06ddf24832\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvwsp" Jul 23 09:16:54.013138 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.011600 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5fdaa22f-7fb2-42dd-8c10-2e422fa19df2-multus-conf-dir\") pod \"multus-4gw2w\" (UID: \"5fdaa22f-7fb2-42dd-8c10-2e422fa19df2\") " pod="openshift-multus/multus-4gw2w" Jul 23 09:16:54.013138 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.011619 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/95ff7554-5e90-41c1-9ab5-93d6120788fa-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xgtzd\" (UID: \"95ff7554-5e90-41c1-9ab5-93d6120788fa\") " pod="openshift-multus/multus-additional-cni-plugins-xgtzd" Jul 23 09:16:54.013138 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.011622 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5362d831-0050-47f8-acb1-fe06ddf24832-host-cni-bin\") pod \"ovnkube-node-gvwsp\" (UID: \"5362d831-0050-47f8-acb1-fe06ddf24832\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvwsp" Jul 23 09:16:54.013138 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.011659 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5362d831-0050-47f8-acb1-fe06ddf24832-host-cni-bin\") pod \"ovnkube-node-gvwsp\" (UID: \"5362d831-0050-47f8-acb1-fe06ddf24832\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvwsp" Jul 23 09:16:54.013138 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.011629 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5fdaa22f-7fb2-42dd-8c10-2e422fa19df2-cni-binary-copy\") pod \"multus-4gw2w\" (UID: \"5fdaa22f-7fb2-42dd-8c10-2e422fa19df2\") " pod="openshift-multus/multus-4gw2w" Jul 23 09:16:54.013138 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.011738 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5362d831-0050-47f8-acb1-fe06ddf24832-ovnkube-config\") pod \"ovnkube-node-gvwsp\" (UID: \"5362d831-0050-47f8-acb1-fe06ddf24832\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvwsp" Jul 23 09:16:54.013138 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.011759 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrwmp\" (UniqueName: \"kubernetes.io/projected/d676a11c-2214-4008-8801-efeb43be1ded-kube-api-access-qrwmp\") pod \"node-ca-9r4c5\" (UID: \"d676a11c-2214-4008-8801-efeb43be1ded\") " pod="openshift-image-registry/node-ca-9r4c5" Jul 23 09:16:54.013609 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.011782 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/95ff7554-5e90-41c1-9ab5-93d6120788fa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xgtzd\" (UID: \"95ff7554-5e90-41c1-9ab5-93d6120788fa\") " pod="openshift-multus/multus-additional-cni-plugins-xgtzd" Jul 23 09:16:54.013609 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.011800 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5362d831-0050-47f8-acb1-fe06ddf24832-host-slash\") pod \"ovnkube-node-gvwsp\" (UID: \"5362d831-0050-47f8-acb1-fe06ddf24832\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvwsp" Jul 23 09:16:54.013609 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.011816 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5362d831-0050-47f8-acb1-fe06ddf24832-run-openvswitch\") pod \"ovnkube-node-gvwsp\" (UID: \"5362d831-0050-47f8-acb1-fe06ddf24832\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvwsp" Jul 23 09:16:54.013609 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.011818 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5362d831-0050-47f8-acb1-fe06ddf24832-env-overrides\") pod \"ovnkube-node-gvwsp\" (UID: \"5362d831-0050-47f8-acb1-fe06ddf24832\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvwsp" Jul 23 09:16:54.013609 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.011841 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5362d831-0050-47f8-acb1-fe06ddf24832-log-socket\") pod \"ovnkube-node-gvwsp\" (UID: \"5362d831-0050-47f8-acb1-fe06ddf24832\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvwsp" Jul 23 09:16:54.013609 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.011869 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwp5n\" (UniqueName: \"kubernetes.io/projected/5362d831-0050-47f8-acb1-fe06ddf24832-kube-api-access-vwp5n\") pod \"ovnkube-node-gvwsp\" (UID: \"5362d831-0050-47f8-acb1-fe06ddf24832\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvwsp" Jul 23 09:16:54.013609 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.011873 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5362d831-0050-47f8-acb1-fe06ddf24832-host-slash\") pod \"ovnkube-node-gvwsp\" (UID: \"5362d831-0050-47f8-acb1-fe06ddf24832\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvwsp" Jul 23 09:16:54.013609 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.011894 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5fdaa22f-7fb2-42dd-8c10-2e422fa19df2-system-cni-dir\") pod \"multus-4gw2w\" (UID: \"5fdaa22f-7fb2-42dd-8c10-2e422fa19df2\") " pod="openshift-multus/multus-4gw2w" Jul 23 09:16:54.013609 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.011922 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/95ff7554-5e90-41c1-9ab5-93d6120788fa-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xgtzd\" (UID: \"95ff7554-5e90-41c1-9ab5-93d6120788fa\") " pod="openshift-multus/multus-additional-cni-plugins-xgtzd" Jul 23 09:16:54.013609 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.011953 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5362d831-0050-47f8-acb1-fe06ddf24832-run-openvswitch\") pod \"ovnkube-node-gvwsp\" (UID: \"5362d831-0050-47f8-acb1-fe06ddf24832\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvwsp" Jul 23 09:16:54.013609 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.011969 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5fdaa22f-7fb2-42dd-8c10-2e422fa19df2-multus-cni-dir\") pod \"multus-4gw2w\" (UID: \"5fdaa22f-7fb2-42dd-8c10-2e422fa19df2\") " pod="openshift-multus/multus-4gw2w" Jul 23 09:16:54.013609 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.011972 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/95ff7554-5e90-41c1-9ab5-93d6120788fa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xgtzd\" (UID: \"95ff7554-5e90-41c1-9ab5-93d6120788fa\") " pod="openshift-multus/multus-additional-cni-plugins-xgtzd" Jul 23 09:16:54.013609 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.011923 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5362d831-0050-47f8-acb1-fe06ddf24832-log-socket\") pod \"ovnkube-node-gvwsp\" (UID: \"5362d831-0050-47f8-acb1-fe06ddf24832\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvwsp" Jul 23 09:16:54.013609 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.011997 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm59d\" (UniqueName: \"kubernetes.io/projected/5fdaa22f-7fb2-42dd-8c10-2e422fa19df2-kube-api-access-nm59d\") pod \"multus-4gw2w\" (UID: \"5fdaa22f-7fb2-42dd-8c10-2e422fa19df2\") " pod="openshift-multus/multus-4gw2w" Jul 23 09:16:54.013609 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.012012 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5fdaa22f-7fb2-42dd-8c10-2e422fa19df2-system-cni-dir\") pod \"multus-4gw2w\" (UID: \"5fdaa22f-7fb2-42dd-8c10-2e422fa19df2\") " pod="openshift-multus/multus-4gw2w" Jul 23 09:16:54.014153 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.012052 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5fdaa22f-7fb2-42dd-8c10-2e422fa19df2-multus-cni-dir\") pod \"multus-4gw2w\" (UID: \"5fdaa22f-7fb2-42dd-8c10-2e422fa19df2\") " pod="openshift-multus/multus-4gw2w" Jul 23 09:16:54.014153 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.012055 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/381423db-deea-45d8-8cff-408186294f3e-metrics-certs\") pod \"network-metrics-daemon-b7fmg\" (UID: \"381423db-deea-45d8-8cff-408186294f3e\") " pod="openshift-multus/network-metrics-daemon-b7fmg" Jul 23 09:16:54.014153 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.012059 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/34d12a5c-a8ee-40d7-a365-14d8ef2d94ef-iptables-alerter-script\") pod \"iptables-alerter-z2nbl\" (UID: \"34d12a5c-a8ee-40d7-a365-14d8ef2d94ef\") " pod="openshift-network-operator/iptables-alerter-z2nbl" Jul 23 09:16:54.014153 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.012100 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5362d831-0050-47f8-acb1-fe06ddf24832-host-run-netns\") pod \"ovnkube-node-gvwsp\" (UID: \"5362d831-0050-47f8-acb1-fe06ddf24832\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvwsp" Jul 23 09:16:54.014153 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:16:54.012166 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 09:16:54.014153 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.012204 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d676a11c-2214-4008-8801-efeb43be1ded-serviceca\") pod \"node-ca-9r4c5\" (UID: \"d676a11c-2214-4008-8801-efeb43be1ded\") " pod="openshift-image-registry/node-ca-9r4c5" Jul 23 09:16:54.014153 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.012204 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5362d831-0050-47f8-acb1-fe06ddf24832-host-run-netns\") pod \"ovnkube-node-gvwsp\" (UID: \"5362d831-0050-47f8-acb1-fe06ddf24832\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvwsp" Jul 23 09:16:54.014153 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.012240 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5fdaa22f-7fb2-42dd-8c10-2e422fa19df2-host-run-multus-certs\") pod \"multus-4gw2w\" (UID: \"5fdaa22f-7fb2-42dd-8c10-2e422fa19df2\") " pod="openshift-multus/multus-4gw2w" Jul 23 09:16:54.014153 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.012308 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5362d831-0050-47f8-acb1-fe06ddf24832-ovnkube-config\") pod \"ovnkube-node-gvwsp\" (UID: \"5362d831-0050-47f8-acb1-fe06ddf24832\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvwsp" Jul 23 09:16:54.014153 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.012320 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5fdaa22f-7fb2-42dd-8c10-2e422fa19df2-host-run-multus-certs\") pod \"multus-4gw2w\" (UID: \"5fdaa22f-7fb2-42dd-8c10-2e422fa19df2\") " pod="openshift-multus/multus-4gw2w" Jul 23 09:16:54.014153 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:16:54.012355 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/381423db-deea-45d8-8cff-408186294f3e-metrics-certs podName:381423db-deea-45d8-8cff-408186294f3e nodeName:}" failed. No retries permitted until 2026-07-23 09:16:54.512337553 +0000 UTC m=+3.051616109 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/381423db-deea-45d8-8cff-408186294f3e-metrics-certs") pod "network-metrics-daemon-b7fmg" (UID: "381423db-deea-45d8-8cff-408186294f3e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 09:16:54.014153 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.012538 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/95ff7554-5e90-41c1-9ab5-93d6120788fa-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xgtzd\" (UID: \"95ff7554-5e90-41c1-9ab5-93d6120788fa\") " pod="openshift-multus/multus-additional-cni-plugins-xgtzd" Jul 23 09:16:54.014153 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.012557 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d676a11c-2214-4008-8801-efeb43be1ded-serviceca\") pod \"node-ca-9r4c5\" (UID: \"d676a11c-2214-4008-8801-efeb43be1ded\") " pod="openshift-image-registry/node-ca-9r4c5" Jul 23 09:16:54.014153 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.013773 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5362d831-0050-47f8-acb1-fe06ddf24832-ovn-node-metrics-cert\") pod \"ovnkube-node-gvwsp\" (UID: \"5362d831-0050-47f8-acb1-fe06ddf24832\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvwsp" Jul 23 09:16:54.017561 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:16:54.017538 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 09:16:54.017645 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:16:54.017568 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 09:16:54.017645 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:16:54.017578 2576 projected.go:196] Error preparing data for projected volume kube-api-access-99lgf for pod openshift-network-diagnostics/network-check-target-b24sn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 09:16:54.017760 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:16:54.017670 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/82e553d7-cc99-4f29-99ba-1816621870c6-kube-api-access-99lgf podName:82e553d7-cc99-4f29-99ba-1816621870c6 nodeName:}" failed. No retries permitted until 2026-07-23 09:16:54.517631505 +0000 UTC m=+3.056910077 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-99lgf" (UniqueName: "kubernetes.io/projected/82e553d7-cc99-4f29-99ba-1816621870c6-kube-api-access-99lgf") pod "network-check-target-b24sn" (UID: "82e553d7-cc99-4f29-99ba-1816621870c6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 09:16:54.019923 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.019896 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2qkt\" (UniqueName: \"kubernetes.io/projected/95ff7554-5e90-41c1-9ab5-93d6120788fa-kube-api-access-w2qkt\") pod \"multus-additional-cni-plugins-xgtzd\" (UID: \"95ff7554-5e90-41c1-9ab5-93d6120788fa\") " pod="openshift-multus/multus-additional-cni-plugins-xgtzd" Jul 23 09:16:54.020367 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.020342 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xptqf\" (UniqueName: \"kubernetes.io/projected/34d12a5c-a8ee-40d7-a365-14d8ef2d94ef-kube-api-access-xptqf\") pod \"iptables-alerter-z2nbl\" (UID: \"34d12a5c-a8ee-40d7-a365-14d8ef2d94ef\") " pod="openshift-network-operator/iptables-alerter-z2nbl" Jul 23 09:16:54.020450 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.020420 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmxq5\" (UniqueName: \"kubernetes.io/projected/381423db-deea-45d8-8cff-408186294f3e-kube-api-access-pmxq5\") pod \"network-metrics-daemon-b7fmg\" (UID: \"381423db-deea-45d8-8cff-408186294f3e\") " pod="openshift-multus/network-metrics-daemon-b7fmg" Jul 23 09:16:54.022153 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.022123 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm59d\" (UniqueName: \"kubernetes.io/projected/5fdaa22f-7fb2-42dd-8c10-2e422fa19df2-kube-api-access-nm59d\") pod \"multus-4gw2w\" (UID: \"5fdaa22f-7fb2-42dd-8c10-2e422fa19df2\") " pod="openshift-multus/multus-4gw2w" Jul 23 09:16:54.022412 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.022387 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t5w8\" (UniqueName: \"kubernetes.io/projected/72bcfd4f-fce8-46b6-b3f1-52e2ba6306fa-kube-api-access-7t5w8\") pod \"node-resolver-gf7qk\" (UID: \"72bcfd4f-fce8-46b6-b3f1-52e2ba6306fa\") " pod="openshift-dns/node-resolver-gf7qk" Jul 23 09:16:54.022491 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.022465 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrwmp\" (UniqueName: \"kubernetes.io/projected/d676a11c-2214-4008-8801-efeb43be1ded-kube-api-access-qrwmp\") pod \"node-ca-9r4c5\" (UID: \"d676a11c-2214-4008-8801-efeb43be1ded\") " pod="openshift-image-registry/node-ca-9r4c5" Jul 23 09:16:54.022491 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.022479 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwp5n\" (UniqueName: \"kubernetes.io/projected/5362d831-0050-47f8-acb1-fe06ddf24832-kube-api-access-vwp5n\") pod \"ovnkube-node-gvwsp\" (UID: \"5362d831-0050-47f8-acb1-fe06ddf24832\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvwsp" Jul 23 09:16:54.034716 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:54.034664 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e4c6298602a592029f63da426ffeb44.slice/crio-64ea38958aec3d268b508276987c2b092ae717c2b7af6b5d410197d24ab94fed WatchSource:0}: Error finding container 64ea38958aec3d268b508276987c2b092ae717c2b7af6b5d410197d24ab94fed: Status 404 returned error can't find the container with id 64ea38958aec3d268b508276987c2b092ae717c2b7af6b5d410197d24ab94fed Jul 23 09:16:54.035165 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:54.035148 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00d27bbe0e5e208cc7ea070ef5013896.slice/crio-26a1627019d85deb7f6a3a27c87d792f4f2a8c2dd56c83a32170686e23c9b881 WatchSource:0}: Error finding container 26a1627019d85deb7f6a3a27c87d792f4f2a8c2dd56c83a32170686e23c9b881: Status 404 returned error can't find the container with id 26a1627019d85deb7f6a3a27c87d792f4f2a8c2dd56c83a32170686e23c9b881 Jul 23 09:16:54.041406 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.041389 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 09:16:54.125863 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.125837 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-czm9w" Jul 23 09:16:54.135679 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:54.135648 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4122a2d_66ca_4ec5_969b_14b89809eaa0.slice/crio-194d0bc050f586f33c192cbfc4e08edadf9cfbab6183afa9de37b35768e7f15a WatchSource:0}: Error finding container 194d0bc050f586f33c192cbfc4e08edadf9cfbab6183afa9de37b35768e7f15a: Status 404 returned error can't find the container with id 194d0bc050f586f33c192cbfc4e08edadf9cfbab6183afa9de37b35768e7f15a Jul 23 09:16:54.145857 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.145840 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drzcv" Jul 23 09:16:54.153622 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:54.153601 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bccf2b8_bb0e_4371_bd49_0c889d86206f.slice/crio-49798cbd5b7ea1632b88b1042c425c363ace8e345e7f317a662d4bda65cf56de WatchSource:0}: Error finding container 49798cbd5b7ea1632b88b1042c425c363ace8e345e7f317a662d4bda65cf56de: Status 404 returned error can't find the container with id 49798cbd5b7ea1632b88b1042c425c363ace8e345e7f317a662d4bda65cf56de Jul 23 09:16:54.157596 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.157573 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8zd5l" Jul 23 09:16:54.165291 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:54.165274 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba46e6f4_7695_4578_89a5_c95d6a27890a.slice/crio-8f14fdda0d1fe6d9561bc672f73bdffa8dcc02d27e16221780c50cc1bc476057 WatchSource:0}: Error finding container 8f14fdda0d1fe6d9561bc672f73bdffa8dcc02d27e16221780c50cc1bc476057: Status 404 returned error can't find the container with id 8f14fdda0d1fe6d9561bc672f73bdffa8dcc02d27e16221780c50cc1bc476057 Jul 23 09:16:54.173637 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.173609 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9r4c5" Jul 23 09:16:54.181132 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:54.181114 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd676a11c_2214_4008_8801_efeb43be1ded.slice/crio-1bbb6cb08a0585870464b236de17fbe85470c698e966785048daf1ddefc9d489 WatchSource:0}: Error finding container 1bbb6cb08a0585870464b236de17fbe85470c698e966785048daf1ddefc9d489: Status 404 returned error can't find the container with id 1bbb6cb08a0585870464b236de17fbe85470c698e966785048daf1ddefc9d489 Jul 23 09:16:54.204662 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.204643 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gf7qk" Jul 23 09:16:54.211891 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:54.211873 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72bcfd4f_fce8_46b6_b3f1_52e2ba6306fa.slice/crio-871521f760521f30e6d81a718de53e3990b7c9852e86541aeb34dbb2b038b95c WatchSource:0}: Error finding container 871521f760521f30e6d81a718de53e3990b7c9852e86541aeb34dbb2b038b95c: Status 404 returned error can't find the container with id 871521f760521f30e6d81a718de53e3990b7c9852e86541aeb34dbb2b038b95c Jul 23 09:16:54.215180 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.215166 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4gw2w" Jul 23 09:16:54.223070 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:54.223039 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fdaa22f_7fb2_42dd_8c10_2e422fa19df2.slice/crio-2b9be04bc79badbf29231df847861c0a961ece32c6cbce75e317bcaee5e3714f WatchSource:0}: Error finding container 2b9be04bc79badbf29231df847861c0a961ece32c6cbce75e317bcaee5e3714f: Status 404 returned error can't find the container with id 2b9be04bc79badbf29231df847861c0a961ece32c6cbce75e317bcaee5e3714f Jul 23 09:16:54.236219 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.236201 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xgtzd" Jul 23 09:16:54.243669 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:54.243651 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95ff7554_5e90_41c1_9ab5_93d6120788fa.slice/crio-abf03a275876ebe9568b699645339894157362f186f88f93a69c92bb1717a82e WatchSource:0}: Error finding container abf03a275876ebe9568b699645339894157362f186f88f93a69c92bb1717a82e: Status 404 returned error can't find the container with id abf03a275876ebe9568b699645339894157362f186f88f93a69c92bb1717a82e Jul 23 09:16:54.246013 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.245981 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-z2nbl" Jul 23 09:16:54.250615 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.250595 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gvwsp" Jul 23 09:16:54.254903 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:54.254881 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34d12a5c_a8ee_40d7_a365_14d8ef2d94ef.slice/crio-509999d2e6d2f54ee571903b3bce9c39460537c42a8b7552422945a90df1c48f WatchSource:0}: Error finding container 509999d2e6d2f54ee571903b3bce9c39460537c42a8b7552422945a90df1c48f: Status 404 returned error can't find the container with id 509999d2e6d2f54ee571903b3bce9c39460537c42a8b7552422945a90df1c48f Jul 23 09:16:54.259064 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:16:54.259024 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5362d831_0050_47f8_acb1_fe06ddf24832.slice/crio-107f28cb6cafc23543a6e6f10cc15300638dbe1fdbe5fa2132de608860df4c10 WatchSource:0}: Error finding container 107f28cb6cafc23543a6e6f10cc15300638dbe1fdbe5fa2132de608860df4c10: Status 404 returned error can't find the container with id 107f28cb6cafc23543a6e6f10cc15300638dbe1fdbe5fa2132de608860df4c10 Jul 23 09:16:54.415479 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.415451 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/33072424-e286-4580-a3b5-0ccf6b4cbcf8-original-pull-secret\") pod \"global-pull-secret-syncer-l6fgq\" (UID: \"33072424-e286-4580-a3b5-0ccf6b4cbcf8\") " pod="kube-system/global-pull-secret-syncer-l6fgq" Jul 23 09:16:54.415605 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:16:54.415553 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 09:16:54.415605 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:16:54.415599 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33072424-e286-4580-a3b5-0ccf6b4cbcf8-original-pull-secret podName:33072424-e286-4580-a3b5-0ccf6b4cbcf8 nodeName:}" failed. No retries permitted until 2026-07-23 09:16:55.415585286 +0000 UTC m=+3.954863842 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/33072424-e286-4580-a3b5-0ccf6b4cbcf8-original-pull-secret") pod "global-pull-secret-syncer-l6fgq" (UID: "33072424-e286-4580-a3b5-0ccf6b4cbcf8") : object "kube-system"/"original-pull-secret" not registered Jul 23 09:16:54.516844 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.516808 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/381423db-deea-45d8-8cff-408186294f3e-metrics-certs\") pod \"network-metrics-daemon-b7fmg\" (UID: \"381423db-deea-45d8-8cff-408186294f3e\") " pod="openshift-multus/network-metrics-daemon-b7fmg" Jul 23 09:16:54.516996 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:16:54.516976 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 09:16:54.517061 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:16:54.517053 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/381423db-deea-45d8-8cff-408186294f3e-metrics-certs podName:381423db-deea-45d8-8cff-408186294f3e nodeName:}" failed. No retries permitted until 2026-07-23 09:16:55.51703108 +0000 UTC m=+4.056309657 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/381423db-deea-45d8-8cff-408186294f3e-metrics-certs") pod "network-metrics-daemon-b7fmg" (UID: "381423db-deea-45d8-8cff-408186294f3e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 09:16:54.618194 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.618160 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99lgf\" (UniqueName: \"kubernetes.io/projected/82e553d7-cc99-4f29-99ba-1816621870c6-kube-api-access-99lgf\") pod \"network-check-target-b24sn\" (UID: \"82e553d7-cc99-4f29-99ba-1816621870c6\") " pod="openshift-network-diagnostics/network-check-target-b24sn" Jul 23 09:16:54.618420 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:16:54.618403 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 09:16:54.618497 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:16:54.618433 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 09:16:54.618497 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:16:54.618445 2576 projected.go:196] Error preparing data for projected volume kube-api-access-99lgf for pod openshift-network-diagnostics/network-check-target-b24sn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 09:16:54.618598 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:16:54.618502 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/82e553d7-cc99-4f29-99ba-1816621870c6-kube-api-access-99lgf podName:82e553d7-cc99-4f29-99ba-1816621870c6 nodeName:}" failed. No retries permitted until 2026-07-23 09:16:55.618484668 +0000 UTC m=+4.157763228 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-99lgf" (UniqueName: "kubernetes.io/projected/82e553d7-cc99-4f29-99ba-1816621870c6-kube-api-access-99lgf") pod "network-check-target-b24sn" (UID: "82e553d7-cc99-4f29-99ba-1816621870c6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 09:16:54.870432 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.870315 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-22 09:11:53 +0000 UTC" deadline="2028-02-29 17:48:28.391330494 +0000 UTC" Jul 23 09:16:54.870432 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.870387 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14072h31m33.520946739s" Jul 23 09:16:54.980305 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.980177 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b7fmg" Jul 23 09:16:54.980470 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:16:54.980338 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b7fmg" podUID="381423db-deea-45d8-8cff-408186294f3e" Jul 23 09:16:54.980948 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:54.980871 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b24sn" Jul 23 09:16:54.981057 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:16:54.980975 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b24sn" podUID="82e553d7-cc99-4f29-99ba-1816621870c6" Jul 23 09:16:55.014136 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:55.014085 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4gw2w" event={"ID":"5fdaa22f-7fb2-42dd-8c10-2e422fa19df2","Type":"ContainerStarted","Data":"2b9be04bc79badbf29231df847861c0a961ece32c6cbce75e317bcaee5e3714f"} Jul 23 09:16:55.019499 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:55.019477 2576 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 09:16:55.023940 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:55.023912 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9r4c5" event={"ID":"d676a11c-2214-4008-8801-efeb43be1ded","Type":"ContainerStarted","Data":"1bbb6cb08a0585870464b236de17fbe85470c698e966785048daf1ddefc9d489"} Jul 23 09:16:55.035764 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:55.035736 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drzcv" event={"ID":"7bccf2b8-bb0e-4371-bd49-0c889d86206f","Type":"ContainerStarted","Data":"49798cbd5b7ea1632b88b1042c425c363ace8e345e7f317a662d4bda65cf56de"} Jul 23 09:16:55.042531 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:55.042504 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-czm9w" event={"ID":"c4122a2d-66ca-4ec5-969b-14b89809eaa0","Type":"ContainerStarted","Data":"194d0bc050f586f33c192cbfc4e08edadf9cfbab6183afa9de37b35768e7f15a"} Jul 23 09:16:55.059012 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:55.058986 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-145.ec2.internal" event={"ID":"6e4c6298602a592029f63da426ffeb44","Type":"ContainerStarted","Data":"64ea38958aec3d268b508276987c2b092ae717c2b7af6b5d410197d24ab94fed"} Jul 23 09:16:55.075327 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:55.075297 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gf7qk" event={"ID":"72bcfd4f-fce8-46b6-b3f1-52e2ba6306fa","Type":"ContainerStarted","Data":"871521f760521f30e6d81a718de53e3990b7c9852e86541aeb34dbb2b038b95c"} Jul 23 09:16:55.086698 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:55.086671 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8zd5l" event={"ID":"ba46e6f4-7695-4578-89a5-c95d6a27890a","Type":"ContainerStarted","Data":"8f14fdda0d1fe6d9561bc672f73bdffa8dcc02d27e16221780c50cc1bc476057"} Jul 23 09:16:55.105664 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:55.105635 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-145.ec2.internal" event={"ID":"00d27bbe0e5e208cc7ea070ef5013896","Type":"ContainerStarted","Data":"26a1627019d85deb7f6a3a27c87d792f4f2a8c2dd56c83a32170686e23c9b881"} Jul 23 09:16:55.117594 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:55.117564 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gvwsp" event={"ID":"5362d831-0050-47f8-acb1-fe06ddf24832","Type":"ContainerStarted","Data":"107f28cb6cafc23543a6e6f10cc15300638dbe1fdbe5fa2132de608860df4c10"} Jul 23 09:16:55.120329 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:55.120302 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-z2nbl" event={"ID":"34d12a5c-a8ee-40d7-a365-14d8ef2d94ef","Type":"ContainerStarted","Data":"509999d2e6d2f54ee571903b3bce9c39460537c42a8b7552422945a90df1c48f"} Jul 23 09:16:55.124888 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:55.124817 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xgtzd" event={"ID":"95ff7554-5e90-41c1-9ab5-93d6120788fa","Type":"ContainerStarted","Data":"abf03a275876ebe9568b699645339894157362f186f88f93a69c92bb1717a82e"} Jul 23 09:16:55.427177 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:55.426534 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/33072424-e286-4580-a3b5-0ccf6b4cbcf8-original-pull-secret\") pod \"global-pull-secret-syncer-l6fgq\" (UID: \"33072424-e286-4580-a3b5-0ccf6b4cbcf8\") " pod="kube-system/global-pull-secret-syncer-l6fgq" Jul 23 09:16:55.427177 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:16:55.426704 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 09:16:55.427177 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:16:55.426761 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33072424-e286-4580-a3b5-0ccf6b4cbcf8-original-pull-secret podName:33072424-e286-4580-a3b5-0ccf6b4cbcf8 nodeName:}" failed. No retries permitted until 2026-07-23 09:16:57.426743057 +0000 UTC m=+5.966021618 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/33072424-e286-4580-a3b5-0ccf6b4cbcf8-original-pull-secret") pod "global-pull-secret-syncer-l6fgq" (UID: "33072424-e286-4580-a3b5-0ccf6b4cbcf8") : object "kube-system"/"original-pull-secret" not registered Jul 23 09:16:55.527944 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:55.527416 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/381423db-deea-45d8-8cff-408186294f3e-metrics-certs\") pod \"network-metrics-daemon-b7fmg\" (UID: \"381423db-deea-45d8-8cff-408186294f3e\") " pod="openshift-multus/network-metrics-daemon-b7fmg" Jul 23 09:16:55.527944 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:16:55.527576 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 09:16:55.527944 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:16:55.527629 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/381423db-deea-45d8-8cff-408186294f3e-metrics-certs podName:381423db-deea-45d8-8cff-408186294f3e nodeName:}" failed. No retries permitted until 2026-07-23 09:16:57.527613438 +0000 UTC m=+6.066891999 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/381423db-deea-45d8-8cff-408186294f3e-metrics-certs") pod "network-metrics-daemon-b7fmg" (UID: "381423db-deea-45d8-8cff-408186294f3e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 09:16:55.629300 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:55.629183 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99lgf\" (UniqueName: \"kubernetes.io/projected/82e553d7-cc99-4f29-99ba-1816621870c6-kube-api-access-99lgf\") pod \"network-check-target-b24sn\" (UID: \"82e553d7-cc99-4f29-99ba-1816621870c6\") " pod="openshift-network-diagnostics/network-check-target-b24sn" Jul 23 09:16:55.629466 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:16:55.629374 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 09:16:55.629466 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:16:55.629398 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 09:16:55.629466 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:16:55.629409 2576 projected.go:196] Error preparing data for projected volume kube-api-access-99lgf for pod openshift-network-diagnostics/network-check-target-b24sn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 09:16:55.629609 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:16:55.629476 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/82e553d7-cc99-4f29-99ba-1816621870c6-kube-api-access-99lgf podName:82e553d7-cc99-4f29-99ba-1816621870c6 nodeName:}" failed. No retries permitted until 2026-07-23 09:16:57.629456321 +0000 UTC m=+6.168734902 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-99lgf" (UniqueName: "kubernetes.io/projected/82e553d7-cc99-4f29-99ba-1816621870c6-kube-api-access-99lgf") pod "network-check-target-b24sn" (UID: "82e553d7-cc99-4f29-99ba-1816621870c6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 09:16:55.871415 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:55.871321 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-22 09:11:53 +0000 UTC" deadline="2028-03-07 06:08:20.89495374 +0000 UTC" Jul 23 09:16:55.871415 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:55.871362 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14228h51m25.023597495s" Jul 23 09:16:55.980207 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:55.980173 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6fgq" Jul 23 09:16:55.980381 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:16:55.980350 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6fgq" podUID="33072424-e286-4580-a3b5-0ccf6b4cbcf8" Jul 23 09:16:56.990265 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:56.989880 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b24sn" Jul 23 09:16:56.990265 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:16:56.990001 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b24sn" podUID="82e553d7-cc99-4f29-99ba-1816621870c6" Jul 23 09:16:56.990764 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:56.989881 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b7fmg" Jul 23 09:16:56.990764 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:16:56.990688 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b7fmg" podUID="381423db-deea-45d8-8cff-408186294f3e" Jul 23 09:16:57.445476 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:57.445439 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/33072424-e286-4580-a3b5-0ccf6b4cbcf8-original-pull-secret\") pod \"global-pull-secret-syncer-l6fgq\" (UID: \"33072424-e286-4580-a3b5-0ccf6b4cbcf8\") " pod="kube-system/global-pull-secret-syncer-l6fgq" Jul 23 09:16:57.445659 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:16:57.445640 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 09:16:57.445722 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:16:57.445701 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33072424-e286-4580-a3b5-0ccf6b4cbcf8-original-pull-secret podName:33072424-e286-4580-a3b5-0ccf6b4cbcf8 nodeName:}" failed. No retries permitted until 2026-07-23 09:17:01.445683587 +0000 UTC m=+9.984962154 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/33072424-e286-4580-a3b5-0ccf6b4cbcf8-original-pull-secret") pod "global-pull-secret-syncer-l6fgq" (UID: "33072424-e286-4580-a3b5-0ccf6b4cbcf8") : object "kube-system"/"original-pull-secret" not registered Jul 23 09:16:57.546087 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:57.546048 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/381423db-deea-45d8-8cff-408186294f3e-metrics-certs\") pod \"network-metrics-daemon-b7fmg\" (UID: \"381423db-deea-45d8-8cff-408186294f3e\") " pod="openshift-multus/network-metrics-daemon-b7fmg" Jul 23 09:16:57.546270 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:16:57.546184 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 09:16:57.546333 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:16:57.546274 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/381423db-deea-45d8-8cff-408186294f3e-metrics-certs podName:381423db-deea-45d8-8cff-408186294f3e nodeName:}" failed. No retries permitted until 2026-07-23 09:17:01.54625303 +0000 UTC m=+10.085531601 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/381423db-deea-45d8-8cff-408186294f3e-metrics-certs") pod "network-metrics-daemon-b7fmg" (UID: "381423db-deea-45d8-8cff-408186294f3e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 09:16:57.646710 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:57.646624 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99lgf\" (UniqueName: \"kubernetes.io/projected/82e553d7-cc99-4f29-99ba-1816621870c6-kube-api-access-99lgf\") pod \"network-check-target-b24sn\" (UID: \"82e553d7-cc99-4f29-99ba-1816621870c6\") " pod="openshift-network-diagnostics/network-check-target-b24sn" Jul 23 09:16:57.646889 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:16:57.646869 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 09:16:57.646966 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:16:57.646911 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 09:16:57.646966 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:16:57.646924 2576 projected.go:196] Error preparing data for projected volume kube-api-access-99lgf for pod openshift-network-diagnostics/network-check-target-b24sn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 09:16:57.647065 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:16:57.646997 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/82e553d7-cc99-4f29-99ba-1816621870c6-kube-api-access-99lgf podName:82e553d7-cc99-4f29-99ba-1816621870c6 nodeName:}" failed. No retries permitted until 2026-07-23 09:17:01.646979898 +0000 UTC m=+10.186258468 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-99lgf" (UniqueName: "kubernetes.io/projected/82e553d7-cc99-4f29-99ba-1816621870c6-kube-api-access-99lgf") pod "network-check-target-b24sn" (UID: "82e553d7-cc99-4f29-99ba-1816621870c6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 09:16:57.979808 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:57.979776 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6fgq" Jul 23 09:16:57.979986 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:16:57.979899 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6fgq" podUID="33072424-e286-4580-a3b5-0ccf6b4cbcf8" Jul 23 09:16:58.980150 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:58.979456 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b24sn" Jul 23 09:16:58.980150 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:16:58.979614 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b24sn" podUID="82e553d7-cc99-4f29-99ba-1816621870c6" Jul 23 09:16:58.980150 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:58.979456 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b7fmg" Jul 23 09:16:58.980150 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:16:58.980074 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b7fmg" podUID="381423db-deea-45d8-8cff-408186294f3e" Jul 23 09:16:59.980292 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:16:59.980110 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6fgq" Jul 23 09:16:59.980292 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:16:59.980264 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6fgq" podUID="33072424-e286-4580-a3b5-0ccf6b4cbcf8" Jul 23 09:17:00.979669 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:00.979624 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b7fmg" Jul 23 09:17:00.979835 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:00.979638 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b24sn" Jul 23 09:17:00.979835 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:00.979789 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b7fmg" podUID="381423db-deea-45d8-8cff-408186294f3e" Jul 23 09:17:00.979945 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:00.979831 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b24sn" podUID="82e553d7-cc99-4f29-99ba-1816621870c6" Jul 23 09:17:01.479843 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:01.479794 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/33072424-e286-4580-a3b5-0ccf6b4cbcf8-original-pull-secret\") pod \"global-pull-secret-syncer-l6fgq\" (UID: \"33072424-e286-4580-a3b5-0ccf6b4cbcf8\") " pod="kube-system/global-pull-secret-syncer-l6fgq" Jul 23 09:17:01.480349 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:01.479931 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 09:17:01.480349 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:01.479995 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33072424-e286-4580-a3b5-0ccf6b4cbcf8-original-pull-secret podName:33072424-e286-4580-a3b5-0ccf6b4cbcf8 nodeName:}" failed. No retries permitted until 2026-07-23 09:17:09.479975867 +0000 UTC m=+18.019254436 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/33072424-e286-4580-a3b5-0ccf6b4cbcf8-original-pull-secret") pod "global-pull-secret-syncer-l6fgq" (UID: "33072424-e286-4580-a3b5-0ccf6b4cbcf8") : object "kube-system"/"original-pull-secret" not registered Jul 23 09:17:01.580661 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:01.580568 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/381423db-deea-45d8-8cff-408186294f3e-metrics-certs\") pod \"network-metrics-daemon-b7fmg\" (UID: \"381423db-deea-45d8-8cff-408186294f3e\") " pod="openshift-multus/network-metrics-daemon-b7fmg" Jul 23 09:17:01.580913 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:01.580707 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 09:17:01.580913 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:01.580776 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/381423db-deea-45d8-8cff-408186294f3e-metrics-certs podName:381423db-deea-45d8-8cff-408186294f3e nodeName:}" failed. No retries permitted until 2026-07-23 09:17:09.58075629 +0000 UTC m=+18.120034857 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/381423db-deea-45d8-8cff-408186294f3e-metrics-certs") pod "network-metrics-daemon-b7fmg" (UID: "381423db-deea-45d8-8cff-408186294f3e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 09:17:01.681227 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:01.681187 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99lgf\" (UniqueName: \"kubernetes.io/projected/82e553d7-cc99-4f29-99ba-1816621870c6-kube-api-access-99lgf\") pod \"network-check-target-b24sn\" (UID: \"82e553d7-cc99-4f29-99ba-1816621870c6\") " pod="openshift-network-diagnostics/network-check-target-b24sn" Jul 23 09:17:01.681419 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:01.681395 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 09:17:01.681478 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:01.681428 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 09:17:01.681478 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:01.681441 2576 projected.go:196] Error preparing data for projected volume kube-api-access-99lgf for pod openshift-network-diagnostics/network-check-target-b24sn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 09:17:01.681570 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:01.681512 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/82e553d7-cc99-4f29-99ba-1816621870c6-kube-api-access-99lgf podName:82e553d7-cc99-4f29-99ba-1816621870c6 nodeName:}" failed. No retries permitted until 2026-07-23 09:17:09.681485441 +0000 UTC m=+18.220764019 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-99lgf" (UniqueName: "kubernetes.io/projected/82e553d7-cc99-4f29-99ba-1816621870c6-kube-api-access-99lgf") pod "network-check-target-b24sn" (UID: "82e553d7-cc99-4f29-99ba-1816621870c6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 09:17:01.980230 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:01.980194 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6fgq" Jul 23 09:17:01.980422 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:01.980366 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6fgq" podUID="33072424-e286-4580-a3b5-0ccf6b4cbcf8" Jul 23 09:17:02.981251 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:02.980628 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b7fmg" Jul 23 09:17:02.981251 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:02.980717 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b7fmg" podUID="381423db-deea-45d8-8cff-408186294f3e" Jul 23 09:17:02.981251 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:02.981054 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b24sn" Jul 23 09:17:02.981251 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:02.981152 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b24sn" podUID="82e553d7-cc99-4f29-99ba-1816621870c6" Jul 23 09:17:03.979545 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:03.979507 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6fgq" Jul 23 09:17:03.979730 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:03.979626 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6fgq" podUID="33072424-e286-4580-a3b5-0ccf6b4cbcf8" Jul 23 09:17:04.979812 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:04.979778 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b7fmg" Jul 23 09:17:04.980276 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:04.979922 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b7fmg" podUID="381423db-deea-45d8-8cff-408186294f3e" Jul 23 09:17:04.980276 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:04.979982 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b24sn" Jul 23 09:17:04.980276 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:04.980100 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b24sn" podUID="82e553d7-cc99-4f29-99ba-1816621870c6" Jul 23 09:17:05.979301 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:05.979212 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6fgq" Jul 23 09:17:05.979446 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:05.979341 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6fgq" podUID="33072424-e286-4580-a3b5-0ccf6b4cbcf8" Jul 23 09:17:06.980370 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:06.980330 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b7fmg" Jul 23 09:17:06.980853 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:06.980330 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b24sn" Jul 23 09:17:06.980853 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:06.980481 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b7fmg" podUID="381423db-deea-45d8-8cff-408186294f3e" Jul 23 09:17:06.980853 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:06.980554 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b24sn" podUID="82e553d7-cc99-4f29-99ba-1816621870c6" Jul 23 09:17:07.979701 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:07.979666 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6fgq" Jul 23 09:17:07.979895 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:07.979800 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6fgq" podUID="33072424-e286-4580-a3b5-0ccf6b4cbcf8" Jul 23 09:17:08.979732 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:08.979698 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b7fmg" Jul 23 09:17:08.980158 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:08.979701 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b24sn" Jul 23 09:17:08.980158 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:08.979840 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b7fmg" podUID="381423db-deea-45d8-8cff-408186294f3e" Jul 23 09:17:08.980158 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:08.979957 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b24sn" podUID="82e553d7-cc99-4f29-99ba-1816621870c6" Jul 23 09:17:09.538901 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:09.538866 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/33072424-e286-4580-a3b5-0ccf6b4cbcf8-original-pull-secret\") pod \"global-pull-secret-syncer-l6fgq\" (UID: \"33072424-e286-4580-a3b5-0ccf6b4cbcf8\") " pod="kube-system/global-pull-secret-syncer-l6fgq" Jul 23 09:17:09.539116 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:09.539026 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 09:17:09.539116 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:09.539102 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33072424-e286-4580-a3b5-0ccf6b4cbcf8-original-pull-secret podName:33072424-e286-4580-a3b5-0ccf6b4cbcf8 nodeName:}" failed. No retries permitted until 2026-07-23 09:17:25.5390824 +0000 UTC m=+34.078360957 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/33072424-e286-4580-a3b5-0ccf6b4cbcf8-original-pull-secret") pod "global-pull-secret-syncer-l6fgq" (UID: "33072424-e286-4580-a3b5-0ccf6b4cbcf8") : object "kube-system"/"original-pull-secret" not registered Jul 23 09:17:09.639572 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:09.639543 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/381423db-deea-45d8-8cff-408186294f3e-metrics-certs\") pod \"network-metrics-daemon-b7fmg\" (UID: \"381423db-deea-45d8-8cff-408186294f3e\") " pod="openshift-multus/network-metrics-daemon-b7fmg" Jul 23 09:17:09.639725 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:09.639697 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 09:17:09.639787 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:09.639769 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/381423db-deea-45d8-8cff-408186294f3e-metrics-certs podName:381423db-deea-45d8-8cff-408186294f3e nodeName:}" failed. No retries permitted until 2026-07-23 09:17:25.639750347 +0000 UTC m=+34.179028909 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/381423db-deea-45d8-8cff-408186294f3e-metrics-certs") pod "network-metrics-daemon-b7fmg" (UID: "381423db-deea-45d8-8cff-408186294f3e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 09:17:09.741035 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:09.740804 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99lgf\" (UniqueName: \"kubernetes.io/projected/82e553d7-cc99-4f29-99ba-1816621870c6-kube-api-access-99lgf\") pod \"network-check-target-b24sn\" (UID: \"82e553d7-cc99-4f29-99ba-1816621870c6\") " pod="openshift-network-diagnostics/network-check-target-b24sn" Jul 23 09:17:09.741208 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:09.740977 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 09:17:09.741208 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:09.741092 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 09:17:09.741208 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:09.741106 2576 projected.go:196] Error preparing data for projected volume kube-api-access-99lgf for pod openshift-network-diagnostics/network-check-target-b24sn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 09:17:09.741208 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:09.741165 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/82e553d7-cc99-4f29-99ba-1816621870c6-kube-api-access-99lgf podName:82e553d7-cc99-4f29-99ba-1816621870c6 nodeName:}" failed. No retries permitted until 2026-07-23 09:17:25.741151446 +0000 UTC m=+34.280430000 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-99lgf" (UniqueName: "kubernetes.io/projected/82e553d7-cc99-4f29-99ba-1816621870c6-kube-api-access-99lgf") pod "network-check-target-b24sn" (UID: "82e553d7-cc99-4f29-99ba-1816621870c6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 09:17:09.979708 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:09.979680 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6fgq" Jul 23 09:17:09.979914 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:09.979783 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6fgq" podUID="33072424-e286-4580-a3b5-0ccf6b4cbcf8" Jul 23 09:17:10.979557 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:10.979520 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b24sn" Jul 23 09:17:10.979748 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:10.979524 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b7fmg" Jul 23 09:17:10.979748 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:10.979663 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b24sn" podUID="82e553d7-cc99-4f29-99ba-1816621870c6" Jul 23 09:17:10.979872 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:10.979812 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b7fmg" podUID="381423db-deea-45d8-8cff-408186294f3e" Jul 23 09:17:11.979499 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:11.979460 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6fgq" Jul 23 09:17:11.979925 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:11.979578 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6fgq" podUID="33072424-e286-4580-a3b5-0ccf6b4cbcf8" Jul 23 09:17:12.980811 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:12.980780 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b7fmg" Jul 23 09:17:12.981289 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:12.980888 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b7fmg" podUID="381423db-deea-45d8-8cff-408186294f3e" Jul 23 09:17:12.981289 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:12.980942 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b24sn" Jul 23 09:17:12.981289 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:12.981039 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b24sn" podUID="82e553d7-cc99-4f29-99ba-1816621870c6" Jul 23 09:17:13.979853 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:13.979821 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6fgq" Jul 23 09:17:13.980031 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:13.979942 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6fgq" podUID="33072424-e286-4580-a3b5-0ccf6b4cbcf8" Jul 23 09:17:14.980028 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:14.979852 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b7fmg" Jul 23 09:17:14.980681 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:14.979890 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b24sn" Jul 23 09:17:14.980681 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:14.980129 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b7fmg" podUID="381423db-deea-45d8-8cff-408186294f3e" Jul 23 09:17:14.980681 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:14.980157 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b24sn" podUID="82e553d7-cc99-4f29-99ba-1816621870c6" Jul 23 09:17:15.175753 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:15.175707 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gvwsp" event={"ID":"5362d831-0050-47f8-acb1-fe06ddf24832","Type":"ContainerStarted","Data":"5acd122ac23e622bbc56102fa68b3e96500802c8e26c037b855150140ce54957"} Jul 23 09:17:15.175917 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:15.175795 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gvwsp" event={"ID":"5362d831-0050-47f8-acb1-fe06ddf24832","Type":"ContainerStarted","Data":"e00d39a584985c6fc98933c3cfce6464b0a4dbf49e110095aeeac4d5c6241658"} Jul 23 09:17:15.175917 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:15.175833 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gvwsp" event={"ID":"5362d831-0050-47f8-acb1-fe06ddf24832","Type":"ContainerStarted","Data":"3c47174a8e7d21dbdbd4f996d53cbb91237384b3f8de6efb6dd159be911b5a6c"} Jul 23 09:17:15.175917 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:15.175868 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gvwsp" event={"ID":"5362d831-0050-47f8-acb1-fe06ddf24832","Type":"ContainerStarted","Data":"41442297e8e50bceab8da1f066032001f75e100e8e753b8ef6a38efe78fa29aa"} Jul 23 09:17:15.175917 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:15.175897 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gvwsp" event={"ID":"5362d831-0050-47f8-acb1-fe06ddf24832","Type":"ContainerStarted","Data":"3d2cc5f6a76656c356650204b7084bc6d0a2cd80b5b57ff95c1a8f5b11b31b3e"} Jul 23 09:17:15.176122 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:15.175932 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gvwsp" event={"ID":"5362d831-0050-47f8-acb1-fe06ddf24832","Type":"ContainerStarted","Data":"2723008dbd9a96b8660a1185e8331adb72dacea1fb5de49b65a13d4fddb54774"} Jul 23 09:17:15.177393 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:15.177364 2576 generic.go:356] "Generic (PLEG): container finished" podID="95ff7554-5e90-41c1-9ab5-93d6120788fa" containerID="af5c98185167aefdbf0b3c4f5fe3bd0ec51e214a81e81c77cd92890353dd69ce" exitCode=0 Jul 23 09:17:15.177544 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:15.177506 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xgtzd" event={"ID":"95ff7554-5e90-41c1-9ab5-93d6120788fa","Type":"ContainerDied","Data":"af5c98185167aefdbf0b3c4f5fe3bd0ec51e214a81e81c77cd92890353dd69ce"} Jul 23 09:17:15.179233 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:15.179214 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4gw2w" event={"ID":"5fdaa22f-7fb2-42dd-8c10-2e422fa19df2","Type":"ContainerStarted","Data":"807b4648e0223196e9b3a3896aafc9ede1c766c8a2fd1116d70372b687506f68"} Jul 23 09:17:15.181272 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:15.181225 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9r4c5" event={"ID":"d676a11c-2214-4008-8801-efeb43be1ded","Type":"ContainerStarted","Data":"75fe04db19d20ed0486684bc43aa40fe508b0ab0bbe873456e018f82b5520cc2"} Jul 23 09:17:15.182847 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:15.182822 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drzcv" event={"ID":"7bccf2b8-bb0e-4371-bd49-0c889d86206f","Type":"ContainerStarted","Data":"8154fbdb18e7cf7e5c34fb4a3fd178ca9be42be51a254fa176c8884733b48010"} Jul 23 09:17:15.184384 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:15.184360 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-czm9w" event={"ID":"c4122a2d-66ca-4ec5-969b-14b89809eaa0","Type":"ContainerStarted","Data":"4d0aa13cc641fedcecef2196a6d22910ea28f5cc3f466a5ed0f674ef4f5e81fe"} Jul 23 09:17:15.186069 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:15.185904 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-145.ec2.internal" event={"ID":"6e4c6298602a592029f63da426ffeb44","Type":"ContainerStarted","Data":"6691de05c17fae7bc3b09644c94f85ca72a24a6ae409b56a236928ce87d44d94"} Jul 23 09:17:15.187425 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:15.187401 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gf7qk" event={"ID":"72bcfd4f-fce8-46b6-b3f1-52e2ba6306fa","Type":"ContainerStarted","Data":"c06353101fd17a4c6dc023f350365ac389775fac5acc58f6cacf1f1ac55a1a98"} Jul 23 09:17:15.188841 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:15.188812 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8zd5l" event={"ID":"ba46e6f4-7695-4578-89a5-c95d6a27890a","Type":"ContainerStarted","Data":"925ece8a00e5b745cac3e6b9ff07058e48b40ce7aae017774ff51681cf3ebae9"} Jul 23 09:17:15.190489 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:15.190465 2576 generic.go:356] "Generic (PLEG): container finished" podID="00d27bbe0e5e208cc7ea070ef5013896" containerID="780cad3927fc1450064e891f8870c0318369687971df9f6664afebc8ade32eb5" exitCode=0 Jul 23 09:17:15.190568 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:15.190541 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-145.ec2.internal" event={"ID":"00d27bbe0e5e208cc7ea070ef5013896","Type":"ContainerDied","Data":"780cad3927fc1450064e891f8870c0318369687971df9f6664afebc8ade32eb5"} Jul 23 09:17:15.294670 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:15.294618 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-9r4c5" podStartSLOduration=2.4813366869999998 podStartE2EDuration="22.294597497s" totalImagesPullingTime="19.81326081s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 09:16:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:17:15.267153921 +0000 UTC m=+23.806432476" watchObservedRunningTime="2026-07-23 09:17:15.294597497 +0000 UTC m=+23.833876075" Jul 23 09:17:15.367322 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:15.367205 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-gf7qk" podStartSLOduration=2.562293018 podStartE2EDuration="22.367185272s" totalImagesPullingTime="19.804892254s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 09:16:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:17:15.328005622 +0000 UTC m=+23.867284199" watchObservedRunningTime="2026-07-23 09:17:15.367185272 +0000 UTC m=+23.906463852" Jul 23 09:17:15.394578 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:15.394526 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-4gw2w" podStartSLOduration=2.452819381 podStartE2EDuration="22.394505343s" totalImagesPullingTime="19.941685962s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 09:16:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:17:15.366981199 +0000 UTC m=+23.906259776" watchObservedRunningTime="2026-07-23 09:17:15.394505343 +0000 UTC m=+23.933783921" Jul 23 09:17:15.395129 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:15.395098 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-145.ec2.internal" podStartSLOduration=22.395088955 podStartE2EDuration="22.395088955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 09:16:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:17:15.394704772 +0000 UTC m=+23.933983349" watchObservedRunningTime="2026-07-23 09:17:15.395088955 +0000 UTC m=+23.934367533" Jul 23 09:17:15.464327 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:15.464276 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-czm9w" podStartSLOduration=2.605462962 podStartE2EDuration="22.464259958s" totalImagesPullingTime="19.858796996s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:16:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:17:15.426793175 +0000 UTC m=+23.966071753" watchObservedRunningTime="2026-07-23 09:17:15.464259958 +0000 UTC m=+24.003538536" Jul 23 09:17:15.464464 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:15.464449 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-8zd5l" podStartSLOduration=2.5541535140000002 podStartE2EDuration="22.464441959s" totalImagesPullingTime="19.910288445s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 09:16:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:17:15.464396889 +0000 UTC m=+24.003675446" watchObservedRunningTime="2026-07-23 09:17:15.464441959 +0000 UTC m=+24.003720552" Jul 23 09:17:15.931494 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:15.931458 2576 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 23 09:17:15.979872 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:15.979841 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6fgq" Jul 23 09:17:15.979990 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:15.979953 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6fgq" podUID="33072424-e286-4580-a3b5-0ccf6b4cbcf8" Jul 23 09:17:16.195380 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:16.195341 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-145.ec2.internal" event={"ID":"00d27bbe0e5e208cc7ea070ef5013896","Type":"ContainerStarted","Data":"bd6e35134d1e4f48aeeb0eef56b8720028b24f4813aa8bbbc376bc58cc7c3d84"} Jul 23 09:17:16.197121 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:16.197087 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-z2nbl" event={"ID":"34d12a5c-a8ee-40d7-a365-14d8ef2d94ef","Type":"ContainerStarted","Data":"9c3f7d40211c94f1da86d1d3dcf6c893420d3c15bd7dd929a5de2a13fff227d0"} Jul 23 09:17:16.199000 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:16.198974 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drzcv" event={"ID":"7bccf2b8-bb0e-4371-bd49-0c889d86206f","Type":"ContainerStarted","Data":"50356e6f4ee1f494d55b06a23174ce5a7ae416213765826ea393380c1f8e7d0b"} Jul 23 09:17:16.258570 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:16.258520 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-145.ec2.internal" podStartSLOduration=23.258505256 podStartE2EDuration="23.258505256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 09:16:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:17:16.221522439 +0000 UTC m=+24.760801016" watchObservedRunningTime="2026-07-23 09:17:16.258505256 +0000 UTC m=+24.797783832" Jul 23 09:17:16.887578 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:16.887445 2576 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-23T09:17:15.931484755Z","UUID":"6597e32b-3d4f-443c-bf5d-108355724a11","Handler":null,"Name":"","Endpoint":""} Jul 23 09:17:16.889091 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:16.889068 2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 23 09:17:16.889185 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:16.889106 2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 23 09:17:16.979779 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:16.979751 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b7fmg" Jul 23 09:17:16.979911 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:16.979873 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b7fmg" podUID="381423db-deea-45d8-8cff-408186294f3e" Jul 23 09:17:16.979977 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:16.979915 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b24sn" Jul 23 09:17:16.980031 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:16.980001 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b24sn" podUID="82e553d7-cc99-4f29-99ba-1816621870c6" Jul 23 09:17:17.203303 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:17.203275 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drzcv" event={"ID":"7bccf2b8-bb0e-4371-bd49-0c889d86206f","Type":"ContainerStarted","Data":"f80ad0791fc55daee1582ded193ce4bf194e28376ed110072221cae5d510f659"} Jul 23 09:17:17.206490 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:17.206461 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gvwsp" event={"ID":"5362d831-0050-47f8-acb1-fe06ddf24832","Type":"ContainerStarted","Data":"5e21e06a1524eb88d344f68224dcc391eda1df0a6edc3fdeb91efe7dabe96244"} Jul 23 09:17:17.231190 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:17.231144 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-z2nbl" podStartSLOduration=4.409547317 podStartE2EDuration="24.231129594s" totalImagesPullingTime="19.821582277s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 09:16:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:17:16.259284678 +0000 UTC m=+24.798563254" watchObservedRunningTime="2026-07-23 09:17:17.231129594 +0000 UTC m=+25.770408173" Jul 23 09:17:17.979573 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:17.979544 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6fgq" Jul 23 09:17:17.979796 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:17.979636 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6fgq" podUID="33072424-e286-4580-a3b5-0ccf6b4cbcf8" Jul 23 09:17:18.077738 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:18.077706 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-czm9w" Jul 23 09:17:18.078400 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:18.078377 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-czm9w" Jul 23 09:17:18.098235 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:18.098186 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drzcv" podStartSLOduration=2.78338849 podStartE2EDuration="25.098168513s" totalImagesPullingTime="22.314780023s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 09:16:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:17:17.231310716 +0000 UTC m=+25.770589294" watchObservedRunningTime="2026-07-23 09:17:18.098168513 +0000 UTC m=+26.637447093" Jul 23 09:17:18.209044 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:18.209008 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-czm9w" Jul 23 09:17:18.209607 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:18.209583 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-czm9w" Jul 23 09:17:18.980281 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:18.980234 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b7fmg" Jul 23 09:17:18.980447 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:18.980260 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b24sn" Jul 23 09:17:18.980447 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:18.980404 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b7fmg" podUID="381423db-deea-45d8-8cff-408186294f3e" Jul 23 09:17:18.980527 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:18.980442 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b24sn" podUID="82e553d7-cc99-4f29-99ba-1816621870c6" Jul 23 09:17:19.980100 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:19.979917 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6fgq" Jul 23 09:17:19.980727 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:19.980199 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6fgq" podUID="33072424-e286-4580-a3b5-0ccf6b4cbcf8" Jul 23 09:17:20.217175 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:20.216155 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gvwsp" event={"ID":"5362d831-0050-47f8-acb1-fe06ddf24832","Type":"ContainerStarted","Data":"13301ded49205933227d29380d1bb79bf348aacd984a6f64273b3808e393d929"} Jul 23 09:17:20.217365 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:20.217262 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gvwsp" Jul 23 09:17:20.217365 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:20.217321 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gvwsp" Jul 23 09:17:20.217365 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:20.217358 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gvwsp" Jul 23 09:17:20.220003 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:20.219978 2576 generic.go:356] "Generic (PLEG): container finished" podID="95ff7554-5e90-41c1-9ab5-93d6120788fa" containerID="bea1ca3f8cc7bf6b1394cd9e4774027fc3a3ceb2cb7e9bd1852beb2f594d8f26" exitCode=0 Jul 23 09:17:20.220119 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:20.220041 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xgtzd" event={"ID":"95ff7554-5e90-41c1-9ab5-93d6120788fa","Type":"ContainerDied","Data":"bea1ca3f8cc7bf6b1394cd9e4774027fc3a3ceb2cb7e9bd1852beb2f594d8f26"} Jul 23 09:17:20.233403 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:20.233380 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gvwsp" Jul 23 09:17:20.233815 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:20.233796 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gvwsp" Jul 23 09:17:20.245775 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:20.245737 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-gvwsp" podStartSLOduration=7.427990613 podStartE2EDuration="27.245724887s" totalImagesPullingTime="19.817734274s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 09:16:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:17:20.244653291 +0000 UTC m=+28.783931871" watchObservedRunningTime="2026-07-23 09:17:20.245724887 +0000 UTC m=+28.785003463" Jul 23 09:17:20.979735 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:20.979704 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b7fmg" Jul 23 09:17:20.979950 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:20.979704 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b24sn" Jul 23 09:17:20.979950 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:20.979823 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b7fmg" podUID="381423db-deea-45d8-8cff-408186294f3e" Jul 23 09:17:20.979950 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:20.979875 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b24sn" podUID="82e553d7-cc99-4f29-99ba-1816621870c6" Jul 23 09:17:21.550550 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:21.548395 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-l6fgq"] Jul 23 09:17:21.550550 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:21.548563 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6fgq" Jul 23 09:17:21.550550 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:21.548690 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6fgq" podUID="33072424-e286-4580-a3b5-0ccf6b4cbcf8" Jul 23 09:17:21.551982 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:21.551942 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-b7fmg"] Jul 23 09:17:21.552101 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:21.552064 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b7fmg" Jul 23 09:17:21.552173 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:21.552154 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b7fmg" podUID="381423db-deea-45d8-8cff-408186294f3e" Jul 23 09:17:21.552619 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:21.552595 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-b24sn"] Jul 23 09:17:21.552736 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:21.552686 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b24sn" Jul 23 09:17:21.552803 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:21.552775 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b24sn" podUID="82e553d7-cc99-4f29-99ba-1816621870c6" Jul 23 09:17:22.225371 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:22.225337 2576 generic.go:356] "Generic (PLEG): container finished" podID="95ff7554-5e90-41c1-9ab5-93d6120788fa" containerID="b7359a3fd52d04dfb66e5f8fb69521e1614e09db649a1f213dc1acaf97691814" exitCode=0 Jul 23 09:17:22.225536 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:22.225420 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xgtzd" event={"ID":"95ff7554-5e90-41c1-9ab5-93d6120788fa","Type":"ContainerDied","Data":"b7359a3fd52d04dfb66e5f8fb69521e1614e09db649a1f213dc1acaf97691814"} Jul 23 09:17:22.980093 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:22.980058 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6fgq" Jul 23 09:17:22.980494 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:22.980171 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6fgq" podUID="33072424-e286-4580-a3b5-0ccf6b4cbcf8" Jul 23 09:17:22.980494 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:22.980249 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b24sn" Jul 23 09:17:22.980494 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:22.980353 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b24sn" podUID="82e553d7-cc99-4f29-99ba-1816621870c6" Jul 23 09:17:22.980494 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:22.980397 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b7fmg" Jul 23 09:17:22.980494 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:22.980479 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b7fmg" podUID="381423db-deea-45d8-8cff-408186294f3e" Jul 23 09:17:24.235893 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:24.235844 2576 generic.go:356] "Generic (PLEG): container finished" podID="95ff7554-5e90-41c1-9ab5-93d6120788fa" containerID="75cfd8ee6ef181104f6667000e72bca3bbe919c1d21d905b83a8ebac2d3f0125" exitCode=0 Jul 23 09:17:24.236464 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:24.235895 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xgtzd" event={"ID":"95ff7554-5e90-41c1-9ab5-93d6120788fa","Type":"ContainerDied","Data":"75cfd8ee6ef181104f6667000e72bca3bbe919c1d21d905b83a8ebac2d3f0125"} Jul 23 09:17:24.982948 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:24.982917 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b24sn" Jul 23 09:17:24.983127 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:24.983025 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6fgq" Jul 23 09:17:24.983127 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:24.983031 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b24sn" podUID="82e553d7-cc99-4f29-99ba-1816621870c6" Jul 23 09:17:24.983127 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:24.983060 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b7fmg" Jul 23 09:17:24.983267 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:24.983128 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6fgq" podUID="33072424-e286-4580-a3b5-0ccf6b4cbcf8" Jul 23 09:17:24.983267 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:24.983220 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b7fmg" podUID="381423db-deea-45d8-8cff-408186294f3e" Jul 23 09:17:25.564191 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:25.564100 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/33072424-e286-4580-a3b5-0ccf6b4cbcf8-original-pull-secret\") pod \"global-pull-secret-syncer-l6fgq\" (UID: \"33072424-e286-4580-a3b5-0ccf6b4cbcf8\") " pod="kube-system/global-pull-secret-syncer-l6fgq" Jul 23 09:17:25.564639 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:25.564227 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 09:17:25.564639 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:25.564316 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33072424-e286-4580-a3b5-0ccf6b4cbcf8-original-pull-secret podName:33072424-e286-4580-a3b5-0ccf6b4cbcf8 nodeName:}" failed. No retries permitted until 2026-07-23 09:17:57.564300305 +0000 UTC m=+66.103578864 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/33072424-e286-4580-a3b5-0ccf6b4cbcf8-original-pull-secret") pod "global-pull-secret-syncer-l6fgq" (UID: "33072424-e286-4580-a3b5-0ccf6b4cbcf8") : object "kube-system"/"original-pull-secret" not registered Jul 23 09:17:25.665232 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:25.665196 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/381423db-deea-45d8-8cff-408186294f3e-metrics-certs\") pod \"network-metrics-daemon-b7fmg\" (UID: \"381423db-deea-45d8-8cff-408186294f3e\") " pod="openshift-multus/network-metrics-daemon-b7fmg" Jul 23 09:17:25.665421 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:25.665399 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 09:17:25.665492 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:25.665482 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/381423db-deea-45d8-8cff-408186294f3e-metrics-certs podName:381423db-deea-45d8-8cff-408186294f3e nodeName:}" failed. No retries permitted until 2026-07-23 09:17:57.665460479 +0000 UTC m=+66.204739039 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/381423db-deea-45d8-8cff-408186294f3e-metrics-certs") pod "network-metrics-daemon-b7fmg" (UID: "381423db-deea-45d8-8cff-408186294f3e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 09:17:25.766557 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:25.766525 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99lgf\" (UniqueName: \"kubernetes.io/projected/82e553d7-cc99-4f29-99ba-1816621870c6-kube-api-access-99lgf\") pod \"network-check-target-b24sn\" (UID: \"82e553d7-cc99-4f29-99ba-1816621870c6\") " pod="openshift-network-diagnostics/network-check-target-b24sn" Jul 23 09:17:25.766740 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:25.766672 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 09:17:25.766740 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:25.766692 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 09:17:25.766740 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:25.766703 2576 projected.go:196] Error preparing data for projected volume kube-api-access-99lgf for pod openshift-network-diagnostics/network-check-target-b24sn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 09:17:25.766905 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:25.766762 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/82e553d7-cc99-4f29-99ba-1816621870c6-kube-api-access-99lgf podName:82e553d7-cc99-4f29-99ba-1816621870c6 nodeName:}" failed. No retries permitted until 2026-07-23 09:17:57.766744676 +0000 UTC m=+66.306023232 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-99lgf" (UniqueName: "kubernetes.io/projected/82e553d7-cc99-4f29-99ba-1816621870c6-kube-api-access-99lgf") pod "network-check-target-b24sn" (UID: "82e553d7-cc99-4f29-99ba-1816621870c6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 09:17:26.803965 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.803934 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-145.ec2.internal" event="NodeReady" Jul 23 09:17:26.804380 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.804085 2576 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 23 09:17:26.854903 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.854876 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7989bd97d7-g2rdd"] Jul 23 09:17:26.857435 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.857375 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7989bd97d7-g2rdd" Jul 23 09:17:26.858905 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.858880 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-555d9d7885-pxlst"] Jul 23 09:17:26.860538 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.860515 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-5szxg"] Jul 23 09:17:26.860648 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.860595 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-pxlst" Jul 23 09:17:26.862810 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.862789 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8zbdn"] Jul 23 09:17:26.863377 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.863357 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-5szxg" Jul 23 09:17:26.868270 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:26.867438 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-135-145.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-console-operator\": no relationship found between node 'ip-10-0-135-145.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" Jul 23 09:17:26.868270 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.867569 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-sp78t"] Jul 23 09:17:26.868270 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.867910 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8zbdn" Jul 23 09:17:26.871286 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.871264 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7494768586-7px6v"] Jul 23 09:17:26.871399 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.871320 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-sp78t" Jul 23 09:17:26.873859 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:26.873810 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"console-operator-dockercfg-w58fm\" is forbidden: User \"system:node:ip-10-0-135-145.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-console-operator\": no relationship found between node 'ip-10-0-135-145.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-w58fm\"" type="*v1.Secret" Jul 23 09:17:26.873973 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.873934 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-nfv7r"] Jul 23 09:17:26.873973 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:26.873934 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"console-operator-config\" is forbidden: User \"system:node:ip-10-0-135-145.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-console-operator\": no relationship found between node 'ip-10-0-135-145.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" type="*v1.ConfigMap" Jul 23 09:17:26.874093 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:26.873947 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"trusted-ca\" is forbidden: User \"system:node:ip-10-0-135-145.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-console-operator\": no relationship found between node 'ip-10-0-135-145.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" type="*v1.ConfigMap" Jul 23 09:17:26.874093 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:26.873998 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"serving-cert\" is forbidden: User \"system:node:ip-10-0-135-145.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-console-operator\": no relationship found between node 'ip-10-0-135-145.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" type="*v1.Secret" Jul 23 09:17:26.874192 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.874131 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7494768586-7px6v" Jul 23 09:17:26.874735 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:26.874711 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"cluster-monitoring-operator-dockercfg-xqgv5\" is forbidden: User \"system:node:ip-10-0-135-145.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-135-145.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-xqgv5\"" type="*v1.Secret" Jul 23 09:17:26.874831 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:26.874711 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"cluster-monitoring-operator-tls\" is forbidden: User \"system:node:ip-10-0-135-145.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-135-145.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" type="*v1.Secret" Jul 23 09:17:26.874831 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:26.874757 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"telemetry-config\" is forbidden: User \"system:node:ip-10-0-135-145.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-135-145.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" type="*v1.ConfigMap" Jul 23 09:17:26.875425 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.875403 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 23 09:17:26.875534 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:26.875450 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"serving-cert\" is forbidden: User \"system:node:ip-10-0-135-145.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-service-ca-operator\": no relationship found between node 'ip-10-0-135-145.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" type="*v1.Secret" Jul 23 09:17:26.875534 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:26.875400 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-135-145.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-135-145.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" Jul 23 09:17:26.875657 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:26.875564 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-135-145.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-service-ca-operator\": no relationship found between node 'ip-10-0-135-145.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" Jul 23 09:17:26.875727 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.875712 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-nfv7r" Jul 23 09:17:26.875817 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:26.875793 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-135-145.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-135-145.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" Jul 23 09:17:26.876084 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:26.875986 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"service-ca-operator-dockercfg-5g67b\" is forbidden: User \"system:node:ip-10-0-135-145.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-service-ca-operator\": no relationship found between node 'ip-10-0-135-145.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-5g67b\"" type="*v1.Secret" Jul 23 09:17:26.876202 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:26.876091 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"default-ingress-cert\" is forbidden: User \"system:node:ip-10-0-135-145.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-135-145.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" type="*v1.Secret" Jul 23 09:17:26.876202 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:26.876161 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-135-145.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-service-ca-operator\": no relationship found between node 'ip-10-0-135-145.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" Jul 23 09:17:26.876202 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:26.876171 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"service-ca-operator-config\" is forbidden: User \"system:node:ip-10-0-135-145.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-service-ca-operator\": no relationship found between node 'ip-10-0-135-145.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" type="*v1.ConfigMap" Jul 23 09:17:26.876414 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.876258 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 23 09:17:26.877780 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:26.877753 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"router-stats-default\" is forbidden: User \"system:node:ip-10-0-135-145.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-135-145.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" type="*v1.Secret" Jul 23 09:17:26.877920 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:26.877891 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"service-ca-bundle\" is forbidden: User \"system:node:ip-10-0-135-145.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-135-145.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" type="*v1.ConfigMap" Jul 23 09:17:26.878037 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:26.878017 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-135-145.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-135-145.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" Jul 23 09:17:26.879287 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:26.879263 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"router-metrics-certs-default\" is forbidden: User \"system:node:ip-10-0-135-145.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-135-145.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" type="*v1.Secret" Jul 23 09:17:26.879408 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:26.879386 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"router-dockercfg-4dsz5\" is forbidden: User \"system:node:ip-10-0-135-145.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-135-145.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"router-dockercfg-4dsz5\"" type="*v1.Secret" Jul 23 09:17:26.880000 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.879983 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-q5wx9\"" Jul 23 09:17:26.880086 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:26.879980 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"serving-cert\" is forbidden: User \"system:node:ip-10-0-135-145.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-kube-storage-version-migrator-operator\": no relationship found between node 'ip-10-0-135-145.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" type="*v1.Secret" Jul 23 09:17:26.881371 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.881307 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 23 09:17:26.881552 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.881530 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 23 09:17:26.881872 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.881852 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 23 09:17:26.883211 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.883160 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 23 09:17:26.883339 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.883322 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-cqw2m\"" Jul 23 09:17:26.883428 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:26.883406 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"config\" is forbidden: User \"system:node:ip-10-0-135-145.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-kube-storage-version-migrator-operator\": no relationship found between node 'ip-10-0-135-145.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" type="*v1.ConfigMap" Jul 23 09:17:26.883599 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:26.883561 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-135-145.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-135-145.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" Jul 23 09:17:26.884026 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:26.883999 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-135-145.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-kube-storage-version-migrator-operator\": no relationship found between node 'ip-10-0-135-145.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" Jul 23 09:17:26.884703 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.884685 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 23 09:17:26.885991 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:26.885962 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"kube-storage-version-migrator-operator-dockercfg-vjvsg\" is forbidden: User \"system:node:ip-10-0-135-145.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-kube-storage-version-migrator-operator\": no relationship found between node 'ip-10-0-135-145.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-vjvsg\"" type="*v1.Secret" Jul 23 09:17:26.888185 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.888165 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 23 09:17:26.888312 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:26.888277 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-135-145.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-kube-storage-version-migrator-operator\": no relationship found between node 'ip-10-0-135-145.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" Jul 23 09:17:26.894927 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.894906 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 23 09:17:26.895553 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.895534 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 23 09:17:26.898042 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.898023 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-5szxg"] Jul 23 09:17:26.898705 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.898683 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7989bd97d7-g2rdd"] Jul 23 09:17:26.900273 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.900252 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-sp78t"] Jul 23 09:17:26.900374 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.900290 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8zbdn"] Jul 23 09:17:26.905699 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.905679 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7494768586-7px6v"] Jul 23 09:17:26.906136 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.906118 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-k296x"] Jul 23 09:17:26.908732 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.908713 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-pxlst"] Jul 23 09:17:26.908813 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.908750 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-nfv7r"] Jul 23 09:17:26.908887 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.908825 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-k296x" Jul 23 09:17:26.920925 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.920905 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 23 09:17:26.921319 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.921302 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 23 09:17:26.921536 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.921386 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-xtr4d"] Jul 23 09:17:26.923798 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.923783 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-wbd74\"" Jul 23 09:17:26.925050 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.925016 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-mgk4m"] Jul 23 09:17:26.925168 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.925148 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-xtr4d" Jul 23 09:17:26.926977 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.926956 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-b5cz2"] Jul 23 09:17:26.927293 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.927273 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mgk4m" Jul 23 09:17:26.928337 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.928314 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 23 09:17:26.928429 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.928338 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 23 09:17:26.928429 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.928323 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 23 09:17:26.929992 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.929972 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-k296x"] Jul 23 09:17:26.930192 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.930178 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-b5cz2" Jul 23 09:17:26.932106 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.932085 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 23 09:17:26.932287 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.932271 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 23 09:17:26.932513 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.932497 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-x59bn\"" Jul 23 09:17:26.934472 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.934452 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-p8qpt\"" Jul 23 09:17:26.935277 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.935252 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 23 09:17:26.935368 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.935263 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 23 09:17:26.935553 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.935532 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-hmp5k\"" Jul 23 09:17:26.942489 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.942459 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mgk4m"] Jul 23 09:17:26.944694 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.944336 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-xtr4d"] Jul 23 09:17:26.945850 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.945813 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-b5cz2"] Jul 23 09:17:26.976103 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.976074 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1260ceaa-4aa5-4130-9ea7-ff8212c2d751-installation-pull-secrets\") pod \"image-registry-7989bd97d7-g2rdd\" (UID: \"1260ceaa-4aa5-4130-9ea7-ff8212c2d751\") " pod="openshift-image-registry/image-registry-7989bd97d7-g2rdd" Jul 23 09:17:26.976302 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.976283 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/59ee07ce-4747-496f-9876-81f2c2358981-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-8zbdn\" (UID: \"59ee07ce-4747-496f-9876-81f2c2358981\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8zbdn" Jul 23 09:17:26.976384 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.976355 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st24m\" (UniqueName: \"kubernetes.io/projected/b5c91d72-2e17-44e7-83ab-4b9ce3a89b43-kube-api-access-st24m\") pod \"service-ca-operator-5bcbc94c78-sp78t\" (UID: \"b5c91d72-2e17-44e7-83ab-4b9ce3a89b43\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-sp78t" Jul 23 09:17:26.976439 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.976401 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1260ceaa-4aa5-4130-9ea7-ff8212c2d751-image-registry-private-configuration\") pod \"image-registry-7989bd97d7-g2rdd\" (UID: \"1260ceaa-4aa5-4130-9ea7-ff8212c2d751\") " pod="openshift-image-registry/image-registry-7989bd97d7-g2rdd" Jul 23 09:17:26.976439 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.976420 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/989625f0-3f97-4f66-8d67-2e8f47a13a6c-serving-cert\") pod \"insights-operator-555d9d7885-pxlst\" (UID: \"989625f0-3f97-4f66-8d67-2e8f47a13a6c\") " pod="openshift-insights/insights-operator-555d9d7885-pxlst" Jul 23 09:17:26.976580 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.976439 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4908aff3-196e-44fa-b8e6-c0e81b476433-serving-cert\") pod \"console-operator-565b7b569d-5szxg\" (UID: \"4908aff3-196e-44fa-b8e6-c0e81b476433\") " pod="openshift-console-operator/console-operator-565b7b569d-5szxg" Jul 23 09:17:26.976580 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.976453 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bddaeac-9272-4031-b216-a31fbe58c89d-metrics-certs\") pod \"router-default-7494768586-7px6v\" (UID: \"6bddaeac-9272-4031-b216-a31fbe58c89d\") " pod="openshift-ingress/router-default-7494768586-7px6v" Jul 23 09:17:26.976580 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.976468 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1260ceaa-4aa5-4130-9ea7-ff8212c2d751-bound-sa-token\") pod \"image-registry-7989bd97d7-g2rdd\" (UID: \"1260ceaa-4aa5-4130-9ea7-ff8212c2d751\") " pod="openshift-image-registry/image-registry-7989bd97d7-g2rdd" Jul 23 09:17:26.976580 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.976505 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ef3505b4-c02c-487a-8314-f6793087ad3a-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-nfv7r\" (UID: \"ef3505b4-c02c-487a-8314-f6793087ad3a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-nfv7r" Jul 23 09:17:26.976580 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.976571 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1260ceaa-4aa5-4130-9ea7-ff8212c2d751-trusted-ca\") pod \"image-registry-7989bd97d7-g2rdd\" (UID: \"1260ceaa-4aa5-4130-9ea7-ff8212c2d751\") " pod="openshift-image-registry/image-registry-7989bd97d7-g2rdd" Jul 23 09:17:26.976806 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.976600 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/989625f0-3f97-4f66-8d67-2e8f47a13a6c-service-ca-bundle\") pod \"insights-operator-555d9d7885-pxlst\" (UID: \"989625f0-3f97-4f66-8d67-2e8f47a13a6c\") " pod="openshift-insights/insights-operator-555d9d7885-pxlst" Jul 23 09:17:26.976806 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.976627 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ef3505b4-c02c-487a-8314-f6793087ad3a-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-nfv7r\" (UID: \"ef3505b4-c02c-487a-8314-f6793087ad3a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-nfv7r" Jul 23 09:17:26.976806 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.976674 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4908aff3-196e-44fa-b8e6-c0e81b476433-trusted-ca\") pod \"console-operator-565b7b569d-5szxg\" (UID: \"4908aff3-196e-44fa-b8e6-c0e81b476433\") " pod="openshift-console-operator/console-operator-565b7b569d-5szxg" Jul 23 09:17:26.976806 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.976760 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1260ceaa-4aa5-4130-9ea7-ff8212c2d751-registry-tls\") pod \"image-registry-7989bd97d7-g2rdd\" (UID: \"1260ceaa-4aa5-4130-9ea7-ff8212c2d751\") " pod="openshift-image-registry/image-registry-7989bd97d7-g2rdd" Jul 23 09:17:26.976980 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.976814 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b5c91d72-2e17-44e7-83ab-4b9ce3a89b43-config\") pod \"service-ca-operator-5bcbc94c78-sp78t\" (UID: \"b5c91d72-2e17-44e7-83ab-4b9ce3a89b43\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-sp78t" Jul 23 09:17:26.976980 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.976842 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vvmp\" (UniqueName: \"kubernetes.io/projected/1260ceaa-4aa5-4130-9ea7-ff8212c2d751-kube-api-access-6vvmp\") pod \"image-registry-7989bd97d7-g2rdd\" (UID: \"1260ceaa-4aa5-4130-9ea7-ff8212c2d751\") " pod="openshift-image-registry/image-registry-7989bd97d7-g2rdd" Jul 23 09:17:26.976980 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.976867 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp4tw\" (UniqueName: \"kubernetes.io/projected/59ee07ce-4747-496f-9876-81f2c2358981-kube-api-access-tp4tw\") pod \"cluster-monitoring-operator-76cf8966f8-8zbdn\" (UID: \"59ee07ce-4747-496f-9876-81f2c2358981\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8zbdn" Jul 23 09:17:26.976980 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.976893 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1260ceaa-4aa5-4130-9ea7-ff8212c2d751-ca-trust-extracted\") pod \"image-registry-7989bd97d7-g2rdd\" (UID: \"1260ceaa-4aa5-4130-9ea7-ff8212c2d751\") " pod="openshift-image-registry/image-registry-7989bd97d7-g2rdd" Jul 23 09:17:26.976980 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.976922 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/989625f0-3f97-4f66-8d67-2e8f47a13a6c-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-pxlst\" (UID: \"989625f0-3f97-4f66-8d67-2e8f47a13a6c\") " pod="openshift-insights/insights-operator-555d9d7885-pxlst" Jul 23 09:17:26.976980 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.976946 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4908aff3-196e-44fa-b8e6-c0e81b476433-config\") pod \"console-operator-565b7b569d-5szxg\" (UID: \"4908aff3-196e-44fa-b8e6-c0e81b476433\") " pod="openshift-console-operator/console-operator-565b7b569d-5szxg" Jul 23 09:17:26.976980 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.976968 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqj8w\" (UniqueName: \"kubernetes.io/projected/4908aff3-196e-44fa-b8e6-c0e81b476433-kube-api-access-kqj8w\") pod \"console-operator-565b7b569d-5szxg\" (UID: \"4908aff3-196e-44fa-b8e6-c0e81b476433\") " pod="openshift-console-operator/console-operator-565b7b569d-5szxg" Jul 23 09:17:26.977295 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.976993 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6bddaeac-9272-4031-b216-a31fbe58c89d-stats-auth\") pod \"router-default-7494768586-7px6v\" (UID: \"6bddaeac-9272-4031-b216-a31fbe58c89d\") " pod="openshift-ingress/router-default-7494768586-7px6v" Jul 23 09:17:26.977295 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.977071 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/989625f0-3f97-4f66-8d67-2e8f47a13a6c-snapshots\") pod \"insights-operator-555d9d7885-pxlst\" (UID: \"989625f0-3f97-4f66-8d67-2e8f47a13a6c\") " pod="openshift-insights/insights-operator-555d9d7885-pxlst" Jul 23 09:17:26.977295 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.977153 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxmxd\" (UniqueName: \"kubernetes.io/projected/ef3505b4-c02c-487a-8314-f6793087ad3a-kube-api-access-vxmxd\") pod \"kube-storage-version-migrator-operator-7658d9b749-nfv7r\" (UID: \"ef3505b4-c02c-487a-8314-f6793087ad3a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-nfv7r" Jul 23 09:17:26.977295 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.977187 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1260ceaa-4aa5-4130-9ea7-ff8212c2d751-registry-certificates\") pod \"image-registry-7989bd97d7-g2rdd\" (UID: \"1260ceaa-4aa5-4130-9ea7-ff8212c2d751\") " pod="openshift-image-registry/image-registry-7989bd97d7-g2rdd" Jul 23 09:17:26.977295 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.977214 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/989625f0-3f97-4f66-8d67-2e8f47a13a6c-tmp\") pod \"insights-operator-555d9d7885-pxlst\" (UID: \"989625f0-3f97-4f66-8d67-2e8f47a13a6c\") " pod="openshift-insights/insights-operator-555d9d7885-pxlst" Jul 23 09:17:26.977295 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.977250 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b5c91d72-2e17-44e7-83ab-4b9ce3a89b43-serving-cert\") pod \"service-ca-operator-5bcbc94c78-sp78t\" (UID: \"b5c91d72-2e17-44e7-83ab-4b9ce3a89b43\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-sp78t" Jul 23 09:17:26.977295 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.977281 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6bddaeac-9272-4031-b216-a31fbe58c89d-default-certificate\") pod \"router-default-7494768586-7px6v\" (UID: \"6bddaeac-9272-4031-b216-a31fbe58c89d\") " pod="openshift-ingress/router-default-7494768586-7px6v" Jul 23 09:17:26.977615 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.977306 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h47rf\" (UniqueName: \"kubernetes.io/projected/6bddaeac-9272-4031-b216-a31fbe58c89d-kube-api-access-h47rf\") pod \"router-default-7494768586-7px6v\" (UID: \"6bddaeac-9272-4031-b216-a31fbe58c89d\") " pod="openshift-ingress/router-default-7494768586-7px6v" Jul 23 09:17:26.977615 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.977342 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqpjk\" (UniqueName: \"kubernetes.io/projected/989625f0-3f97-4f66-8d67-2e8f47a13a6c-kube-api-access-rqpjk\") pod \"insights-operator-555d9d7885-pxlst\" (UID: \"989625f0-3f97-4f66-8d67-2e8f47a13a6c\") " pod="openshift-insights/insights-operator-555d9d7885-pxlst" Jul 23 09:17:26.977615 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.977416 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/59ee07ce-4747-496f-9876-81f2c2358981-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-8zbdn\" (UID: \"59ee07ce-4747-496f-9876-81f2c2358981\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8zbdn" Jul 23 09:17:26.977615 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.977448 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6bddaeac-9272-4031-b216-a31fbe58c89d-service-ca-bundle\") pod \"router-default-7494768586-7px6v\" (UID: \"6bddaeac-9272-4031-b216-a31fbe58c89d\") " pod="openshift-ingress/router-default-7494768586-7px6v" Jul 23 09:17:26.980028 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.980005 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b7fmg" Jul 23 09:17:26.980132 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.980059 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b24sn" Jul 23 09:17:26.980132 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.980057 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6fgq" Jul 23 09:17:26.983304 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.983103 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tghcj\"" Jul 23 09:17:26.983304 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.983226 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-s2sdc\"" Jul 23 09:17:26.983653 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.983634 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 23 09:17:26.984220 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.984199 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 23 09:17:26.994090 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.994059 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-cg599"] Jul 23 09:17:26.995994 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.995974 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cg599" Jul 23 09:17:26.999693 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.999676 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-bcmmh\"" Jul 23 09:17:26.999831 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:26.999813 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 23 09:17:27.000127 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.000110 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 23 09:17:27.000758 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.000744 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 23 09:17:27.015514 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.015495 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-cg599"] Jul 23 09:17:27.078226 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.078197 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn958\" (UniqueName: \"kubernetes.io/projected/a17a8c5e-a4ee-4e04-b1dc-03ec2e43c60f-kube-api-access-vn958\") pod \"cluster-samples-operator-79b7c55f87-xtr4d\" (UID: \"a17a8c5e-a4ee-4e04-b1dc-03ec2e43c60f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-xtr4d" Jul 23 09:17:27.078377 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.078283 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1260ceaa-4aa5-4130-9ea7-ff8212c2d751-trusted-ca\") pod \"image-registry-7989bd97d7-g2rdd\" (UID: \"1260ceaa-4aa5-4130-9ea7-ff8212c2d751\") " pod="openshift-image-registry/image-registry-7989bd97d7-g2rdd" Jul 23 09:17:27.078377 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.078321 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/989625f0-3f97-4f66-8d67-2e8f47a13a6c-service-ca-bundle\") pod \"insights-operator-555d9d7885-pxlst\" (UID: \"989625f0-3f97-4f66-8d67-2e8f47a13a6c\") " pod="openshift-insights/insights-operator-555d9d7885-pxlst" Jul 23 09:17:27.078377 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.078349 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ef3505b4-c02c-487a-8314-f6793087ad3a-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-nfv7r\" (UID: \"ef3505b4-c02c-487a-8314-f6793087ad3a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-nfv7r" Jul 23 09:17:27.078377 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.078373 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4908aff3-196e-44fa-b8e6-c0e81b476433-trusted-ca\") pod \"console-operator-565b7b569d-5szxg\" (UID: \"4908aff3-196e-44fa-b8e6-c0e81b476433\") " pod="openshift-console-operator/console-operator-565b7b569d-5szxg" Jul 23 09:17:27.078575 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.078440 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31bba45e-487f-4c59-ac81-d8e3b90a549d-tmp-dir\") pod \"dns-default-mgk4m\" (UID: \"31bba45e-487f-4c59-ac81-d8e3b90a549d\") " pod="openshift-dns/dns-default-mgk4m" Jul 23 09:17:27.078575 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.078476 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1260ceaa-4aa5-4130-9ea7-ff8212c2d751-registry-tls\") pod \"image-registry-7989bd97d7-g2rdd\" (UID: \"1260ceaa-4aa5-4130-9ea7-ff8212c2d751\") " pod="openshift-image-registry/image-registry-7989bd97d7-g2rdd" Jul 23 09:17:27.078575 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.078507 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhwc2\" (UniqueName: \"kubernetes.io/projected/31bba45e-487f-4c59-ac81-d8e3b90a549d-kube-api-access-xhwc2\") pod \"dns-default-mgk4m\" (UID: \"31bba45e-487f-4c59-ac81-d8e3b90a549d\") " pod="openshift-dns/dns-default-mgk4m" Jul 23 09:17:27.078575 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.078548 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b5c91d72-2e17-44e7-83ab-4b9ce3a89b43-config\") pod \"service-ca-operator-5bcbc94c78-sp78t\" (UID: \"b5c91d72-2e17-44e7-83ab-4b9ce3a89b43\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-sp78t" Jul 23 09:17:27.078769 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:27.078636 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 09:17:27.078769 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:27.078652 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7989bd97d7-g2rdd: secret "image-registry-tls" not found Jul 23 09:17:27.078769 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.078659 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vvmp\" (UniqueName: \"kubernetes.io/projected/1260ceaa-4aa5-4130-9ea7-ff8212c2d751-kube-api-access-6vvmp\") pod \"image-registry-7989bd97d7-g2rdd\" (UID: \"1260ceaa-4aa5-4130-9ea7-ff8212c2d751\") " pod="openshift-image-registry/image-registry-7989bd97d7-g2rdd" Jul 23 09:17:27.078769 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.078695 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp4tw\" (UniqueName: \"kubernetes.io/projected/59ee07ce-4747-496f-9876-81f2c2358981-kube-api-access-tp4tw\") pod \"cluster-monitoring-operator-76cf8966f8-8zbdn\" (UID: \"59ee07ce-4747-496f-9876-81f2c2358981\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8zbdn" Jul 23 09:17:27.078769 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.078727 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1260ceaa-4aa5-4130-9ea7-ff8212c2d751-ca-trust-extracted\") pod \"image-registry-7989bd97d7-g2rdd\" (UID: \"1260ceaa-4aa5-4130-9ea7-ff8212c2d751\") " pod="openshift-image-registry/image-registry-7989bd97d7-g2rdd" Jul 23 09:17:27.078769 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.078754 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a17a8c5e-a4ee-4e04-b1dc-03ec2e43c60f-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-xtr4d\" (UID: \"a17a8c5e-a4ee-4e04-b1dc-03ec2e43c60f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-xtr4d" Jul 23 09:17:27.079053 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.078780 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/989625f0-3f97-4f66-8d67-2e8f47a13a6c-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-pxlst\" (UID: \"989625f0-3f97-4f66-8d67-2e8f47a13a6c\") " pod="openshift-insights/insights-operator-555d9d7885-pxlst" Jul 23 09:17:27.079053 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.078805 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4908aff3-196e-44fa-b8e6-c0e81b476433-config\") pod \"console-operator-565b7b569d-5szxg\" (UID: \"4908aff3-196e-44fa-b8e6-c0e81b476433\") " pod="openshift-console-operator/console-operator-565b7b569d-5szxg" Jul 23 09:17:27.079053 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.078830 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqj8w\" (UniqueName: \"kubernetes.io/projected/4908aff3-196e-44fa-b8e6-c0e81b476433-kube-api-access-kqj8w\") pod \"console-operator-565b7b569d-5szxg\" (UID: \"4908aff3-196e-44fa-b8e6-c0e81b476433\") " pod="openshift-console-operator/console-operator-565b7b569d-5szxg" Jul 23 09:17:27.079053 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.078855 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6bddaeac-9272-4031-b216-a31fbe58c89d-stats-auth\") pod \"router-default-7494768586-7px6v\" (UID: \"6bddaeac-9272-4031-b216-a31fbe58c89d\") " pod="openshift-ingress/router-default-7494768586-7px6v" Jul 23 09:17:27.079053 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.078881 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/989625f0-3f97-4f66-8d67-2e8f47a13a6c-snapshots\") pod \"insights-operator-555d9d7885-pxlst\" (UID: \"989625f0-3f97-4f66-8d67-2e8f47a13a6c\") " pod="openshift-insights/insights-operator-555d9d7885-pxlst" Jul 23 09:17:27.079053 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.078907 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxmxd\" (UniqueName: \"kubernetes.io/projected/ef3505b4-c02c-487a-8314-f6793087ad3a-kube-api-access-vxmxd\") pod \"kube-storage-version-migrator-operator-7658d9b749-nfv7r\" (UID: \"ef3505b4-c02c-487a-8314-f6793087ad3a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-nfv7r" Jul 23 09:17:27.079053 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.078931 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/31bba45e-487f-4c59-ac81-d8e3b90a549d-config-volume\") pod \"dns-default-mgk4m\" (UID: \"31bba45e-487f-4c59-ac81-d8e3b90a549d\") " pod="openshift-dns/dns-default-mgk4m" Jul 23 09:17:27.079053 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.078958 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1260ceaa-4aa5-4130-9ea7-ff8212c2d751-registry-certificates\") pod \"image-registry-7989bd97d7-g2rdd\" (UID: \"1260ceaa-4aa5-4130-9ea7-ff8212c2d751\") " pod="openshift-image-registry/image-registry-7989bd97d7-g2rdd" Jul 23 09:17:27.079053 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.078982 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/989625f0-3f97-4f66-8d67-2e8f47a13a6c-tmp\") pod \"insights-operator-555d9d7885-pxlst\" (UID: \"989625f0-3f97-4f66-8d67-2e8f47a13a6c\") " pod="openshift-insights/insights-operator-555d9d7885-pxlst" Jul 23 09:17:27.079053 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.078984 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/989625f0-3f97-4f66-8d67-2e8f47a13a6c-service-ca-bundle\") pod \"insights-operator-555d9d7885-pxlst\" (UID: \"989625f0-3f97-4f66-8d67-2e8f47a13a6c\") " pod="openshift-insights/insights-operator-555d9d7885-pxlst" Jul 23 09:17:27.079053 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.079023 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1260ceaa-4aa5-4130-9ea7-ff8212c2d751-ca-trust-extracted\") pod \"image-registry-7989bd97d7-g2rdd\" (UID: \"1260ceaa-4aa5-4130-9ea7-ff8212c2d751\") " pod="openshift-image-registry/image-registry-7989bd97d7-g2rdd" Jul 23 09:17:27.079053 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:27.079039 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1260ceaa-4aa5-4130-9ea7-ff8212c2d751-registry-tls podName:1260ceaa-4aa5-4130-9ea7-ff8212c2d751 nodeName:}" failed. No retries permitted until 2026-07-23 09:17:27.57901941 +0000 UTC m=+36.118297972 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1260ceaa-4aa5-4130-9ea7-ff8212c2d751-registry-tls") pod "image-registry-7989bd97d7-g2rdd" (UID: "1260ceaa-4aa5-4130-9ea7-ff8212c2d751") : secret "image-registry-tls" not found Jul 23 09:17:27.079667 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.079186 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/31bba45e-487f-4c59-ac81-d8e3b90a549d-metrics-tls\") pod \"dns-default-mgk4m\" (UID: \"31bba45e-487f-4c59-ac81-d8e3b90a549d\") " pod="openshift-dns/dns-default-mgk4m" Jul 23 09:17:27.079667 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.079198 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/989625f0-3f97-4f66-8d67-2e8f47a13a6c-tmp\") pod \"insights-operator-555d9d7885-pxlst\" (UID: \"989625f0-3f97-4f66-8d67-2e8f47a13a6c\") " pod="openshift-insights/insights-operator-555d9d7885-pxlst" Jul 23 09:17:27.079667 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.079222 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b5c91d72-2e17-44e7-83ab-4b9ce3a89b43-serving-cert\") pod \"service-ca-operator-5bcbc94c78-sp78t\" (UID: \"b5c91d72-2e17-44e7-83ab-4b9ce3a89b43\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-sp78t" Jul 23 09:17:27.079667 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.079272 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6bddaeac-9272-4031-b216-a31fbe58c89d-default-certificate\") pod \"router-default-7494768586-7px6v\" (UID: \"6bddaeac-9272-4031-b216-a31fbe58c89d\") " pod="openshift-ingress/router-default-7494768586-7px6v" Jul 23 09:17:27.079667 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.079287 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1260ceaa-4aa5-4130-9ea7-ff8212c2d751-trusted-ca\") pod \"image-registry-7989bd97d7-g2rdd\" (UID: \"1260ceaa-4aa5-4130-9ea7-ff8212c2d751\") " pod="openshift-image-registry/image-registry-7989bd97d7-g2rdd" Jul 23 09:17:27.079667 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.079377 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h47rf\" (UniqueName: \"kubernetes.io/projected/6bddaeac-9272-4031-b216-a31fbe58c89d-kube-api-access-h47rf\") pod \"router-default-7494768586-7px6v\" (UID: \"6bddaeac-9272-4031-b216-a31fbe58c89d\") " pod="openshift-ingress/router-default-7494768586-7px6v" Jul 23 09:17:27.079667 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.079410 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqpjk\" (UniqueName: \"kubernetes.io/projected/989625f0-3f97-4f66-8d67-2e8f47a13a6c-kube-api-access-rqpjk\") pod \"insights-operator-555d9d7885-pxlst\" (UID: \"989625f0-3f97-4f66-8d67-2e8f47a13a6c\") " pod="openshift-insights/insights-operator-555d9d7885-pxlst" Jul 23 09:17:27.079953 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.079603 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/59ee07ce-4747-496f-9876-81f2c2358981-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-8zbdn\" (UID: \"59ee07ce-4747-496f-9876-81f2c2358981\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8zbdn" Jul 23 09:17:27.079953 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.079848 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6bddaeac-9272-4031-b216-a31fbe58c89d-service-ca-bundle\") pod \"router-default-7494768586-7px6v\" (UID: \"6bddaeac-9272-4031-b216-a31fbe58c89d\") " pod="openshift-ingress/router-default-7494768586-7px6v" Jul 23 09:17:27.079953 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.079677 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1260ceaa-4aa5-4130-9ea7-ff8212c2d751-registry-certificates\") pod \"image-registry-7989bd97d7-g2rdd\" (UID: \"1260ceaa-4aa5-4130-9ea7-ff8212c2d751\") " pod="openshift-image-registry/image-registry-7989bd97d7-g2rdd" Jul 23 09:17:27.079953 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.079728 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/989625f0-3f97-4f66-8d67-2e8f47a13a6c-snapshots\") pod \"insights-operator-555d9d7885-pxlst\" (UID: \"989625f0-3f97-4f66-8d67-2e8f47a13a6c\") " pod="openshift-insights/insights-operator-555d9d7885-pxlst" Jul 23 09:17:27.079953 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.079770 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/989625f0-3f97-4f66-8d67-2e8f47a13a6c-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-pxlst\" (UID: \"989625f0-3f97-4f66-8d67-2e8f47a13a6c\") " pod="openshift-insights/insights-operator-555d9d7885-pxlst" Jul 23 09:17:27.079953 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.079931 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1260ceaa-4aa5-4130-9ea7-ff8212c2d751-installation-pull-secrets\") pod \"image-registry-7989bd97d7-g2rdd\" (UID: \"1260ceaa-4aa5-4130-9ea7-ff8212c2d751\") " pod="openshift-image-registry/image-registry-7989bd97d7-g2rdd" Jul 23 09:17:27.080189 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.079963 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/59ee07ce-4747-496f-9876-81f2c2358981-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-8zbdn\" (UID: \"59ee07ce-4747-496f-9876-81f2c2358981\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8zbdn" Jul 23 09:17:27.080189 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.080015 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st24m\" (UniqueName: \"kubernetes.io/projected/b5c91d72-2e17-44e7-83ab-4b9ce3a89b43-kube-api-access-st24m\") pod \"service-ca-operator-5bcbc94c78-sp78t\" (UID: \"b5c91d72-2e17-44e7-83ab-4b9ce3a89b43\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-sp78t" Jul 23 09:17:27.080189 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.080069 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1260ceaa-4aa5-4130-9ea7-ff8212c2d751-image-registry-private-configuration\") pod \"image-registry-7989bd97d7-g2rdd\" (UID: \"1260ceaa-4aa5-4130-9ea7-ff8212c2d751\") " pod="openshift-image-registry/image-registry-7989bd97d7-g2rdd" Jul 23 09:17:27.080189 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.080096 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/989625f0-3f97-4f66-8d67-2e8f47a13a6c-serving-cert\") pod \"insights-operator-555d9d7885-pxlst\" (UID: \"989625f0-3f97-4f66-8d67-2e8f47a13a6c\") " pod="openshift-insights/insights-operator-555d9d7885-pxlst" Jul 23 09:17:27.080189 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.080125 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4908aff3-196e-44fa-b8e6-c0e81b476433-serving-cert\") pod \"console-operator-565b7b569d-5szxg\" (UID: \"4908aff3-196e-44fa-b8e6-c0e81b476433\") " pod="openshift-console-operator/console-operator-565b7b569d-5szxg" Jul 23 09:17:27.080189 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.080151 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bddaeac-9272-4031-b216-a31fbe58c89d-metrics-certs\") pod \"router-default-7494768586-7px6v\" (UID: \"6bddaeac-9272-4031-b216-a31fbe58c89d\") " pod="openshift-ingress/router-default-7494768586-7px6v" Jul 23 09:17:27.080189 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.080176 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1260ceaa-4aa5-4130-9ea7-ff8212c2d751-bound-sa-token\") pod \"image-registry-7989bd97d7-g2rdd\" (UID: \"1260ceaa-4aa5-4130-9ea7-ff8212c2d751\") " pod="openshift-image-registry/image-registry-7989bd97d7-g2rdd" Jul 23 09:17:27.080547 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.080202 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ef3505b4-c02c-487a-8314-f6793087ad3a-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-nfv7r\" (UID: \"ef3505b4-c02c-487a-8314-f6793087ad3a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-nfv7r" Jul 23 09:17:27.080547 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.080232 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd7lq\" (UniqueName: \"kubernetes.io/projected/e11b7a19-c8e6-4d52-8450-b86a0ca3fcd4-kube-api-access-qd7lq\") pod \"volume-data-source-validator-78cf4c6f6-k296x\" (UID: \"e11b7a19-c8e6-4d52-8450-b86a0ca3fcd4\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-k296x" Jul 23 09:17:27.080547 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.080278 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqztw\" (UniqueName: \"kubernetes.io/projected/fae77c41-37e2-42c2-92a1-8908061bab7a-kube-api-access-vqztw\") pod \"network-check-source-7bc5b6bf85-b5cz2\" (UID: \"fae77c41-37e2-42c2-92a1-8908061bab7a\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-b5cz2" Jul 23 09:17:27.084652 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.084630 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/989625f0-3f97-4f66-8d67-2e8f47a13a6c-serving-cert\") pod \"insights-operator-555d9d7885-pxlst\" (UID: \"989625f0-3f97-4f66-8d67-2e8f47a13a6c\") " pod="openshift-insights/insights-operator-555d9d7885-pxlst" Jul 23 09:17:27.084762 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.084712 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1260ceaa-4aa5-4130-9ea7-ff8212c2d751-image-registry-private-configuration\") pod \"image-registry-7989bd97d7-g2rdd\" (UID: \"1260ceaa-4aa5-4130-9ea7-ff8212c2d751\") " pod="openshift-image-registry/image-registry-7989bd97d7-g2rdd" Jul 23 09:17:27.084762 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.084712 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1260ceaa-4aa5-4130-9ea7-ff8212c2d751-installation-pull-secrets\") pod \"image-registry-7989bd97d7-g2rdd\" (UID: \"1260ceaa-4aa5-4130-9ea7-ff8212c2d751\") " pod="openshift-image-registry/image-registry-7989bd97d7-g2rdd" Jul 23 09:17:27.109743 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.109720 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1260ceaa-4aa5-4130-9ea7-ff8212c2d751-bound-sa-token\") pod \"image-registry-7989bd97d7-g2rdd\" (UID: \"1260ceaa-4aa5-4130-9ea7-ff8212c2d751\") " pod="openshift-image-registry/image-registry-7989bd97d7-g2rdd" Jul 23 09:17:27.109743 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.109733 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqpjk\" (UniqueName: \"kubernetes.io/projected/989625f0-3f97-4f66-8d67-2e8f47a13a6c-kube-api-access-rqpjk\") pod \"insights-operator-555d9d7885-pxlst\" (UID: \"989625f0-3f97-4f66-8d67-2e8f47a13a6c\") " pod="openshift-insights/insights-operator-555d9d7885-pxlst" Jul 23 09:17:27.111447 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.111416 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vvmp\" (UniqueName: \"kubernetes.io/projected/1260ceaa-4aa5-4130-9ea7-ff8212c2d751-kube-api-access-6vvmp\") pod \"image-registry-7989bd97d7-g2rdd\" (UID: \"1260ceaa-4aa5-4130-9ea7-ff8212c2d751\") " pod="openshift-image-registry/image-registry-7989bd97d7-g2rdd" Jul 23 09:17:27.180105 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.180038 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-pxlst" Jul 23 09:17:27.181381 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.181360 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhwc2\" (UniqueName: \"kubernetes.io/projected/31bba45e-487f-4c59-ac81-d8e3b90a549d-kube-api-access-xhwc2\") pod \"dns-default-mgk4m\" (UID: \"31bba45e-487f-4c59-ac81-d8e3b90a549d\") " pod="openshift-dns/dns-default-mgk4m" Jul 23 09:17:27.181492 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.181407 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a17a8c5e-a4ee-4e04-b1dc-03ec2e43c60f-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-xtr4d\" (UID: \"a17a8c5e-a4ee-4e04-b1dc-03ec2e43c60f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-xtr4d" Jul 23 09:17:27.181548 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:27.181505 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 23 09:17:27.181606 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.181533 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/31bba45e-487f-4c59-ac81-d8e3b90a549d-config-volume\") pod \"dns-default-mgk4m\" (UID: \"31bba45e-487f-4c59-ac81-d8e3b90a549d\") " pod="openshift-dns/dns-default-mgk4m" Jul 23 09:17:27.181606 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:27.181573 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a17a8c5e-a4ee-4e04-b1dc-03ec2e43c60f-samples-operator-tls podName:a17a8c5e-a4ee-4e04-b1dc-03ec2e43c60f nodeName:}" failed. No retries permitted until 2026-07-23 09:17:27.68155421 +0000 UTC m=+36.220832770 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a17a8c5e-a4ee-4e04-b1dc-03ec2e43c60f-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-xtr4d" (UID: "a17a8c5e-a4ee-4e04-b1dc-03ec2e43c60f") : secret "samples-operator-tls" not found Jul 23 09:17:27.181606 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.181601 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/31bba45e-487f-4c59-ac81-d8e3b90a549d-metrics-tls\") pod \"dns-default-mgk4m\" (UID: \"31bba45e-487f-4c59-ac81-d8e3b90a549d\") " pod="openshift-dns/dns-default-mgk4m" Jul 23 09:17:27.181759 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.181686 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsqvb\" (UniqueName: \"kubernetes.io/projected/022429e9-cac7-4ace-ae03-c39b633d6196-kube-api-access-nsqvb\") pod \"ingress-canary-cg599\" (UID: \"022429e9-cac7-4ace-ae03-c39b633d6196\") " pod="openshift-ingress-canary/ingress-canary-cg599" Jul 23 09:17:27.181759 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:27.181736 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 09:17:27.181839 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:27.181771 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31bba45e-487f-4c59-ac81-d8e3b90a549d-metrics-tls podName:31bba45e-487f-4c59-ac81-d8e3b90a549d nodeName:}" failed. No retries permitted until 2026-07-23 09:17:27.681760898 +0000 UTC m=+36.221039456 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/31bba45e-487f-4c59-ac81-d8e3b90a549d-metrics-tls") pod "dns-default-mgk4m" (UID: "31bba45e-487f-4c59-ac81-d8e3b90a549d") : secret "dns-default-metrics-tls" not found Jul 23 09:17:27.181839 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.181738 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd7lq\" (UniqueName: \"kubernetes.io/projected/e11b7a19-c8e6-4d52-8450-b86a0ca3fcd4-kube-api-access-qd7lq\") pod \"volume-data-source-validator-78cf4c6f6-k296x\" (UID: \"e11b7a19-c8e6-4d52-8450-b86a0ca3fcd4\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-k296x" Jul 23 09:17:27.181839 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.181810 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqztw\" (UniqueName: \"kubernetes.io/projected/fae77c41-37e2-42c2-92a1-8908061bab7a-kube-api-access-vqztw\") pod \"network-check-source-7bc5b6bf85-b5cz2\" (UID: \"fae77c41-37e2-42c2-92a1-8908061bab7a\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-b5cz2" Jul 23 09:17:27.181839 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.181833 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn958\" (UniqueName: \"kubernetes.io/projected/a17a8c5e-a4ee-4e04-b1dc-03ec2e43c60f-kube-api-access-vn958\") pod \"cluster-samples-operator-79b7c55f87-xtr4d\" (UID: \"a17a8c5e-a4ee-4e04-b1dc-03ec2e43c60f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-xtr4d" Jul 23 09:17:27.181979 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.181860 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/022429e9-cac7-4ace-ae03-c39b633d6196-cert\") pod \"ingress-canary-cg599\" (UID: \"022429e9-cac7-4ace-ae03-c39b633d6196\") " pod="openshift-ingress-canary/ingress-canary-cg599" Jul 23 09:17:27.181979 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.181916 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31bba45e-487f-4c59-ac81-d8e3b90a549d-tmp-dir\") pod \"dns-default-mgk4m\" (UID: \"31bba45e-487f-4c59-ac81-d8e3b90a549d\") " pod="openshift-dns/dns-default-mgk4m" Jul 23 09:17:27.182076 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.182033 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/31bba45e-487f-4c59-ac81-d8e3b90a549d-config-volume\") pod \"dns-default-mgk4m\" (UID: \"31bba45e-487f-4c59-ac81-d8e3b90a549d\") " pod="openshift-dns/dns-default-mgk4m" Jul 23 09:17:27.182259 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.182216 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31bba45e-487f-4c59-ac81-d8e3b90a549d-tmp-dir\") pod \"dns-default-mgk4m\" (UID: \"31bba45e-487f-4c59-ac81-d8e3b90a549d\") " pod="openshift-dns/dns-default-mgk4m" Jul 23 09:17:27.199671 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.199646 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn958\" (UniqueName: \"kubernetes.io/projected/a17a8c5e-a4ee-4e04-b1dc-03ec2e43c60f-kube-api-access-vn958\") pod \"cluster-samples-operator-79b7c55f87-xtr4d\" (UID: \"a17a8c5e-a4ee-4e04-b1dc-03ec2e43c60f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-xtr4d" Jul 23 09:17:27.201706 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.201687 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhwc2\" (UniqueName: \"kubernetes.io/projected/31bba45e-487f-4c59-ac81-d8e3b90a549d-kube-api-access-xhwc2\") pod \"dns-default-mgk4m\" (UID: \"31bba45e-487f-4c59-ac81-d8e3b90a549d\") " pod="openshift-dns/dns-default-mgk4m" Jul 23 09:17:27.210977 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.210958 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd7lq\" (UniqueName: \"kubernetes.io/projected/e11b7a19-c8e6-4d52-8450-b86a0ca3fcd4-kube-api-access-qd7lq\") pod \"volume-data-source-validator-78cf4c6f6-k296x\" (UID: \"e11b7a19-c8e6-4d52-8450-b86a0ca3fcd4\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-k296x" Jul 23 09:17:27.211341 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.211324 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqztw\" (UniqueName: \"kubernetes.io/projected/fae77c41-37e2-42c2-92a1-8908061bab7a-kube-api-access-vqztw\") pod \"network-check-source-7bc5b6bf85-b5cz2\" (UID: \"fae77c41-37e2-42c2-92a1-8908061bab7a\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-b5cz2" Jul 23 09:17:27.225708 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.225689 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-k296x" Jul 23 09:17:27.254174 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.254105 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-b5cz2" Jul 23 09:17:27.282768 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.282743 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/022429e9-cac7-4ace-ae03-c39b633d6196-cert\") pod \"ingress-canary-cg599\" (UID: \"022429e9-cac7-4ace-ae03-c39b633d6196\") " pod="openshift-ingress-canary/ingress-canary-cg599" Jul 23 09:17:27.283021 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.282997 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsqvb\" (UniqueName: \"kubernetes.io/projected/022429e9-cac7-4ace-ae03-c39b633d6196-kube-api-access-nsqvb\") pod \"ingress-canary-cg599\" (UID: \"022429e9-cac7-4ace-ae03-c39b633d6196\") " pod="openshift-ingress-canary/ingress-canary-cg599" Jul 23 09:17:27.285289 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.285265 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/022429e9-cac7-4ace-ae03-c39b633d6196-cert\") pod \"ingress-canary-cg599\" (UID: \"022429e9-cac7-4ace-ae03-c39b633d6196\") " pod="openshift-ingress-canary/ingress-canary-cg599" Jul 23 09:17:27.292867 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.292844 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsqvb\" (UniqueName: \"kubernetes.io/projected/022429e9-cac7-4ace-ae03-c39b633d6196-kube-api-access-nsqvb\") pod \"ingress-canary-cg599\" (UID: \"022429e9-cac7-4ace-ae03-c39b633d6196\") " pod="openshift-ingress-canary/ingress-canary-cg599" Jul 23 09:17:27.312119 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.312095 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cg599" Jul 23 09:17:27.586202 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.586118 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1260ceaa-4aa5-4130-9ea7-ff8212c2d751-registry-tls\") pod \"image-registry-7989bd97d7-g2rdd\" (UID: \"1260ceaa-4aa5-4130-9ea7-ff8212c2d751\") " pod="openshift-image-registry/image-registry-7989bd97d7-g2rdd" Jul 23 09:17:27.586347 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:27.586255 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 09:17:27.586347 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:27.586269 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7989bd97d7-g2rdd: secret "image-registry-tls" not found Jul 23 09:17:27.586347 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:27.586322 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1260ceaa-4aa5-4130-9ea7-ff8212c2d751-registry-tls podName:1260ceaa-4aa5-4130-9ea7-ff8212c2d751 nodeName:}" failed. No retries permitted until 2026-07-23 09:17:28.586308118 +0000 UTC m=+37.125586673 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1260ceaa-4aa5-4130-9ea7-ff8212c2d751-registry-tls") pod "image-registry-7989bd97d7-g2rdd" (UID: "1260ceaa-4aa5-4130-9ea7-ff8212c2d751") : secret "image-registry-tls" not found Jul 23 09:17:27.680266 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.680219 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 23 09:17:27.686755 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.686720 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a17a8c5e-a4ee-4e04-b1dc-03ec2e43c60f-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-xtr4d\" (UID: \"a17a8c5e-a4ee-4e04-b1dc-03ec2e43c60f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-xtr4d" Jul 23 09:17:27.686896 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.686773 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/31bba45e-487f-4c59-ac81-d8e3b90a549d-metrics-tls\") pod \"dns-default-mgk4m\" (UID: \"31bba45e-487f-4c59-ac81-d8e3b90a549d\") " pod="openshift-dns/dns-default-mgk4m" Jul 23 09:17:27.686896 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:27.686886 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 23 09:17:27.687006 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:27.686941 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a17a8c5e-a4ee-4e04-b1dc-03ec2e43c60f-samples-operator-tls podName:a17a8c5e-a4ee-4e04-b1dc-03ec2e43c60f nodeName:}" failed. No retries permitted until 2026-07-23 09:17:28.686926892 +0000 UTC m=+37.226205451 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a17a8c5e-a4ee-4e04-b1dc-03ec2e43c60f-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-xtr4d" (UID: "a17a8c5e-a4ee-4e04-b1dc-03ec2e43c60f") : secret "samples-operator-tls" not found Jul 23 09:17:27.687006 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:27.686886 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 09:17:27.687006 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:27.687003 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31bba45e-487f-4c59-ac81-d8e3b90a549d-metrics-tls podName:31bba45e-487f-4c59-ac81-d8e3b90a549d nodeName:}" failed. No retries permitted until 2026-07-23 09:17:28.68699073 +0000 UTC m=+37.226269286 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/31bba45e-487f-4c59-ac81-d8e3b90a549d-metrics-tls") pod "dns-default-mgk4m" (UID: "31bba45e-487f-4c59-ac81-d8e3b90a549d") : secret "dns-default-metrics-tls" not found Jul 23 09:17:27.689898 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.689871 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqj8w\" (UniqueName: \"kubernetes.io/projected/4908aff3-196e-44fa-b8e6-c0e81b476433-kube-api-access-kqj8w\") pod \"console-operator-565b7b569d-5szxg\" (UID: \"4908aff3-196e-44fa-b8e6-c0e81b476433\") " pod="openshift-console-operator/console-operator-565b7b569d-5szxg" Jul 23 09:17:27.766799 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.766769 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 23 09:17:27.816199 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.816170 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 23 09:17:27.819461 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.819439 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-vjvsg\"" Jul 23 09:17:27.820476 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:27.820453 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 23 09:17:27.820562 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:27.820546 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59ee07ce-4747-496f-9876-81f2c2358981-cluster-monitoring-operator-tls podName:59ee07ce-4747-496f-9876-81f2c2358981 nodeName:}" failed. No retries permitted until 2026-07-23 09:17:28.320525436 +0000 UTC m=+36.859803994 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/59ee07ce-4747-496f-9876-81f2c2358981-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-8zbdn" (UID: "59ee07ce-4747-496f-9876-81f2c2358981") : secret "cluster-monitoring-operator-tls" not found Jul 23 09:17:27.863040 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.862966 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 23 09:17:27.872216 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.872186 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b5c91d72-2e17-44e7-83ab-4b9ce3a89b43-serving-cert\") pod \"service-ca-operator-5bcbc94c78-sp78t\" (UID: \"b5c91d72-2e17-44e7-83ab-4b9ce3a89b43\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-sp78t" Jul 23 09:17:27.878492 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.878464 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-xqgv5\"" Jul 23 09:17:27.904999 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.904972 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 23 09:17:27.911252 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.911216 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ef3505b4-c02c-487a-8314-f6793087ad3a-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-nfv7r\" (UID: \"ef3505b4-c02c-487a-8314-f6793087ad3a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-nfv7r" Jul 23 09:17:27.923762 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.923743 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 23 09:17:27.934380 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.934358 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 23 09:17:27.941526 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.941505 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ef3505b4-c02c-487a-8314-f6793087ad3a-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-nfv7r\" (UID: \"ef3505b4-c02c-487a-8314-f6793087ad3a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-nfv7r" Jul 23 09:17:27.943770 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.943753 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 23 09:17:27.960809 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.960784 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 23 09:17:27.970060 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:27.970041 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6bddaeac-9272-4031-b216-a31fbe58c89d-service-ca-bundle podName:6bddaeac-9272-4031-b216-a31fbe58c89d nodeName:}" failed. No retries permitted until 2026-07-23 09:17:28.470019558 +0000 UTC m=+37.009298114 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6bddaeac-9272-4031-b216-a31fbe58c89d-service-ca-bundle") pod "router-default-7494768586-7px6v" (UID: "6bddaeac-9272-4031-b216-a31fbe58c89d") : configmap references non-existent config key: service-ca.crt Jul 23 09:17:27.986191 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.986169 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 23 09:17:27.989741 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:27.989720 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4908aff3-196e-44fa-b8e6-c0e81b476433-trusted-ca\") pod \"console-operator-565b7b569d-5szxg\" (UID: \"4908aff3-196e-44fa-b8e6-c0e81b476433\") " pod="openshift-console-operator/console-operator-565b7b569d-5szxg" Jul 23 09:17:28.011971 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:28.011951 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 23 09:17:28.019477 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:28.019453 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4908aff3-196e-44fa-b8e6-c0e81b476433-config\") pod \"console-operator-565b7b569d-5szxg\" (UID: \"4908aff3-196e-44fa-b8e6-c0e81b476433\") " pod="openshift-console-operator/console-operator-565b7b569d-5szxg" Jul 23 09:17:28.021195 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:28.021177 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-4dsz5\"" Jul 23 09:17:28.034184 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:28.034164 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-5g67b\"" Jul 23 09:17:28.078946 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:28.078920 2576 configmap.go:193] Couldn't get configMap openshift-service-ca-operator/service-ca-operator-config: failed to sync configmap cache: timed out waiting for the condition Jul 23 09:17:28.079025 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:28.078991 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b5c91d72-2e17-44e7-83ab-4b9ce3a89b43-config podName:b5c91d72-2e17-44e7-83ab-4b9ce3a89b43 nodeName:}" failed. No retries permitted until 2026-07-23 09:17:28.578973474 +0000 UTC m=+37.118252032 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "config" (UniqueName: "kubernetes.io/configmap/b5c91d72-2e17-44e7-83ab-4b9ce3a89b43-config") pod "service-ca-operator-5bcbc94c78-sp78t" (UID: "b5c91d72-2e17-44e7-83ab-4b9ce3a89b43") : failed to sync configmap cache: timed out waiting for the condition Jul 23 09:17:28.080011 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:28.079992 2576 secret.go:189] Couldn't get secret openshift-ingress/router-stats-default: failed to sync secret cache: timed out waiting for the condition Jul 23 09:17:28.080011 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:28.080006 2576 secret.go:189] Couldn't get secret openshift-ingress/default-ingress-cert: failed to sync secret cache: timed out waiting for the condition Jul 23 09:17:28.080131 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:28.080058 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bddaeac-9272-4031-b216-a31fbe58c89d-stats-auth podName:6bddaeac-9272-4031-b216-a31fbe58c89d nodeName:}" failed. No retries permitted until 2026-07-23 09:17:28.580043537 +0000 UTC m=+37.119322096 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "stats-auth" (UniqueName: "kubernetes.io/secret/6bddaeac-9272-4031-b216-a31fbe58c89d-stats-auth") pod "router-default-7494768586-7px6v" (UID: "6bddaeac-9272-4031-b216-a31fbe58c89d") : failed to sync secret cache: timed out waiting for the condition Jul 23 09:17:28.080131 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:28.080074 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bddaeac-9272-4031-b216-a31fbe58c89d-default-certificate podName:6bddaeac-9272-4031-b216-a31fbe58c89d nodeName:}" failed. No retries permitted until 2026-07-23 09:17:28.580065606 +0000 UTC m=+37.119344161 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "default-certificate" (UniqueName: "kubernetes.io/secret/6bddaeac-9272-4031-b216-a31fbe58c89d-default-certificate") pod "router-default-7494768586-7px6v" (UID: "6bddaeac-9272-4031-b216-a31fbe58c89d") : failed to sync secret cache: timed out waiting for the condition Jul 23 09:17:28.081116 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:28.081085 2576 configmap.go:193] Couldn't get configMap openshift-monitoring/telemetry-config: failed to sync configmap cache: timed out waiting for the condition Jul 23 09:17:28.081116 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:28.081108 2576 secret.go:189] Couldn't get secret openshift-console-operator/serving-cert: failed to sync secret cache: timed out waiting for the condition Jul 23 09:17:28.081270 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:28.081146 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/59ee07ce-4747-496f-9876-81f2c2358981-telemetry-config podName:59ee07ce-4747-496f-9876-81f2c2358981 nodeName:}" failed. No retries permitted until 2026-07-23 09:17:28.581129938 +0000 UTC m=+37.120408495 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "telemetry-config" (UniqueName: "kubernetes.io/configmap/59ee07ce-4747-496f-9876-81f2c2358981-telemetry-config") pod "cluster-monitoring-operator-76cf8966f8-8zbdn" (UID: "59ee07ce-4747-496f-9876-81f2c2358981") : failed to sync configmap cache: timed out waiting for the condition Jul 23 09:17:28.081270 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:28.081095 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: failed to sync secret cache: timed out waiting for the condition Jul 23 09:17:28.081270 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:28.081167 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4908aff3-196e-44fa-b8e6-c0e81b476433-serving-cert podName:4908aff3-196e-44fa-b8e6-c0e81b476433 nodeName:}" failed. No retries permitted until 2026-07-23 09:17:28.581156086 +0000 UTC m=+37.120434647 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "serving-cert" (UniqueName: "kubernetes.io/secret/4908aff3-196e-44fa-b8e6-c0e81b476433-serving-cert") pod "console-operator-565b7b569d-5szxg" (UID: "4908aff3-196e-44fa-b8e6-c0e81b476433") : failed to sync secret cache: timed out waiting for the condition Jul 23 09:17:28.081270 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:28.081208 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bddaeac-9272-4031-b216-a31fbe58c89d-metrics-certs podName:6bddaeac-9272-4031-b216-a31fbe58c89d nodeName:}" failed. No retries permitted until 2026-07-23 09:17:28.581197759 +0000 UTC m=+37.120476313 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6bddaeac-9272-4031-b216-a31fbe58c89d-metrics-certs") pod "router-default-7494768586-7px6v" (UID: "6bddaeac-9272-4031-b216-a31fbe58c89d") : failed to sync secret cache: timed out waiting for the condition Jul 23 09:17:28.091588 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:28.091566 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 23 09:17:28.103861 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:28.103837 2576 projected.go:291] Couldn't get configMap openshift-ingress/kube-root-ca.crt: failed to sync configmap cache: timed out waiting for the condition Jul 23 09:17:28.103939 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:28.103870 2576 projected.go:196] Error preparing data for projected volume kube-api-access-h47rf for pod openshift-ingress/router-default-7494768586-7px6v: failed to sync configmap cache: timed out waiting for the condition Jul 23 09:17:28.103939 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:28.103919 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6bddaeac-9272-4031-b216-a31fbe58c89d-kube-api-access-h47rf podName:6bddaeac-9272-4031-b216-a31fbe58c89d nodeName:}" failed. No retries permitted until 2026-07-23 09:17:28.603904528 +0000 UTC m=+37.143183084 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-h47rf" (UniqueName: "kubernetes.io/projected/6bddaeac-9272-4031-b216-a31fbe58c89d-kube-api-access-h47rf") pod "router-default-7494768586-7px6v" (UID: "6bddaeac-9272-4031-b216-a31fbe58c89d") : failed to sync configmap cache: timed out waiting for the condition Jul 23 09:17:28.119968 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:28.119915 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-w58fm\"" Jul 23 09:17:28.142485 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:28.142468 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 23 09:17:28.156803 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:28.156775 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 23 09:17:28.215873 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:28.214433 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 23 09:17:28.290146 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:28.289902 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 23 09:17:28.295659 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:28.295630 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 23 09:17:28.301606 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:28.301583 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxmxd\" (UniqueName: \"kubernetes.io/projected/ef3505b4-c02c-487a-8314-f6793087ad3a-kube-api-access-vxmxd\") pod \"kube-storage-version-migrator-operator-7658d9b749-nfv7r\" (UID: \"ef3505b4-c02c-487a-8314-f6793087ad3a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-nfv7r" Jul 23 09:17:28.331996 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:28.331050 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-k296x"] Jul 23 09:17:28.335441 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:28.335420 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 23 09:17:28.336527 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:17:28.336498 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode11b7a19_c8e6_4d52_8450_b86a0ca3fcd4.slice/crio-1996a25520346b21ad8c3c128b25b4b654d138378fc8a106548c21dbcfd39f8c WatchSource:0}: Error finding container 1996a25520346b21ad8c3c128b25b4b654d138378fc8a106548c21dbcfd39f8c: Status 404 returned error can't find the container with id 1996a25520346b21ad8c3c128b25b4b654d138378fc8a106548c21dbcfd39f8c Jul 23 09:17:28.340256 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:28.340225 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 23 09:17:28.346376 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:28.346355 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-cg599"] Jul 23 09:17:28.348010 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:28.347958 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-b5cz2"] Jul 23 09:17:28.350205 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:17:28.350175 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod022429e9_cac7_4ace_ae03_c39b633d6196.slice/crio-8cfa877f1abe0a352a038cd70531c2b8bfa13b068896510a1c3e528e77b86dba WatchSource:0}: Error finding container 8cfa877f1abe0a352a038cd70531c2b8bfa13b068896510a1c3e528e77b86dba: Status 404 returned error can't find the container with id 8cfa877f1abe0a352a038cd70531c2b8bfa13b068896510a1c3e528e77b86dba Jul 23 09:17:28.350545 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:28.350518 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-st24m\" (UniqueName: \"kubernetes.io/projected/b5c91d72-2e17-44e7-83ab-4b9ce3a89b43-kube-api-access-st24m\") pod \"service-ca-operator-5bcbc94c78-sp78t\" (UID: \"b5c91d72-2e17-44e7-83ab-4b9ce3a89b43\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-sp78t" Jul 23 09:17:28.351510 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:17:28.351478 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfae77c41_37e2_42c2_92a1_8908061bab7a.slice/crio-63c7b50a99639efc1ea6664effbb674cdfff8f6f90472ea994595ee67a100bff WatchSource:0}: Error finding container 63c7b50a99639efc1ea6664effbb674cdfff8f6f90472ea994595ee67a100bff: Status 404 returned error can't find the container with id 63c7b50a99639efc1ea6664effbb674cdfff8f6f90472ea994595ee67a100bff Jul 23 09:17:28.362145 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:17:28.362128 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod989625f0_3f97_4f66_8d67_2e8f47a13a6c.slice/crio-005964fbff5670ec98bc2d7f4b124e5de376fb36a993986e5e8f10a50bd39d5d WatchSource:0}: Error finding container 005964fbff5670ec98bc2d7f4b124e5de376fb36a993986e5e8f10a50bd39d5d: Status 404 returned error can't find the container with id 005964fbff5670ec98bc2d7f4b124e5de376fb36a993986e5e8f10a50bd39d5d Jul 23 09:17:28.368904 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:28.368873 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-pxlst"] Jul 23 09:17:28.392897 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:28.392873 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/59ee07ce-4747-496f-9876-81f2c2358981-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-8zbdn\" (UID: \"59ee07ce-4747-496f-9876-81f2c2358981\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8zbdn" Jul 23 09:17:28.393022 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:28.393007 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 23 09:17:28.393073 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:28.393064 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59ee07ce-4747-496f-9876-81f2c2358981-cluster-monitoring-operator-tls podName:59ee07ce-4747-496f-9876-81f2c2358981 nodeName:}" failed. No retries permitted until 2026-07-23 09:17:29.393048595 +0000 UTC m=+37.932327150 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/59ee07ce-4747-496f-9876-81f2c2358981-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-8zbdn" (UID: "59ee07ce-4747-496f-9876-81f2c2358981") : secret "cluster-monitoring-operator-tls" not found Jul 23 09:17:28.395118 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:28.395103 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 23 09:17:28.405815 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:28.405796 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp4tw\" (UniqueName: \"kubernetes.io/projected/59ee07ce-4747-496f-9876-81f2c2358981-kube-api-access-tp4tw\") pod \"cluster-monitoring-operator-76cf8966f8-8zbdn\" (UID: \"59ee07ce-4747-496f-9876-81f2c2358981\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8zbdn" Jul 23 09:17:28.418834 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:28.418810 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-nfv7r" Jul 23 09:17:28.465412 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:28.465383 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 23 09:17:28.475345 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:28.475313 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 23 09:17:28.493887 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:28.493863 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6bddaeac-9272-4031-b216-a31fbe58c89d-service-ca-bundle\") pod \"router-default-7494768586-7px6v\" (UID: \"6bddaeac-9272-4031-b216-a31fbe58c89d\") " pod="openshift-ingress/router-default-7494768586-7px6v" Jul 23 09:17:28.494062 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:28.494045 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6bddaeac-9272-4031-b216-a31fbe58c89d-service-ca-bundle podName:6bddaeac-9272-4031-b216-a31fbe58c89d nodeName:}" failed. No retries permitted until 2026-07-23 09:17:29.494022739 +0000 UTC m=+38.033301318 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6bddaeac-9272-4031-b216-a31fbe58c89d-service-ca-bundle") pod "router-default-7494768586-7px6v" (UID: "6bddaeac-9272-4031-b216-a31fbe58c89d") : configmap references non-existent config key: service-ca.crt Jul 23 09:17:28.540376 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:28.540349 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-nfv7r"] Jul 23 09:17:28.543167 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:17:28.543140 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef3505b4_c02c_487a_8314_f6793087ad3a.slice/crio-8e5d661cb649130799811bc604cb5d2fd0efe9dbd04aa195a7dc74fd3ffb5fbb WatchSource:0}: Error finding container 8e5d661cb649130799811bc604cb5d2fd0efe9dbd04aa195a7dc74fd3ffb5fbb: Status 404 returned error can't find the container with id 8e5d661cb649130799811bc604cb5d2fd0efe9dbd04aa195a7dc74fd3ffb5fbb Jul 23 09:17:28.595291 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:28.595268 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6bddaeac-9272-4031-b216-a31fbe58c89d-stats-auth\") pod \"router-default-7494768586-7px6v\" (UID: \"6bddaeac-9272-4031-b216-a31fbe58c89d\") " pod="openshift-ingress/router-default-7494768586-7px6v" Jul 23 09:17:28.595363 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:28.595312 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6bddaeac-9272-4031-b216-a31fbe58c89d-default-certificate\") pod \"router-default-7494768586-7px6v\" (UID: \"6bddaeac-9272-4031-b216-a31fbe58c89d\") " pod="openshift-ingress/router-default-7494768586-7px6v" Jul 23 09:17:28.595475 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:28.595462 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/59ee07ce-4747-496f-9876-81f2c2358981-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-8zbdn\" (UID: \"59ee07ce-4747-496f-9876-81f2c2358981\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8zbdn" Jul 23 09:17:28.595525 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:28.595513 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4908aff3-196e-44fa-b8e6-c0e81b476433-serving-cert\") pod \"console-operator-565b7b569d-5szxg\" (UID: \"4908aff3-196e-44fa-b8e6-c0e81b476433\") " pod="openshift-console-operator/console-operator-565b7b569d-5szxg" Jul 23 09:17:28.595555 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:28.595530 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bddaeac-9272-4031-b216-a31fbe58c89d-metrics-certs\") pod \"router-default-7494768586-7px6v\" (UID: \"6bddaeac-9272-4031-b216-a31fbe58c89d\") " pod="openshift-ingress/router-default-7494768586-7px6v" Jul 23 09:17:28.595591 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:28.595578 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1260ceaa-4aa5-4130-9ea7-ff8212c2d751-registry-tls\") pod \"image-registry-7989bd97d7-g2rdd\" (UID: \"1260ceaa-4aa5-4130-9ea7-ff8212c2d751\") " pod="openshift-image-registry/image-registry-7989bd97d7-g2rdd" Jul 23 09:17:28.595649 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:28.595623 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b5c91d72-2e17-44e7-83ab-4b9ce3a89b43-config\") pod \"service-ca-operator-5bcbc94c78-sp78t\" (UID: \"b5c91d72-2e17-44e7-83ab-4b9ce3a89b43\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-sp78t" Jul 23 09:17:28.595749 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:28.595730 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 23 09:17:28.595921 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:28.595738 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 09:17:28.595921 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:28.595828 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7989bd97d7-g2rdd: secret "image-registry-tls" not found Jul 23 09:17:28.595921 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:28.595813 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bddaeac-9272-4031-b216-a31fbe58c89d-metrics-certs podName:6bddaeac-9272-4031-b216-a31fbe58c89d nodeName:}" failed. No retries permitted until 2026-07-23 09:17:29.595793308 +0000 UTC m=+38.135071876 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6bddaeac-9272-4031-b216-a31fbe58c89d-metrics-certs") pod "router-default-7494768586-7px6v" (UID: "6bddaeac-9272-4031-b216-a31fbe58c89d") : secret "router-metrics-certs-default" not found Jul 23 09:17:28.595921 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:28.595903 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1260ceaa-4aa5-4130-9ea7-ff8212c2d751-registry-tls podName:1260ceaa-4aa5-4130-9ea7-ff8212c2d751 nodeName:}" failed. No retries permitted until 2026-07-23 09:17:30.595884624 +0000 UTC m=+39.135163196 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1260ceaa-4aa5-4130-9ea7-ff8212c2d751-registry-tls") pod "image-registry-7989bd97d7-g2rdd" (UID: "1260ceaa-4aa5-4130-9ea7-ff8212c2d751") : secret "image-registry-tls" not found Jul 23 09:17:28.596220 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:28.596185 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b5c91d72-2e17-44e7-83ab-4b9ce3a89b43-config\") pod \"service-ca-operator-5bcbc94c78-sp78t\" (UID: \"b5c91d72-2e17-44e7-83ab-4b9ce3a89b43\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-sp78t" Jul 23 09:17:28.596290 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:28.596224 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/59ee07ce-4747-496f-9876-81f2c2358981-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-8zbdn\" (UID: \"59ee07ce-4747-496f-9876-81f2c2358981\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8zbdn" Jul 23 09:17:28.598164 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:28.598142 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6bddaeac-9272-4031-b216-a31fbe58c89d-stats-auth\") pod \"router-default-7494768586-7px6v\" (UID: \"6bddaeac-9272-4031-b216-a31fbe58c89d\") " pod="openshift-ingress/router-default-7494768586-7px6v" Jul 23 09:17:28.598532 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:28.598516 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4908aff3-196e-44fa-b8e6-c0e81b476433-serving-cert\") pod \"console-operator-565b7b569d-5szxg\" (UID: \"4908aff3-196e-44fa-b8e6-c0e81b476433\") " pod="openshift-console-operator/console-operator-565b7b569d-5szxg" Jul 23 09:17:28.598578 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:28.598555 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6bddaeac-9272-4031-b216-a31fbe58c89d-default-certificate\") pod \"router-default-7494768586-7px6v\" (UID: \"6bddaeac-9272-4031-b216-a31fbe58c89d\") " pod="openshift-ingress/router-default-7494768586-7px6v" Jul 23 09:17:28.686756 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:28.686702 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-5szxg" Jul 23 09:17:28.696031 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:28.696012 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a17a8c5e-a4ee-4e04-b1dc-03ec2e43c60f-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-xtr4d\" (UID: \"a17a8c5e-a4ee-4e04-b1dc-03ec2e43c60f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-xtr4d" Jul 23 09:17:28.696100 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:28.696051 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/31bba45e-487f-4c59-ac81-d8e3b90a549d-metrics-tls\") pod \"dns-default-mgk4m\" (UID: \"31bba45e-487f-4c59-ac81-d8e3b90a549d\") " pod="openshift-dns/dns-default-mgk4m" Jul 23 09:17:28.696100 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:28.696075 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h47rf\" (UniqueName: \"kubernetes.io/projected/6bddaeac-9272-4031-b216-a31fbe58c89d-kube-api-access-h47rf\") pod \"router-default-7494768586-7px6v\" (UID: \"6bddaeac-9272-4031-b216-a31fbe58c89d\") " pod="openshift-ingress/router-default-7494768586-7px6v" Jul 23 09:17:28.696198 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:28.696134 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 23 09:17:28.696198 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:28.696161 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 09:17:28.696198 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:28.696182 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a17a8c5e-a4ee-4e04-b1dc-03ec2e43c60f-samples-operator-tls podName:a17a8c5e-a4ee-4e04-b1dc-03ec2e43c60f nodeName:}" failed. No retries permitted until 2026-07-23 09:17:30.696167209 +0000 UTC m=+39.235445763 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a17a8c5e-a4ee-4e04-b1dc-03ec2e43c60f-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-xtr4d" (UID: "a17a8c5e-a4ee-4e04-b1dc-03ec2e43c60f") : secret "samples-operator-tls" not found Jul 23 09:17:28.696328 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:28.696216 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31bba45e-487f-4c59-ac81-d8e3b90a549d-metrics-tls podName:31bba45e-487f-4c59-ac81-d8e3b90a549d nodeName:}" failed. No retries permitted until 2026-07-23 09:17:30.696200622 +0000 UTC m=+39.235479179 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/31bba45e-487f-4c59-ac81-d8e3b90a549d-metrics-tls") pod "dns-default-mgk4m" (UID: "31bba45e-487f-4c59-ac81-d8e3b90a549d") : secret "dns-default-metrics-tls" not found Jul 23 09:17:28.698592 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:28.698566 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h47rf\" (UniqueName: \"kubernetes.io/projected/6bddaeac-9272-4031-b216-a31fbe58c89d-kube-api-access-h47rf\") pod \"router-default-7494768586-7px6v\" (UID: \"6bddaeac-9272-4031-b216-a31fbe58c89d\") " pod="openshift-ingress/router-default-7494768586-7px6v" Jul 23 09:17:28.701168 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:28.701152 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-sp78t" Jul 23 09:17:28.824428 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:28.824404 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-5szxg"] Jul 23 09:17:28.826686 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:17:28.826660 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4908aff3_196e_44fa_b8e6_c0e81b476433.slice/crio-80aeffab344320ddad9ce295a48797375d43ee3fd86cb41a456f42a0e79154ca WatchSource:0}: Error finding container 80aeffab344320ddad9ce295a48797375d43ee3fd86cb41a456f42a0e79154ca: Status 404 returned error can't find the container with id 80aeffab344320ddad9ce295a48797375d43ee3fd86cb41a456f42a0e79154ca Jul 23 09:17:28.841102 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:28.841079 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-sp78t"] Jul 23 09:17:28.843889 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:17:28.843863 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5c91d72_2e17_44e7_83ab_4b9ce3a89b43.slice/crio-9e9dda7710e70191e3d27a6c5e5a321e47fe83dbed91a673453766dd11ac111b WatchSource:0}: Error finding container 9e9dda7710e70191e3d27a6c5e5a321e47fe83dbed91a673453766dd11ac111b: Status 404 returned error can't find the container with id 9e9dda7710e70191e3d27a6c5e5a321e47fe83dbed91a673453766dd11ac111b Jul 23 09:17:29.256290 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:29.256182 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-pxlst" event={"ID":"989625f0-3f97-4f66-8d67-2e8f47a13a6c","Type":"ContainerStarted","Data":"005964fbff5670ec98bc2d7f4b124e5de376fb36a993986e5e8f10a50bd39d5d"} Jul 23 09:17:29.258507 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:29.258433 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-k296x" event={"ID":"e11b7a19-c8e6-4d52-8450-b86a0ca3fcd4","Type":"ContainerStarted","Data":"1996a25520346b21ad8c3c128b25b4b654d138378fc8a106548c21dbcfd39f8c"} Jul 23 09:17:29.260142 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:29.260071 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-b5cz2" event={"ID":"fae77c41-37e2-42c2-92a1-8908061bab7a","Type":"ContainerStarted","Data":"63c7b50a99639efc1ea6664effbb674cdfff8f6f90472ea994595ee67a100bff"} Jul 23 09:17:29.261781 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:29.261751 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-nfv7r" event={"ID":"ef3505b4-c02c-487a-8314-f6793087ad3a","Type":"ContainerStarted","Data":"8e5d661cb649130799811bc604cb5d2fd0efe9dbd04aa195a7dc74fd3ffb5fbb"} Jul 23 09:17:29.264515 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:29.264487 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-cg599" event={"ID":"022429e9-cac7-4ace-ae03-c39b633d6196","Type":"ContainerStarted","Data":"8cfa877f1abe0a352a038cd70531c2b8bfa13b068896510a1c3e528e77b86dba"} Jul 23 09:17:29.268598 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:29.268460 2576 generic.go:356] "Generic (PLEG): container finished" podID="95ff7554-5e90-41c1-9ab5-93d6120788fa" containerID="fe0a4eaa12bace62b7c18fb02188c2b6d50b490823df272b4d27bd36c513091c" exitCode=0 Jul 23 09:17:29.268598 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:29.268580 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xgtzd" event={"ID":"95ff7554-5e90-41c1-9ab5-93d6120788fa","Type":"ContainerDied","Data":"fe0a4eaa12bace62b7c18fb02188c2b6d50b490823df272b4d27bd36c513091c"} Jul 23 09:17:29.271409 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:29.271386 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-5szxg" event={"ID":"4908aff3-196e-44fa-b8e6-c0e81b476433","Type":"ContainerStarted","Data":"80aeffab344320ddad9ce295a48797375d43ee3fd86cb41a456f42a0e79154ca"} Jul 23 09:17:29.274650 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:29.274572 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-sp78t" event={"ID":"b5c91d72-2e17-44e7-83ab-4b9ce3a89b43","Type":"ContainerStarted","Data":"9e9dda7710e70191e3d27a6c5e5a321e47fe83dbed91a673453766dd11ac111b"} Jul 23 09:17:29.405515 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:29.405165 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/59ee07ce-4747-496f-9876-81f2c2358981-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-8zbdn\" (UID: \"59ee07ce-4747-496f-9876-81f2c2358981\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8zbdn" Jul 23 09:17:29.405515 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:29.405458 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 23 09:17:29.405619 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:29.405519 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59ee07ce-4747-496f-9876-81f2c2358981-cluster-monitoring-operator-tls podName:59ee07ce-4747-496f-9876-81f2c2358981 nodeName:}" failed. No retries permitted until 2026-07-23 09:17:31.405499558 +0000 UTC m=+39.944778128 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/59ee07ce-4747-496f-9876-81f2c2358981-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-8zbdn" (UID: "59ee07ce-4747-496f-9876-81f2c2358981") : secret "cluster-monitoring-operator-tls" not found Jul 23 09:17:29.506259 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:29.506204 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6bddaeac-9272-4031-b216-a31fbe58c89d-service-ca-bundle\") pod \"router-default-7494768586-7px6v\" (UID: \"6bddaeac-9272-4031-b216-a31fbe58c89d\") " pod="openshift-ingress/router-default-7494768586-7px6v" Jul 23 09:17:29.506449 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:29.506405 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6bddaeac-9272-4031-b216-a31fbe58c89d-service-ca-bundle podName:6bddaeac-9272-4031-b216-a31fbe58c89d nodeName:}" failed. No retries permitted until 2026-07-23 09:17:31.506385816 +0000 UTC m=+40.045664377 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6bddaeac-9272-4031-b216-a31fbe58c89d-service-ca-bundle") pod "router-default-7494768586-7px6v" (UID: "6bddaeac-9272-4031-b216-a31fbe58c89d") : configmap references non-existent config key: service-ca.crt Jul 23 09:17:29.608596 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:29.608558 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bddaeac-9272-4031-b216-a31fbe58c89d-metrics-certs\") pod \"router-default-7494768586-7px6v\" (UID: \"6bddaeac-9272-4031-b216-a31fbe58c89d\") " pod="openshift-ingress/router-default-7494768586-7px6v" Jul 23 09:17:29.608802 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:29.608782 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 23 09:17:29.608886 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:29.608857 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bddaeac-9272-4031-b216-a31fbe58c89d-metrics-certs podName:6bddaeac-9272-4031-b216-a31fbe58c89d nodeName:}" failed. No retries permitted until 2026-07-23 09:17:31.608836675 +0000 UTC m=+40.148115237 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6bddaeac-9272-4031-b216-a31fbe58c89d-metrics-certs") pod "router-default-7494768586-7px6v" (UID: "6bddaeac-9272-4031-b216-a31fbe58c89d") : secret "router-metrics-certs-default" not found Jul 23 09:17:29.717811 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:29.717754 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:17:29.804819 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:29.804722 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:17:30.290042 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:30.289976 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-k296x" event={"ID":"e11b7a19-c8e6-4d52-8450-b86a0ca3fcd4","Type":"ContainerStarted","Data":"31fc0c414ffc714b640463ab954c9e65ebff0b107cbffaf3b5adcc371e80bf3d"} Jul 23 09:17:30.297160 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:30.296287 2576 generic.go:356] "Generic (PLEG): container finished" podID="95ff7554-5e90-41c1-9ab5-93d6120788fa" containerID="9d1c2b5510fdae78ff2a1628c97cc56db653d06460b58f693228b12e52cb8065" exitCode=0 Jul 23 09:17:30.297160 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:30.296326 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xgtzd" event={"ID":"95ff7554-5e90-41c1-9ab5-93d6120788fa","Type":"ContainerDied","Data":"9d1c2b5510fdae78ff2a1628c97cc56db653d06460b58f693228b12e52cb8065"} Jul 23 09:17:30.621254 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:30.620316 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1260ceaa-4aa5-4130-9ea7-ff8212c2d751-registry-tls\") pod \"image-registry-7989bd97d7-g2rdd\" (UID: \"1260ceaa-4aa5-4130-9ea7-ff8212c2d751\") " pod="openshift-image-registry/image-registry-7989bd97d7-g2rdd" Jul 23 09:17:30.621254 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:30.620815 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 09:17:30.621254 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:30.620833 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7989bd97d7-g2rdd: secret "image-registry-tls" not found Jul 23 09:17:30.621254 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:30.620892 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1260ceaa-4aa5-4130-9ea7-ff8212c2d751-registry-tls podName:1260ceaa-4aa5-4130-9ea7-ff8212c2d751 nodeName:}" failed. No retries permitted until 2026-07-23 09:17:34.620871826 +0000 UTC m=+43.160150382 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1260ceaa-4aa5-4130-9ea7-ff8212c2d751-registry-tls") pod "image-registry-7989bd97d7-g2rdd" (UID: "1260ceaa-4aa5-4130-9ea7-ff8212c2d751") : secret "image-registry-tls" not found Jul 23 09:17:30.721328 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:30.721225 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a17a8c5e-a4ee-4e04-b1dc-03ec2e43c60f-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-xtr4d\" (UID: \"a17a8c5e-a4ee-4e04-b1dc-03ec2e43c60f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-xtr4d" Jul 23 09:17:30.721328 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:30.721310 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/31bba45e-487f-4c59-ac81-d8e3b90a549d-metrics-tls\") pod \"dns-default-mgk4m\" (UID: \"31bba45e-487f-4c59-ac81-d8e3b90a549d\") " pod="openshift-dns/dns-default-mgk4m" Jul 23 09:17:30.721541 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:30.721476 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 09:17:30.721592 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:30.721542 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31bba45e-487f-4c59-ac81-d8e3b90a549d-metrics-tls podName:31bba45e-487f-4c59-ac81-d8e3b90a549d nodeName:}" failed. No retries permitted until 2026-07-23 09:17:34.721522486 +0000 UTC m=+43.260801042 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/31bba45e-487f-4c59-ac81-d8e3b90a549d-metrics-tls") pod "dns-default-mgk4m" (UID: "31bba45e-487f-4c59-ac81-d8e3b90a549d") : secret "dns-default-metrics-tls" not found Jul 23 09:17:30.721973 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:30.721954 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 23 09:17:30.722063 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:30.722013 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a17a8c5e-a4ee-4e04-b1dc-03ec2e43c60f-samples-operator-tls podName:a17a8c5e-a4ee-4e04-b1dc-03ec2e43c60f nodeName:}" failed. No retries permitted until 2026-07-23 09:17:34.721997167 +0000 UTC m=+43.261275722 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a17a8c5e-a4ee-4e04-b1dc-03ec2e43c60f-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-xtr4d" (UID: "a17a8c5e-a4ee-4e04-b1dc-03ec2e43c60f") : secret "samples-operator-tls" not found Jul 23 09:17:31.305822 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:31.305774 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xgtzd" event={"ID":"95ff7554-5e90-41c1-9ab5-93d6120788fa","Type":"ContainerStarted","Data":"98e0b8bf04a6be2c7240bfee95c36b312210c276d779cdb33909ee58ec18d055"} Jul 23 09:17:31.369439 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:31.369344 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-xgtzd" podStartSLOduration=7.949687778 podStartE2EDuration="38.369328264s" totalImagesPullingTime="30.419640486s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 09:16:53 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:17:31.368143567 +0000 UTC m=+39.907422146" watchObservedRunningTime="2026-07-23 09:17:31.369328264 +0000 UTC m=+39.908606841" Jul 23 09:17:31.429690 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:31.429650 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/59ee07ce-4747-496f-9876-81f2c2358981-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-8zbdn\" (UID: \"59ee07ce-4747-496f-9876-81f2c2358981\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8zbdn" Jul 23 09:17:31.429879 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:31.429799 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 23 09:17:31.429879 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:31.429871 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59ee07ce-4747-496f-9876-81f2c2358981-cluster-monitoring-operator-tls podName:59ee07ce-4747-496f-9876-81f2c2358981 nodeName:}" failed. No retries permitted until 2026-07-23 09:17:35.429851262 +0000 UTC m=+43.969129818 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/59ee07ce-4747-496f-9876-81f2c2358981-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-8zbdn" (UID: "59ee07ce-4747-496f-9876-81f2c2358981") : secret "cluster-monitoring-operator-tls" not found Jul 23 09:17:31.508465 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:31.508411 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:17:31.530614 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:31.530582 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6bddaeac-9272-4031-b216-a31fbe58c89d-service-ca-bundle\") pod \"router-default-7494768586-7px6v\" (UID: \"6bddaeac-9272-4031-b216-a31fbe58c89d\") " pod="openshift-ingress/router-default-7494768586-7px6v" Jul 23 09:17:31.530808 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:31.530792 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6bddaeac-9272-4031-b216-a31fbe58c89d-service-ca-bundle podName:6bddaeac-9272-4031-b216-a31fbe58c89d nodeName:}" failed. No retries permitted until 2026-07-23 09:17:35.530774709 +0000 UTC m=+44.070053267 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6bddaeac-9272-4031-b216-a31fbe58c89d-service-ca-bundle") pod "router-default-7494768586-7px6v" (UID: "6bddaeac-9272-4031-b216-a31fbe58c89d") : configmap references non-existent config key: service-ca.crt Jul 23 09:17:31.610253 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:31.610217 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:17:31.631563 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:31.631532 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bddaeac-9272-4031-b216-a31fbe58c89d-metrics-certs\") pod \"router-default-7494768586-7px6v\" (UID: \"6bddaeac-9272-4031-b216-a31fbe58c89d\") " pod="openshift-ingress/router-default-7494768586-7px6v" Jul 23 09:17:31.631722 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:31.631703 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 23 09:17:31.631794 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:31.631779 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bddaeac-9272-4031-b216-a31fbe58c89d-metrics-certs podName:6bddaeac-9272-4031-b216-a31fbe58c89d nodeName:}" failed. No retries permitted until 2026-07-23 09:17:35.631758933 +0000 UTC m=+44.171037511 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6bddaeac-9272-4031-b216-a31fbe58c89d-metrics-certs") pod "router-default-7494768586-7px6v" (UID: "6bddaeac-9272-4031-b216-a31fbe58c89d") : secret "router-metrics-certs-default" not found Jul 23 09:17:31.708958 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:31.708916 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:17:34.660067 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:34.660029 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1260ceaa-4aa5-4130-9ea7-ff8212c2d751-registry-tls\") pod \"image-registry-7989bd97d7-g2rdd\" (UID: \"1260ceaa-4aa5-4130-9ea7-ff8212c2d751\") " pod="openshift-image-registry/image-registry-7989bd97d7-g2rdd" Jul 23 09:17:34.660647 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:34.660208 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 09:17:34.660647 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:34.660234 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7989bd97d7-g2rdd: secret "image-registry-tls" not found Jul 23 09:17:34.660647 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:34.660324 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1260ceaa-4aa5-4130-9ea7-ff8212c2d751-registry-tls podName:1260ceaa-4aa5-4130-9ea7-ff8212c2d751 nodeName:}" failed. No retries permitted until 2026-07-23 09:17:42.660300391 +0000 UTC m=+51.199578970 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1260ceaa-4aa5-4130-9ea7-ff8212c2d751-registry-tls") pod "image-registry-7989bd97d7-g2rdd" (UID: "1260ceaa-4aa5-4130-9ea7-ff8212c2d751") : secret "image-registry-tls" not found Jul 23 09:17:34.760746 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:34.760708 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a17a8c5e-a4ee-4e04-b1dc-03ec2e43c60f-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-xtr4d\" (UID: \"a17a8c5e-a4ee-4e04-b1dc-03ec2e43c60f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-xtr4d" Jul 23 09:17:34.760929 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:34.760771 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/31bba45e-487f-4c59-ac81-d8e3b90a549d-metrics-tls\") pod \"dns-default-mgk4m\" (UID: \"31bba45e-487f-4c59-ac81-d8e3b90a549d\") " pod="openshift-dns/dns-default-mgk4m" Jul 23 09:17:34.760929 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:34.760891 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 23 09:17:34.760929 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:34.760923 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 09:17:34.761081 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:34.760973 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a17a8c5e-a4ee-4e04-b1dc-03ec2e43c60f-samples-operator-tls podName:a17a8c5e-a4ee-4e04-b1dc-03ec2e43c60f nodeName:}" failed. No retries permitted until 2026-07-23 09:17:42.760951632 +0000 UTC m=+51.300230211 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a17a8c5e-a4ee-4e04-b1dc-03ec2e43c60f-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-xtr4d" (UID: "a17a8c5e-a4ee-4e04-b1dc-03ec2e43c60f") : secret "samples-operator-tls" not found Jul 23 09:17:34.761081 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:34.760993 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31bba45e-487f-4c59-ac81-d8e3b90a549d-metrics-tls podName:31bba45e-487f-4c59-ac81-d8e3b90a549d nodeName:}" failed. No retries permitted until 2026-07-23 09:17:42.760982787 +0000 UTC m=+51.300261348 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/31bba45e-487f-4c59-ac81-d8e3b90a549d-metrics-tls") pod "dns-default-mgk4m" (UID: "31bba45e-487f-4c59-ac81-d8e3b90a549d") : secret "dns-default-metrics-tls" not found Jul 23 09:17:35.269341 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:35.269299 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:17:35.339166 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:35.339119 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:17:35.448005 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:35.447972 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:17:35.466950 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:35.466919 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/59ee07ce-4747-496f-9876-81f2c2358981-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-8zbdn\" (UID: \"59ee07ce-4747-496f-9876-81f2c2358981\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8zbdn" Jul 23 09:17:35.467117 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:35.467086 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 23 09:17:35.467181 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:35.467162 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59ee07ce-4747-496f-9876-81f2c2358981-cluster-monitoring-operator-tls podName:59ee07ce-4747-496f-9876-81f2c2358981 nodeName:}" failed. No retries permitted until 2026-07-23 09:17:43.467141892 +0000 UTC m=+52.006420454 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/59ee07ce-4747-496f-9876-81f2c2358981-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-8zbdn" (UID: "59ee07ce-4747-496f-9876-81f2c2358981") : secret "cluster-monitoring-operator-tls" not found Jul 23 09:17:35.548811 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:35.548728 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:17:35.568351 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:35.568318 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6bddaeac-9272-4031-b216-a31fbe58c89d-service-ca-bundle\") pod \"router-default-7494768586-7px6v\" (UID: \"6bddaeac-9272-4031-b216-a31fbe58c89d\") " pod="openshift-ingress/router-default-7494768586-7px6v" Jul 23 09:17:35.568521 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:35.568485 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6bddaeac-9272-4031-b216-a31fbe58c89d-service-ca-bundle podName:6bddaeac-9272-4031-b216-a31fbe58c89d nodeName:}" failed. No retries permitted until 2026-07-23 09:17:43.568464831 +0000 UTC m=+52.107743385 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6bddaeac-9272-4031-b216-a31fbe58c89d-service-ca-bundle") pod "router-default-7494768586-7px6v" (UID: "6bddaeac-9272-4031-b216-a31fbe58c89d") : configmap references non-existent config key: service-ca.crt Jul 23 09:17:35.669322 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:35.669289 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bddaeac-9272-4031-b216-a31fbe58c89d-metrics-certs\") pod \"router-default-7494768586-7px6v\" (UID: \"6bddaeac-9272-4031-b216-a31fbe58c89d\") " pod="openshift-ingress/router-default-7494768586-7px6v" Jul 23 09:17:35.669666 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:35.669403 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 23 09:17:35.669666 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:35.669459 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bddaeac-9272-4031-b216-a31fbe58c89d-metrics-certs podName:6bddaeac-9272-4031-b216-a31fbe58c89d nodeName:}" failed. No retries permitted until 2026-07-23 09:17:43.669442192 +0000 UTC m=+52.208720768 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6bddaeac-9272-4031-b216-a31fbe58c89d-metrics-certs") pod "router-default-7494768586-7px6v" (UID: "6bddaeac-9272-4031-b216-a31fbe58c89d") : secret "router-metrics-certs-default" not found Jul 23 09:17:36.127855 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:36.127789 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:17:36.224893 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:36.224864 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:17:36.310168 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:36.310135 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:17:36.616017 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:36.615945 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:17:36.699543 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:36.699457 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:17:37.153922 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:37.153838 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-k296x" podStartSLOduration=12.382501945 podStartE2EDuration="14.153817701s" totalImagesPullingTime="1.771315756s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:17:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:17:31.38979141 +0000 UTC m=+39.929069988" watchObservedRunningTime="2026-07-23 09:17:37.153817701 +0000 UTC m=+45.693096278" Jul 23 09:17:37.155715 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:37.155689 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-kwb49"] Jul 23 09:17:37.179448 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:37.179410 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kwb49"] Jul 23 09:17:37.179595 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:37.179570 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kwb49" Jul 23 09:17:37.182895 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:37.182349 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 23 09:17:37.183408 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:37.183375 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-sjj26\"" Jul 23 09:17:37.183748 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:37.183510 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 23 09:17:37.207663 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:37.207635 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:17:37.284207 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:37.284173 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/81ee471f-b215-43bd-a279-9ee79d67cd2b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kwb49\" (UID: \"81ee471f-b215-43bd-a279-9ee79d67cd2b\") " pod="openshift-insights/insights-runtime-extractor-kwb49" Jul 23 09:17:37.284398 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:37.284235 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/81ee471f-b215-43bd-a279-9ee79d67cd2b-crio-socket\") pod \"insights-runtime-extractor-kwb49\" (UID: \"81ee471f-b215-43bd-a279-9ee79d67cd2b\") " pod="openshift-insights/insights-runtime-extractor-kwb49" Jul 23 09:17:37.284461 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:37.284409 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/81ee471f-b215-43bd-a279-9ee79d67cd2b-data-volume\") pod \"insights-runtime-extractor-kwb49\" (UID: \"81ee471f-b215-43bd-a279-9ee79d67cd2b\") " pod="openshift-insights/insights-runtime-extractor-kwb49" Jul 23 09:17:37.284516 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:37.284491 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/81ee471f-b215-43bd-a279-9ee79d67cd2b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kwb49\" (UID: \"81ee471f-b215-43bd-a279-9ee79d67cd2b\") " pod="openshift-insights/insights-runtime-extractor-kwb49" Jul 23 09:17:37.284554 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:37.284525 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcw76\" (UniqueName: \"kubernetes.io/projected/81ee471f-b215-43bd-a279-9ee79d67cd2b-kube-api-access-pcw76\") pod \"insights-runtime-extractor-kwb49\" (UID: \"81ee471f-b215-43bd-a279-9ee79d67cd2b\") " pod="openshift-insights/insights-runtime-extractor-kwb49" Jul 23 09:17:37.305779 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:37.305743 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:17:37.323406 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:37.323368 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-b5cz2" event={"ID":"fae77c41-37e2-42c2-92a1-8908061bab7a","Type":"ContainerStarted","Data":"bd3c996819a80e6c30eb135eab4640f8076d9fd76ae77663635b8eec72b35fba"} Jul 23 09:17:37.325175 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:37.325145 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-nfv7r" event={"ID":"ef3505b4-c02c-487a-8314-f6793087ad3a","Type":"ContainerStarted","Data":"3f2f1f1a0b9799ea38e9b88cf3926d880b58c666ff994e48992234b54d8281bf"} Jul 23 09:17:37.326945 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:37.326907 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-cg599" event={"ID":"022429e9-cac7-4ace-ae03-c39b633d6196","Type":"ContainerStarted","Data":"107955ed1a127f7acdae656507e264dfd45d02eb8c3ba6ce210c6c28abe5c920"} Jul 23 09:17:37.328784 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:37.328767 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-5szxg_4908aff3-196e-44fa-b8e6-c0e81b476433/console-operator/0.log" Jul 23 09:17:37.328887 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:37.328798 2576 generic.go:356] "Generic (PLEG): container finished" podID="4908aff3-196e-44fa-b8e6-c0e81b476433" containerID="8d1cc8f0dfb234a8c24b5491ad3d6829f9aed4f06fc252803ea0030fdd0ccd66" exitCode=255 Jul 23 09:17:37.328887 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:37.328824 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-5szxg" event={"ID":"4908aff3-196e-44fa-b8e6-c0e81b476433","Type":"ContainerDied","Data":"8d1cc8f0dfb234a8c24b5491ad3d6829f9aed4f06fc252803ea0030fdd0ccd66"} Jul 23 09:17:37.329054 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:37.329035 2576 scope.go:117] "RemoveContainer" containerID="8d1cc8f0dfb234a8c24b5491ad3d6829f9aed4f06fc252803ea0030fdd0ccd66" Jul 23 09:17:37.330668 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:37.330584 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-sp78t" event={"ID":"b5c91d72-2e17-44e7-83ab-4b9ce3a89b43","Type":"ContainerStarted","Data":"6a69790c16310ccc73f0eff984cf246790eec31b639f512ab59121de140d602b"} Jul 23 09:17:37.332176 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:37.332157 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-pxlst" event={"ID":"989625f0-3f97-4f66-8d67-2e8f47a13a6c","Type":"ContainerStarted","Data":"950301a33c82b5d5b3316bfd0c420dffe3d528f0ce62f801a8a6d18ff1ded372"} Jul 23 09:17:37.342893 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:37.342850 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-b5cz2" podStartSLOduration=5.459044355 podStartE2EDuration="13.342833286s" totalImagesPullingTime="7.883788931s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:17:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:17:37.341685029 +0000 UTC m=+45.880963608" watchObservedRunningTime="2026-07-23 09:17:37.342833286 +0000 UTC m=+45.882111847" Jul 23 09:17:37.361569 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:37.361520 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-555d9d7885-pxlst" podStartSLOduration=5.488154824 podStartE2EDuration="13.361501951s" totalImagesPullingTime="7.873347127s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:17:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:17:37.360476063 +0000 UTC m=+45.899754641" watchObservedRunningTime="2026-07-23 09:17:37.361501951 +0000 UTC m=+45.900780535" Jul 23 09:17:37.378191 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:37.378145 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-cg599" podStartSLOduration=3.492121892 podStartE2EDuration="11.378126026s" totalImagesPullingTime="7.886004134s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:17:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:17:37.37779847 +0000 UTC m=+45.917077048" watchObservedRunningTime="2026-07-23 09:17:37.378126026 +0000 UTC m=+45.917404605" Jul 23 09:17:37.385567 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:37.385540 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcw76\" (UniqueName: \"kubernetes.io/projected/81ee471f-b215-43bd-a279-9ee79d67cd2b-kube-api-access-pcw76\") pod \"insights-runtime-extractor-kwb49\" (UID: \"81ee471f-b215-43bd-a279-9ee79d67cd2b\") " pod="openshift-insights/insights-runtime-extractor-kwb49" Jul 23 09:17:37.387294 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:37.385718 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/81ee471f-b215-43bd-a279-9ee79d67cd2b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kwb49\" (UID: \"81ee471f-b215-43bd-a279-9ee79d67cd2b\") " pod="openshift-insights/insights-runtime-extractor-kwb49" Jul 23 09:17:37.387294 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:37.385763 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/81ee471f-b215-43bd-a279-9ee79d67cd2b-crio-socket\") pod \"insights-runtime-extractor-kwb49\" (UID: \"81ee471f-b215-43bd-a279-9ee79d67cd2b\") " pod="openshift-insights/insights-runtime-extractor-kwb49" Jul 23 09:17:37.387294 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:37.385946 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/81ee471f-b215-43bd-a279-9ee79d67cd2b-data-volume\") pod \"insights-runtime-extractor-kwb49\" (UID: \"81ee471f-b215-43bd-a279-9ee79d67cd2b\") " pod="openshift-insights/insights-runtime-extractor-kwb49" Jul 23 09:17:37.387294 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:37.385991 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/81ee471f-b215-43bd-a279-9ee79d67cd2b-crio-socket\") pod \"insights-runtime-extractor-kwb49\" (UID: \"81ee471f-b215-43bd-a279-9ee79d67cd2b\") " pod="openshift-insights/insights-runtime-extractor-kwb49" Jul 23 09:17:37.387294 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:37.386107 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/81ee471f-b215-43bd-a279-9ee79d67cd2b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kwb49\" (UID: \"81ee471f-b215-43bd-a279-9ee79d67cd2b\") " pod="openshift-insights/insights-runtime-extractor-kwb49" Jul 23 09:17:37.387294 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:37.386190 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 23 09:17:37.387294 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:37.386294 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/81ee471f-b215-43bd-a279-9ee79d67cd2b-kube-rbac-proxy-cm podName:81ee471f-b215-43bd-a279-9ee79d67cd2b nodeName:}" failed. No retries permitted until 2026-07-23 09:17:37.886223862 +0000 UTC m=+46.425502419 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/81ee471f-b215-43bd-a279-9ee79d67cd2b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kwb49" (UID: "81ee471f-b215-43bd-a279-9ee79d67cd2b") : configmap "kube-rbac-proxy" not found Jul 23 09:17:37.387294 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:37.386571 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 09:17:37.387294 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:37.386621 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81ee471f-b215-43bd-a279-9ee79d67cd2b-insights-runtime-extractor-tls podName:81ee471f-b215-43bd-a279-9ee79d67cd2b nodeName:}" failed. No retries permitted until 2026-07-23 09:17:37.886604663 +0000 UTC m=+46.425883225 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/81ee471f-b215-43bd-a279-9ee79d67cd2b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kwb49" (UID: "81ee471f-b215-43bd-a279-9ee79d67cd2b") : secret "insights-runtime-extractor-tls" not found Jul 23 09:17:37.387294 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:37.387016 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/81ee471f-b215-43bd-a279-9ee79d67cd2b-data-volume\") pod \"insights-runtime-extractor-kwb49\" (UID: \"81ee471f-b215-43bd-a279-9ee79d67cd2b\") " pod="openshift-insights/insights-runtime-extractor-kwb49" Jul 23 09:17:37.396666 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:37.396619 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcw76\" (UniqueName: \"kubernetes.io/projected/81ee471f-b215-43bd-a279-9ee79d67cd2b-kube-api-access-pcw76\") pod \"insights-runtime-extractor-kwb49\" (UID: \"81ee471f-b215-43bd-a279-9ee79d67cd2b\") " pod="openshift-insights/insights-runtime-extractor-kwb49" Jul 23 09:17:37.419230 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:37.419149 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-sp78t" podStartSLOduration=7.021728209 podStartE2EDuration="14.419131177s" totalImagesPullingTime="7.397402968s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:17:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:17:37.417631721 +0000 UTC m=+45.956910299" watchObservedRunningTime="2026-07-23 09:17:37.419131177 +0000 UTC m=+45.958409755" Jul 23 09:17:37.434341 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:37.434302 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-nfv7r" podStartSLOduration=5.741576597 podStartE2EDuration="13.434288294s" totalImagesPullingTime="7.692711697s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:17:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:17:37.433605418 +0000 UTC m=+45.972883996" watchObservedRunningTime="2026-07-23 09:17:37.434288294 +0000 UTC m=+45.973566876" Jul 23 09:17:37.833169 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:37.833086 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:17:37.891111 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:37.891079 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/81ee471f-b215-43bd-a279-9ee79d67cd2b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kwb49\" (UID: \"81ee471f-b215-43bd-a279-9ee79d67cd2b\") " pod="openshift-insights/insights-runtime-extractor-kwb49" Jul 23 09:17:37.891279 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:37.891200 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 23 09:17:37.891279 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:37.891224 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/81ee471f-b215-43bd-a279-9ee79d67cd2b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kwb49\" (UID: \"81ee471f-b215-43bd-a279-9ee79d67cd2b\") " pod="openshift-insights/insights-runtime-extractor-kwb49" Jul 23 09:17:37.891396 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:37.891282 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/81ee471f-b215-43bd-a279-9ee79d67cd2b-kube-rbac-proxy-cm podName:81ee471f-b215-43bd-a279-9ee79d67cd2b nodeName:}" failed. No retries permitted until 2026-07-23 09:17:38.891263388 +0000 UTC m=+47.430541945 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/81ee471f-b215-43bd-a279-9ee79d67cd2b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kwb49" (UID: "81ee471f-b215-43bd-a279-9ee79d67cd2b") : configmap "kube-rbac-proxy" not found Jul 23 09:17:37.891396 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:37.891341 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 09:17:37.891515 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:37.891401 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81ee471f-b215-43bd-a279-9ee79d67cd2b-insights-runtime-extractor-tls podName:81ee471f-b215-43bd-a279-9ee79d67cd2b nodeName:}" failed. No retries permitted until 2026-07-23 09:17:38.891385058 +0000 UTC m=+47.430663635 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/81ee471f-b215-43bd-a279-9ee79d67cd2b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kwb49" (UID: "81ee471f-b215-43bd-a279-9ee79d67cd2b") : secret "insights-runtime-extractor-tls" not found Jul 23 09:17:37.961015 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:37.960982 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:17:38.094935 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:38.094860 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:17:38.264742 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:38.264712 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:17:38.337121 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:38.337091 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-5szxg_4908aff3-196e-44fa-b8e6-c0e81b476433/console-operator/1.log" Jul 23 09:17:38.337681 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:38.337651 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-5szxg_4908aff3-196e-44fa-b8e6-c0e81b476433/console-operator/0.log" Jul 23 09:17:38.337803 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:38.337711 2576 generic.go:356] "Generic (PLEG): container finished" podID="4908aff3-196e-44fa-b8e6-c0e81b476433" containerID="9212332a19a6600cb44a1fb65dad1c5d95eabf99bb6488aa3b6603b7409fbe6f" exitCode=255 Jul 23 09:17:38.339011 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:38.338655 2576 scope.go:117] "RemoveContainer" containerID="9212332a19a6600cb44a1fb65dad1c5d95eabf99bb6488aa3b6603b7409fbe6f" Jul 23 09:17:38.339011 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:38.338824 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-5szxg_openshift-console-operator(4908aff3-196e-44fa-b8e6-c0e81b476433)\"" pod="openshift-console-operator/console-operator-565b7b569d-5szxg" podUID="4908aff3-196e-44fa-b8e6-c0e81b476433" Jul 23 09:17:38.339225 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:38.339087 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-5szxg" event={"ID":"4908aff3-196e-44fa-b8e6-c0e81b476433","Type":"ContainerDied","Data":"9212332a19a6600cb44a1fb65dad1c5d95eabf99bb6488aa3b6603b7409fbe6f"} Jul 23 09:17:38.339225 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:38.339177 2576 scope.go:117] "RemoveContainer" containerID="8d1cc8f0dfb234a8c24b5491ad3d6829f9aed4f06fc252803ea0030fdd0ccd66" Jul 23 09:17:38.510399 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:38.510367 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:17:38.586414 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:38.586380 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:17:38.687215 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:38.687181 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-5szxg" Jul 23 09:17:38.687398 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:38.687229 2576 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-565b7b569d-5szxg" Jul 23 09:17:38.780777 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:38.780739 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:17:38.882466 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:38.882436 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:17:38.902304 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:38.902271 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/81ee471f-b215-43bd-a279-9ee79d67cd2b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kwb49\" (UID: \"81ee471f-b215-43bd-a279-9ee79d67cd2b\") " pod="openshift-insights/insights-runtime-extractor-kwb49" Jul 23 09:17:38.902393 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:38.902337 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/81ee471f-b215-43bd-a279-9ee79d67cd2b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kwb49\" (UID: \"81ee471f-b215-43bd-a279-9ee79d67cd2b\") " pod="openshift-insights/insights-runtime-extractor-kwb49" Jul 23 09:17:38.902459 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:38.902395 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 23 09:17:38.902459 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:38.902434 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 09:17:38.902459 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:38.902458 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/81ee471f-b215-43bd-a279-9ee79d67cd2b-kube-rbac-proxy-cm podName:81ee471f-b215-43bd-a279-9ee79d67cd2b nodeName:}" failed. No retries permitted until 2026-07-23 09:17:40.902442691 +0000 UTC m=+49.441721246 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/81ee471f-b215-43bd-a279-9ee79d67cd2b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kwb49" (UID: "81ee471f-b215-43bd-a279-9ee79d67cd2b") : configmap "kube-rbac-proxy" not found Jul 23 09:17:38.902570 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:38.902472 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81ee471f-b215-43bd-a279-9ee79d67cd2b-insights-runtime-extractor-tls podName:81ee471f-b215-43bd-a279-9ee79d67cd2b nodeName:}" failed. No retries permitted until 2026-07-23 09:17:40.902466889 +0000 UTC m=+49.441745443 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/81ee471f-b215-43bd-a279-9ee79d67cd2b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kwb49" (UID: "81ee471f-b215-43bd-a279-9ee79d67cd2b") : secret "insights-runtime-extractor-tls" not found Jul 23 09:17:39.229727 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:39.229699 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gf7qk_72bcfd4f-fce8-46b6-b3f1-52e2ba6306fa/dns-node-resolver/0.log" Jul 23 09:17:39.342217 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:39.342193 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-5szxg_4908aff3-196e-44fa-b8e6-c0e81b476433/console-operator/1.log" Jul 23 09:17:39.342584 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:39.342568 2576 scope.go:117] "RemoveContainer" containerID="9212332a19a6600cb44a1fb65dad1c5d95eabf99bb6488aa3b6603b7409fbe6f" Jul 23 09:17:39.342769 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:39.342748 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-5szxg_openshift-console-operator(4908aff3-196e-44fa-b8e6-c0e81b476433)\"" pod="openshift-console-operator/console-operator-565b7b569d-5szxg" podUID="4908aff3-196e-44fa-b8e6-c0e81b476433" Jul 23 09:17:40.029675 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:40.029646 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9r4c5_d676a11c-2214-4008-8801-efeb43be1ded/node-ca/0.log" Jul 23 09:17:40.346204 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:40.346124 2576 scope.go:117] "RemoveContainer" containerID="9212332a19a6600cb44a1fb65dad1c5d95eabf99bb6488aa3b6603b7409fbe6f" Jul 23 09:17:40.346386 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:40.346336 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-5szxg_openshift-console-operator(4908aff3-196e-44fa-b8e6-c0e81b476433)\"" pod="openshift-console-operator/console-operator-565b7b569d-5szxg" podUID="4908aff3-196e-44fa-b8e6-c0e81b476433" Jul 23 09:17:40.409069 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:40.409018 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:17:40.531253 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:40.531210 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:17:40.635219 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:40.635192 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:17:40.739144 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:40.739117 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:17:40.829813 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:40.829792 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-cg599_022429e9-cac7-4ace-ae03-c39b633d6196/serve-healthcheck-canary/0.log" Jul 23 09:17:40.830989 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:40.830966 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:17:40.920713 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:40.920654 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/81ee471f-b215-43bd-a279-9ee79d67cd2b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kwb49\" (UID: \"81ee471f-b215-43bd-a279-9ee79d67cd2b\") " pod="openshift-insights/insights-runtime-extractor-kwb49" Jul 23 09:17:40.920823 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:40.920718 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/81ee471f-b215-43bd-a279-9ee79d67cd2b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kwb49\" (UID: \"81ee471f-b215-43bd-a279-9ee79d67cd2b\") " pod="openshift-insights/insights-runtime-extractor-kwb49" Jul 23 09:17:40.920823 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:40.920786 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 23 09:17:40.920823 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:40.920813 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 09:17:40.920918 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:40.920862 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/81ee471f-b215-43bd-a279-9ee79d67cd2b-kube-rbac-proxy-cm podName:81ee471f-b215-43bd-a279-9ee79d67cd2b nodeName:}" failed. No retries permitted until 2026-07-23 09:17:44.920844697 +0000 UTC m=+53.460123271 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/81ee471f-b215-43bd-a279-9ee79d67cd2b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kwb49" (UID: "81ee471f-b215-43bd-a279-9ee79d67cd2b") : configmap "kube-rbac-proxy" not found Jul 23 09:17:40.920918 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:40.920878 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81ee471f-b215-43bd-a279-9ee79d67cd2b-insights-runtime-extractor-tls podName:81ee471f-b215-43bd-a279-9ee79d67cd2b nodeName:}" failed. No retries permitted until 2026-07-23 09:17:44.920871912 +0000 UTC m=+53.460150467 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/81ee471f-b215-43bd-a279-9ee79d67cd2b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kwb49" (UID: "81ee471f-b215-43bd-a279-9ee79d67cd2b") : secret "insights-runtime-extractor-tls" not found Jul 23 09:17:40.931775 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:40.931755 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:17:41.419251 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:41.419206 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:17:42.736716 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:42.736681 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1260ceaa-4aa5-4130-9ea7-ff8212c2d751-registry-tls\") pod \"image-registry-7989bd97d7-g2rdd\" (UID: \"1260ceaa-4aa5-4130-9ea7-ff8212c2d751\") " pod="openshift-image-registry/image-registry-7989bd97d7-g2rdd" Jul 23 09:17:42.737223 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:42.736872 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 09:17:42.737223 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:42.736898 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7989bd97d7-g2rdd: secret "image-registry-tls" not found Jul 23 09:17:42.737223 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:42.736987 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1260ceaa-4aa5-4130-9ea7-ff8212c2d751-registry-tls podName:1260ceaa-4aa5-4130-9ea7-ff8212c2d751 nodeName:}" failed. No retries permitted until 2026-07-23 09:17:58.736964072 +0000 UTC m=+67.276242642 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1260ceaa-4aa5-4130-9ea7-ff8212c2d751-registry-tls") pod "image-registry-7989bd97d7-g2rdd" (UID: "1260ceaa-4aa5-4130-9ea7-ff8212c2d751") : secret "image-registry-tls" not found Jul 23 09:17:42.837910 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:42.837870 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a17a8c5e-a4ee-4e04-b1dc-03ec2e43c60f-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-xtr4d\" (UID: \"a17a8c5e-a4ee-4e04-b1dc-03ec2e43c60f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-xtr4d" Jul 23 09:17:42.838084 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:42.837918 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/31bba45e-487f-4c59-ac81-d8e3b90a549d-metrics-tls\") pod \"dns-default-mgk4m\" (UID: \"31bba45e-487f-4c59-ac81-d8e3b90a549d\") " pod="openshift-dns/dns-default-mgk4m" Jul 23 09:17:42.838084 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:42.838027 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 23 09:17:42.838175 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:42.838083 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 09:17:42.838175 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:42.838093 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a17a8c5e-a4ee-4e04-b1dc-03ec2e43c60f-samples-operator-tls podName:a17a8c5e-a4ee-4e04-b1dc-03ec2e43c60f nodeName:}" failed. No retries permitted until 2026-07-23 09:17:58.838077463 +0000 UTC m=+67.377356022 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a17a8c5e-a4ee-4e04-b1dc-03ec2e43c60f-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-xtr4d" (UID: "a17a8c5e-a4ee-4e04-b1dc-03ec2e43c60f") : secret "samples-operator-tls" not found Jul 23 09:17:42.838175 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:42.838138 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31bba45e-487f-4c59-ac81-d8e3b90a549d-metrics-tls podName:31bba45e-487f-4c59-ac81-d8e3b90a549d nodeName:}" failed. No retries permitted until 2026-07-23 09:17:58.838124813 +0000 UTC m=+67.377403371 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/31bba45e-487f-4c59-ac81-d8e3b90a549d-metrics-tls") pod "dns-default-mgk4m" (UID: "31bba45e-487f-4c59-ac81-d8e3b90a549d") : secret "dns-default-metrics-tls" not found Jul 23 09:17:43.544404 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:43.544369 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/59ee07ce-4747-496f-9876-81f2c2358981-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-8zbdn\" (UID: \"59ee07ce-4747-496f-9876-81f2c2358981\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8zbdn" Jul 23 09:17:43.544572 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:43.544520 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 23 09:17:43.544618 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:43.544597 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59ee07ce-4747-496f-9876-81f2c2358981-cluster-monitoring-operator-tls podName:59ee07ce-4747-496f-9876-81f2c2358981 nodeName:}" failed. No retries permitted until 2026-07-23 09:17:59.544581689 +0000 UTC m=+68.083860244 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/59ee07ce-4747-496f-9876-81f2c2358981-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-8zbdn" (UID: "59ee07ce-4747-496f-9876-81f2c2358981") : secret "cluster-monitoring-operator-tls" not found Jul 23 09:17:43.644763 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:43.644731 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6bddaeac-9272-4031-b216-a31fbe58c89d-service-ca-bundle\") pod \"router-default-7494768586-7px6v\" (UID: \"6bddaeac-9272-4031-b216-a31fbe58c89d\") " pod="openshift-ingress/router-default-7494768586-7px6v" Jul 23 09:17:43.644920 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:43.644901 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6bddaeac-9272-4031-b216-a31fbe58c89d-service-ca-bundle podName:6bddaeac-9272-4031-b216-a31fbe58c89d nodeName:}" failed. No retries permitted until 2026-07-23 09:17:59.644883996 +0000 UTC m=+68.184162552 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6bddaeac-9272-4031-b216-a31fbe58c89d-service-ca-bundle") pod "router-default-7494768586-7px6v" (UID: "6bddaeac-9272-4031-b216-a31fbe58c89d") : configmap references non-existent config key: service-ca.crt Jul 23 09:17:43.746142 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:43.746110 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bddaeac-9272-4031-b216-a31fbe58c89d-metrics-certs\") pod \"router-default-7494768586-7px6v\" (UID: \"6bddaeac-9272-4031-b216-a31fbe58c89d\") " pod="openshift-ingress/router-default-7494768586-7px6v" Jul 23 09:17:43.746513 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:43.746273 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 23 09:17:43.746513 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:43.746333 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bddaeac-9272-4031-b216-a31fbe58c89d-metrics-certs podName:6bddaeac-9272-4031-b216-a31fbe58c89d nodeName:}" failed. No retries permitted until 2026-07-23 09:17:59.746319416 +0000 UTC m=+68.285597971 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6bddaeac-9272-4031-b216-a31fbe58c89d-metrics-certs") pod "router-default-7494768586-7px6v" (UID: "6bddaeac-9272-4031-b216-a31fbe58c89d") : secret "router-metrics-certs-default" not found Jul 23 09:17:44.187434 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:44.187404 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:17:44.956389 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:44.956355 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/81ee471f-b215-43bd-a279-9ee79d67cd2b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kwb49\" (UID: \"81ee471f-b215-43bd-a279-9ee79d67cd2b\") " pod="openshift-insights/insights-runtime-extractor-kwb49" Jul 23 09:17:44.956759 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:44.956471 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/81ee471f-b215-43bd-a279-9ee79d67cd2b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kwb49\" (UID: \"81ee471f-b215-43bd-a279-9ee79d67cd2b\") " pod="openshift-insights/insights-runtime-extractor-kwb49" Jul 23 09:17:44.956759 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:44.956504 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 09:17:44.956759 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:44.956565 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81ee471f-b215-43bd-a279-9ee79d67cd2b-insights-runtime-extractor-tls podName:81ee471f-b215-43bd-a279-9ee79d67cd2b nodeName:}" failed. No retries permitted until 2026-07-23 09:17:52.956550338 +0000 UTC m=+61.495828893 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/81ee471f-b215-43bd-a279-9ee79d67cd2b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kwb49" (UID: "81ee471f-b215-43bd-a279-9ee79d67cd2b") : secret "insights-runtime-extractor-tls" not found Jul 23 09:17:44.956963 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:44.956946 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/81ee471f-b215-43bd-a279-9ee79d67cd2b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kwb49\" (UID: \"81ee471f-b215-43bd-a279-9ee79d67cd2b\") " pod="openshift-insights/insights-runtime-extractor-kwb49" Jul 23 09:17:45.150985 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:45.150953 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:17:45.329576 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:45.329497 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:17:45.439605 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:45.439570 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:17:45.526035 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:45.526010 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:17:45.626554 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:45.626516 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:17:45.733878 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:45.733848 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:17:45.877306 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:45.877195 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:17:46.643272 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:46.643226 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:17:47.250957 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:47.250922 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:17:47.436207 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:47.436174 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:17:47.825058 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:47.825027 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:17:48.023938 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:48.023906 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:17:48.625344 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:48.625311 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:17:48.849100 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:48.849072 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:17:48.956644 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:48.956576 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:17:49.057140 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:49.057106 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:17:49.429419 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:49.429393 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:17:49.825417 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:49.825339 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:17:49.885378 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:49.885354 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:17:50.021994 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:50.021966 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:17:50.630150 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:50.630125 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:17:51.033637 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:51.033562 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:17:51.428611 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:51.428581 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:17:51.499258 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:51.499191 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:17:51.668918 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:51.668888 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:17:51.836445 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:51.836378 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:17:52.244997 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:52.244969 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gvwsp" Jul 23 09:17:52.692387 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:52.692335 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:17:52.851510 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:52.851471 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:17:53.019768 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:53.019686 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/81ee471f-b215-43bd-a279-9ee79d67cd2b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kwb49\" (UID: \"81ee471f-b215-43bd-a279-9ee79d67cd2b\") " pod="openshift-insights/insights-runtime-extractor-kwb49" Jul 23 09:17:53.021805 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:53.021784 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/81ee471f-b215-43bd-a279-9ee79d67cd2b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kwb49\" (UID: \"81ee471f-b215-43bd-a279-9ee79d67cd2b\") " pod="openshift-insights/insights-runtime-extractor-kwb49" Jul 23 09:17:53.094196 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:53.094167 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-sjj26\"" Jul 23 09:17:53.101514 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:53.101498 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kwb49" Jul 23 09:17:53.264922 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:53.264855 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kwb49"] Jul 23 09:17:53.266843 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:53.266813 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:17:53.267072 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:17:53.267051 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81ee471f_b215_43bd_a279_9ee79d67cd2b.slice/crio-f05b86cfb3b59cf866568199ca6e1295183e5d82ea59b7c95bc88aa9ec4de7c6 WatchSource:0}: Error finding container f05b86cfb3b59cf866568199ca6e1295183e5d82ea59b7c95bc88aa9ec4de7c6: Status 404 returned error can't find the container with id f05b86cfb3b59cf866568199ca6e1295183e5d82ea59b7c95bc88aa9ec4de7c6 Jul 23 09:17:53.384591 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:53.384557 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kwb49" event={"ID":"81ee471f-b215-43bd-a279-9ee79d67cd2b","Type":"ContainerStarted","Data":"f05b86cfb3b59cf866568199ca6e1295183e5d82ea59b7c95bc88aa9ec4de7c6"} Jul 23 09:17:53.448271 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:53.448209 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:17:53.714390 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:53.714352 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:17:53.979895 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:53.979819 2576 scope.go:117] "RemoveContainer" containerID="9212332a19a6600cb44a1fb65dad1c5d95eabf99bb6488aa3b6603b7409fbe6f" Jul 23 09:17:54.057826 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:54.057759 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:17:54.392952 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:54.392921 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-5szxg_4908aff3-196e-44fa-b8e6-c0e81b476433/console-operator/2.log" Jul 23 09:17:54.393461 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:54.393438 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-5szxg_4908aff3-196e-44fa-b8e6-c0e81b476433/console-operator/1.log" Jul 23 09:17:54.393530 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:54.393484 2576 generic.go:356] "Generic (PLEG): container finished" podID="4908aff3-196e-44fa-b8e6-c0e81b476433" containerID="28bf8b78eaff551a1b5c0d969b593fe4b88c837f24a83c81a35eb3d64871a906" exitCode=255 Jul 23 09:17:54.393584 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:54.393538 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-5szxg" event={"ID":"4908aff3-196e-44fa-b8e6-c0e81b476433","Type":"ContainerDied","Data":"28bf8b78eaff551a1b5c0d969b593fe4b88c837f24a83c81a35eb3d64871a906"} Jul 23 09:17:54.393642 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:54.393628 2576 scope.go:117] "RemoveContainer" containerID="9212332a19a6600cb44a1fb65dad1c5d95eabf99bb6488aa3b6603b7409fbe6f" Jul 23 09:17:54.393923 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:54.393897 2576 scope.go:117] "RemoveContainer" containerID="28bf8b78eaff551a1b5c0d969b593fe4b88c837f24a83c81a35eb3d64871a906" Jul 23 09:17:54.394094 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:54.394073 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-565b7b569d-5szxg_openshift-console-operator(4908aff3-196e-44fa-b8e6-c0e81b476433)\"" pod="openshift-console-operator/console-operator-565b7b569d-5szxg" podUID="4908aff3-196e-44fa-b8e6-c0e81b476433" Jul 23 09:17:54.637812 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:54.637774 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:17:55.248821 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:55.248782 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:17:55.399827 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:55.399791 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-5szxg_4908aff3-196e-44fa-b8e6-c0e81b476433/console-operator/2.log" Jul 23 09:17:55.482187 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:55.482161 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:17:55.624111 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:55.624084 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:17:55.690556 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:55.690521 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:17:55.896873 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:55.896774 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:17:57.664935 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:57.664899 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/33072424-e286-4580-a3b5-0ccf6b4cbcf8-original-pull-secret\") pod \"global-pull-secret-syncer-l6fgq\" (UID: \"33072424-e286-4580-a3b5-0ccf6b4cbcf8\") " pod="kube-system/global-pull-secret-syncer-l6fgq" Jul 23 09:17:57.667567 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:57.667545 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 23 09:17:57.677966 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:57.677942 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/33072424-e286-4580-a3b5-0ccf6b4cbcf8-original-pull-secret\") pod \"global-pull-secret-syncer-l6fgq\" (UID: \"33072424-e286-4580-a3b5-0ccf6b4cbcf8\") " pod="kube-system/global-pull-secret-syncer-l6fgq" Jul 23 09:17:57.765913 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:57.765878 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/381423db-deea-45d8-8cff-408186294f3e-metrics-certs\") pod \"network-metrics-daemon-b7fmg\" (UID: \"381423db-deea-45d8-8cff-408186294f3e\") " pod="openshift-multus/network-metrics-daemon-b7fmg" Jul 23 09:17:57.768503 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:57.768484 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 23 09:17:57.778321 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:57.778300 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/381423db-deea-45d8-8cff-408186294f3e-metrics-certs\") pod \"network-metrics-daemon-b7fmg\" (UID: \"381423db-deea-45d8-8cff-408186294f3e\") " pod="openshift-multus/network-metrics-daemon-b7fmg" Jul 23 09:17:57.867256 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:57.867210 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99lgf\" (UniqueName: \"kubernetes.io/projected/82e553d7-cc99-4f29-99ba-1816621870c6-kube-api-access-99lgf\") pod \"network-check-target-b24sn\" (UID: \"82e553d7-cc99-4f29-99ba-1816621870c6\") " pod="openshift-network-diagnostics/network-check-target-b24sn" Jul 23 09:17:57.870121 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:57.870101 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99lgf\" (UniqueName: \"kubernetes.io/projected/82e553d7-cc99-4f29-99ba-1816621870c6-kube-api-access-99lgf\") pod \"network-check-target-b24sn\" (UID: \"82e553d7-cc99-4f29-99ba-1816621870c6\") " pod="openshift-network-diagnostics/network-check-target-b24sn" Jul 23 09:17:57.894067 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:57.894042 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tghcj\"" Jul 23 09:17:57.901083 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:57.901060 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-s2sdc\"" Jul 23 09:17:57.902049 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:57.902032 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b7fmg" Jul 23 09:17:57.907130 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:57.907112 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6fgq" Jul 23 09:17:57.909336 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:57.909285 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b24sn" Jul 23 09:17:58.262117 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:58.262050 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-b24sn"] Jul 23 09:17:58.266716 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:17:58.266692 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82e553d7_cc99_4f29_99ba_1816621870c6.slice/crio-5eb92a6c4fb8cd64bb560cc8630f7250294b48f821a2f7cdd3558c6df5abeb98 WatchSource:0}: Error finding container 5eb92a6c4fb8cd64bb560cc8630f7250294b48f821a2f7cdd3558c6df5abeb98: Status 404 returned error can't find the container with id 5eb92a6c4fb8cd64bb560cc8630f7250294b48f821a2f7cdd3558c6df5abeb98 Jul 23 09:17:58.410962 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:58.410905 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kwb49" event={"ID":"81ee471f-b215-43bd-a279-9ee79d67cd2b","Type":"ContainerStarted","Data":"aa13b6409794de3e2badd90fbd79dfdea6910c1972a4c6ce5dd3480eb29dc119"} Jul 23 09:17:58.412227 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:58.412203 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-b24sn" event={"ID":"82e553d7-cc99-4f29-99ba-1816621870c6","Type":"ContainerStarted","Data":"ebdaa8c1fb392bac11b6332256c7b90bf60e2a64d0ca7794c0cb0110dd23e822"} Jul 23 09:17:58.412343 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:58.412255 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-b24sn" event={"ID":"82e553d7-cc99-4f29-99ba-1816621870c6","Type":"ContainerStarted","Data":"5eb92a6c4fb8cd64bb560cc8630f7250294b48f821a2f7cdd3558c6df5abeb98"} Jul 23 09:17:58.412343 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:58.412304 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-b24sn" Jul 23 09:17:58.430921 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:58.430878 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-b24sn" podStartSLOduration=65.43086241 podStartE2EDuration="1m5.43086241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:16:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:17:58.430690459 +0000 UTC m=+66.969969037" watchObservedRunningTime="2026-07-23 09:17:58.43086241 +0000 UTC m=+66.970140988" Jul 23 09:17:58.485729 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:58.485699 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-l6fgq"] Jul 23 09:17:58.488743 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:58.488720 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-b7fmg"] Jul 23 09:17:58.488743 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:17:58.488737 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33072424_e286_4580_a3b5_0ccf6b4cbcf8.slice/crio-2059315b7b1aed66d0e9804247928c228b6460a09e4abd75b8d0b403ea77db29 WatchSource:0}: Error finding container 2059315b7b1aed66d0e9804247928c228b6460a09e4abd75b8d0b403ea77db29: Status 404 returned error can't find the container with id 2059315b7b1aed66d0e9804247928c228b6460a09e4abd75b8d0b403ea77db29 Jul 23 09:17:58.491576 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:17:58.491552 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod381423db_deea_45d8_8cff_408186294f3e.slice/crio-fad5dd6a282a0afd592d03d4635c82cf1a0969f570e9e3d0ccb21f3e4b8c9061 WatchSource:0}: Error finding container fad5dd6a282a0afd592d03d4635c82cf1a0969f570e9e3d0ccb21f3e4b8c9061: Status 404 returned error can't find the container with id fad5dd6a282a0afd592d03d4635c82cf1a0969f570e9e3d0ccb21f3e4b8c9061 Jul 23 09:17:58.687391 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:58.687315 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-5szxg" Jul 23 09:17:58.687391 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:58.687379 2576 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-565b7b569d-5szxg" Jul 23 09:17:58.687869 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:58.687761 2576 scope.go:117] "RemoveContainer" containerID="28bf8b78eaff551a1b5c0d969b593fe4b88c837f24a83c81a35eb3d64871a906" Jul 23 09:17:58.687974 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:17:58.687952 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-565b7b569d-5szxg_openshift-console-operator(4908aff3-196e-44fa-b8e6-c0e81b476433)\"" pod="openshift-console-operator/console-operator-565b7b569d-5szxg" podUID="4908aff3-196e-44fa-b8e6-c0e81b476433" Jul 23 09:17:58.777348 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:58.777306 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1260ceaa-4aa5-4130-9ea7-ff8212c2d751-registry-tls\") pod \"image-registry-7989bd97d7-g2rdd\" (UID: \"1260ceaa-4aa5-4130-9ea7-ff8212c2d751\") " pod="openshift-image-registry/image-registry-7989bd97d7-g2rdd" Jul 23 09:17:58.780135 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:58.780104 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1260ceaa-4aa5-4130-9ea7-ff8212c2d751-registry-tls\") pod \"image-registry-7989bd97d7-g2rdd\" (UID: \"1260ceaa-4aa5-4130-9ea7-ff8212c2d751\") " pod="openshift-image-registry/image-registry-7989bd97d7-g2rdd" Jul 23 09:17:58.878474 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:58.878438 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a17a8c5e-a4ee-4e04-b1dc-03ec2e43c60f-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-xtr4d\" (UID: \"a17a8c5e-a4ee-4e04-b1dc-03ec2e43c60f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-xtr4d" Jul 23 09:17:58.878612 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:58.878481 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/31bba45e-487f-4c59-ac81-d8e3b90a549d-metrics-tls\") pod \"dns-default-mgk4m\" (UID: \"31bba45e-487f-4c59-ac81-d8e3b90a549d\") " pod="openshift-dns/dns-default-mgk4m" Jul 23 09:17:58.881323 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:58.881295 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/31bba45e-487f-4c59-ac81-d8e3b90a549d-metrics-tls\") pod \"dns-default-mgk4m\" (UID: \"31bba45e-487f-4c59-ac81-d8e3b90a549d\") " pod="openshift-dns/dns-default-mgk4m" Jul 23 09:17:58.881479 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:58.881386 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a17a8c5e-a4ee-4e04-b1dc-03ec2e43c60f-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-xtr4d\" (UID: \"a17a8c5e-a4ee-4e04-b1dc-03ec2e43c60f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-xtr4d" Jul 23 09:17:58.974794 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:58.974545 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-q5wx9\"" Jul 23 09:17:58.982766 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:58.982732 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7989bd97d7-g2rdd" Jul 23 09:17:59.043448 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:59.043415 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-x59bn\"" Jul 23 09:17:59.052176 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:59.051656 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-xtr4d" Jul 23 09:17:59.060290 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:59.060267 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-p8qpt\"" Jul 23 09:17:59.063229 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:59.063202 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mgk4m" Jul 23 09:17:59.177379 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:59.177290 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7989bd97d7-g2rdd"] Jul 23 09:17:59.251868 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:59.251820 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-xtr4d"] Jul 23 09:17:59.273757 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:59.273685 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mgk4m"] Jul 23 09:17:59.278115 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:17:59.278087 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31bba45e_487f_4c59_ac81_d8e3b90a549d.slice/crio-621c600365d45c3c98387388c71415d420141d5d05b1c62ed36b14d39ed73d92 WatchSource:0}: Error finding container 621c600365d45c3c98387388c71415d420141d5d05b1c62ed36b14d39ed73d92: Status 404 returned error can't find the container with id 621c600365d45c3c98387388c71415d420141d5d05b1c62ed36b14d39ed73d92 Jul 23 09:17:59.418541 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:59.418499 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-l6fgq" event={"ID":"33072424-e286-4580-a3b5-0ccf6b4cbcf8","Type":"ContainerStarted","Data":"2059315b7b1aed66d0e9804247928c228b6460a09e4abd75b8d0b403ea77db29"} Jul 23 09:17:59.420376 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:59.420339 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-b7fmg" event={"ID":"381423db-deea-45d8-8cff-408186294f3e","Type":"ContainerStarted","Data":"fad5dd6a282a0afd592d03d4635c82cf1a0969f570e9e3d0ccb21f3e4b8c9061"} Jul 23 09:17:59.422456 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:59.422429 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7989bd97d7-g2rdd" event={"ID":"1260ceaa-4aa5-4130-9ea7-ff8212c2d751","Type":"ContainerStarted","Data":"cf22fe53a4e6e3c9cf57fb73bfbe576fcb984058e856fddabfd27bb839bffe16"} Jul 23 09:17:59.422574 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:59.422486 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7989bd97d7-g2rdd" event={"ID":"1260ceaa-4aa5-4130-9ea7-ff8212c2d751","Type":"ContainerStarted","Data":"702643edcea8ddfea1795989dbb6e483ddfb7e535d604d10f5102df266c78926"} Jul 23 09:17:59.423286 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:59.423265 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7989bd97d7-g2rdd" Jul 23 09:17:59.424632 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:59.424610 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mgk4m" event={"ID":"31bba45e-487f-4c59-ac81-d8e3b90a549d","Type":"ContainerStarted","Data":"621c600365d45c3c98387388c71415d420141d5d05b1c62ed36b14d39ed73d92"} Jul 23 09:17:59.426833 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:59.426810 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-xtr4d" event={"ID":"a17a8c5e-a4ee-4e04-b1dc-03ec2e43c60f","Type":"ContainerStarted","Data":"22aebf12785d0558ffc0df8ef31b9c33184a4b71f0e131a77c50edabab03f715"} Jul 23 09:17:59.444203 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:59.444150 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7989bd97d7-g2rdd" podStartSLOduration=66.444132847 podStartE2EDuration="1m6.444132847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:16:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:17:59.443470851 +0000 UTC m=+67.982749436" watchObservedRunningTime="2026-07-23 09:17:59.444132847 +0000 UTC m=+67.983411427" Jul 23 09:17:59.586054 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:59.585972 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/59ee07ce-4747-496f-9876-81f2c2358981-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-8zbdn\" (UID: \"59ee07ce-4747-496f-9876-81f2c2358981\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8zbdn" Jul 23 09:17:59.589493 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:59.589463 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/59ee07ce-4747-496f-9876-81f2c2358981-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-8zbdn\" (UID: \"59ee07ce-4747-496f-9876-81f2c2358981\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8zbdn" Jul 23 09:17:59.596607 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:59.596581 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-xqgv5\"" Jul 23 09:17:59.604580 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:59.604510 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8zbdn" Jul 23 09:17:59.687922 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:59.687176 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6bddaeac-9272-4031-b216-a31fbe58c89d-service-ca-bundle\") pod \"router-default-7494768586-7px6v\" (UID: \"6bddaeac-9272-4031-b216-a31fbe58c89d\") " pod="openshift-ingress/router-default-7494768586-7px6v" Jul 23 09:17:59.687922 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:59.687874 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6bddaeac-9272-4031-b216-a31fbe58c89d-service-ca-bundle\") pod \"router-default-7494768586-7px6v\" (UID: \"6bddaeac-9272-4031-b216-a31fbe58c89d\") " pod="openshift-ingress/router-default-7494768586-7px6v" Jul 23 09:17:59.788636 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:59.788596 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bddaeac-9272-4031-b216-a31fbe58c89d-metrics-certs\") pod \"router-default-7494768586-7px6v\" (UID: \"6bddaeac-9272-4031-b216-a31fbe58c89d\") " pod="openshift-ingress/router-default-7494768586-7px6v" Jul 23 09:17:59.791711 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:59.791651 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bddaeac-9272-4031-b216-a31fbe58c89d-metrics-certs\") pod \"router-default-7494768586-7px6v\" (UID: \"6bddaeac-9272-4031-b216-a31fbe58c89d\") " pod="openshift-ingress/router-default-7494768586-7px6v" Jul 23 09:17:59.913221 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:59.913189 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-4dsz5\"" Jul 23 09:17:59.921598 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:17:59.921572 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7494768586-7px6v" Jul 23 09:18:00.009815 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:00.009702 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8zbdn"] Jul 23 09:18:00.020390 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:18:00.020356 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59ee07ce_4747_496f_9876_81f2c2358981.slice/crio-f094f08ab8cf62bbdb77b02dff466f41ce88e2b2ceb73f36af317a0402e9e622 WatchSource:0}: Error finding container f094f08ab8cf62bbdb77b02dff466f41ce88e2b2ceb73f36af317a0402e9e622: Status 404 returned error can't find the container with id f094f08ab8cf62bbdb77b02dff466f41ce88e2b2ceb73f36af317a0402e9e622 Jul 23 09:18:00.131342 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:00.130946 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7494768586-7px6v"] Jul 23 09:18:00.173291 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:18:00.173233 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bddaeac_9272_4031_b216_a31fbe58c89d.slice/crio-0c57309b1b16b117793aeec6c08fc67caf42a5928458e21a348137fcae6e4d57 WatchSource:0}: Error finding container 0c57309b1b16b117793aeec6c08fc67caf42a5928458e21a348137fcae6e4d57: Status 404 returned error can't find the container with id 0c57309b1b16b117793aeec6c08fc67caf42a5928458e21a348137fcae6e4d57 Jul 23 09:18:00.436270 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:00.436181 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-b7fmg" event={"ID":"381423db-deea-45d8-8cff-408186294f3e","Type":"ContainerStarted","Data":"d85c1770bfaf3cd446f71bcffc60176193666d1933e97af3dde20b0ed8c53549"} Jul 23 09:18:00.440265 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:00.440104 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7494768586-7px6v" event={"ID":"6bddaeac-9272-4031-b216-a31fbe58c89d","Type":"ContainerStarted","Data":"79ac16cfff971493c9dab6484d6c83b9a0d57acffc6ff384ff33a80f256cb35f"} Jul 23 09:18:00.440265 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:00.440155 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7494768586-7px6v" event={"ID":"6bddaeac-9272-4031-b216-a31fbe58c89d","Type":"ContainerStarted","Data":"0c57309b1b16b117793aeec6c08fc67caf42a5928458e21a348137fcae6e4d57"} Jul 23 09:18:00.445188 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:00.444331 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8zbdn" event={"ID":"59ee07ce-4747-496f-9876-81f2c2358981","Type":"ContainerStarted","Data":"f094f08ab8cf62bbdb77b02dff466f41ce88e2b2ceb73f36af317a0402e9e622"} Jul 23 09:18:00.462707 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:00.462447 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-7494768586-7px6v" podStartSLOduration=36.462426754 podStartE2EDuration="36.462426754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:17:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:18:00.462366844 +0000 UTC m=+69.001645423" watchObservedRunningTime="2026-07-23 09:18:00.462426754 +0000 UTC m=+69.001705325" Jul 23 09:18:00.922625 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:00.922590 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7494768586-7px6v" Jul 23 09:18:00.925604 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:00.925578 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7494768586-7px6v" Jul 23 09:18:01.449252 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:01.449213 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7494768586-7px6v" Jul 23 09:18:01.450670 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:01.450644 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7494768586-7px6v" Jul 23 09:18:02.999608 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:02.999539 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:03.673604 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:03.673572 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:03.790592 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:03.790551 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:04.426078 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:04.426042 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:04.571055 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:04.570972 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:06.058330 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:06.058291 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:06.219699 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:06.219619 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:06.442547 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:06.442516 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:06.471736 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:06.471645 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mgk4m" event={"ID":"31bba45e-487f-4c59-ac81-d8e3b90a549d","Type":"ContainerStarted","Data":"f2a3e9705fde9c22bd8a6ed641306f5fca3e08d7957e852dd937dbac2b1f9810"} Jul 23 09:18:06.471902 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:06.471746 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mgk4m" event={"ID":"31bba45e-487f-4c59-ac81-d8e3b90a549d","Type":"ContainerStarted","Data":"38fb2821b104a76b8c5df1c795ecd1e2cac3e303f5ecb5293226fff7c2ef4e0a"} Jul 23 09:18:06.471902 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:06.471780 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-mgk4m" Jul 23 09:18:06.474323 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:06.474292 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-xtr4d" event={"ID":"a17a8c5e-a4ee-4e04-b1dc-03ec2e43c60f","Type":"ContainerStarted","Data":"9408a14404178ea9e30eb6043a975575a1853b33947bcf58597f84904de7b5f0"} Jul 23 09:18:06.474454 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:06.474343 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-xtr4d" event={"ID":"a17a8c5e-a4ee-4e04-b1dc-03ec2e43c60f","Type":"ContainerStarted","Data":"56760e2f5505eb38fc4e8c51748ec01af6102be5f6f06a7144aaaf646d80ea93"} Jul 23 09:18:06.476566 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:06.476488 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kwb49" event={"ID":"81ee471f-b215-43bd-a279-9ee79d67cd2b","Type":"ContainerStarted","Data":"75781aad4ac58a3837186beb7dc404c4faa7231a3dedb9cddcb09e15d891c372"} Jul 23 09:18:06.478374 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:06.478348 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-l6fgq" event={"ID":"33072424-e286-4580-a3b5-0ccf6b4cbcf8","Type":"ContainerStarted","Data":"75b0dfe6fd3e73edc6aeb3a8eae1d7792aeef4a506e921e381ea5a97a382493a"} Jul 23 09:18:06.480638 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:06.480578 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-b7fmg" event={"ID":"381423db-deea-45d8-8cff-408186294f3e","Type":"ContainerStarted","Data":"f031a3e6ffe2c9eff0393f18a59e6da6de0a8ca81fb500ecb1447b446eb52230"} Jul 23 09:18:06.482636 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:06.482614 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8zbdn" event={"ID":"59ee07ce-4747-496f-9876-81f2c2358981","Type":"ContainerStarted","Data":"ffa1a6151dc0e57207a53f48f1d66069a4a8595ca1653158e6d597c4e88a685d"} Jul 23 09:18:06.503765 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:06.503719 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-mgk4m" podStartSLOduration=34.10352325 podStartE2EDuration="40.503701431s" totalImagesPullingTime="6.400178181s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:17:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:18:06.501904199 +0000 UTC m=+75.041182788" watchObservedRunningTime="2026-07-23 09:18:06.503701431 +0000 UTC m=+75.042980009" Jul 23 09:18:06.559418 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:06.559360 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-xtr4d" podStartSLOduration=36.201962015 podStartE2EDuration="42.559342732s" totalImagesPullingTime="6.357380717s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:17:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:18:06.559027556 +0000 UTC m=+75.098306145" watchObservedRunningTime="2026-07-23 09:18:06.559342732 +0000 UTC m=+75.098621321" Jul 23 09:18:06.623604 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:06.623540 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8zbdn" podStartSLOduration=36.962014648 podStartE2EDuration="42.623518619s" totalImagesPullingTime="5.661503971s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:17:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:18:06.621660388 +0000 UTC m=+75.160938966" watchObservedRunningTime="2026-07-23 09:18:06.623518619 +0000 UTC m=+75.162797199" Jul 23 09:18:06.664053 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:06.663992 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-l6fgq" podStartSLOduration=66.473999941 podStartE2EDuration="1m13.663968129s" totalImagesPullingTime="7.189968188s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 09:16:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:18:06.66108749 +0000 UTC m=+75.200366068" watchObservedRunningTime="2026-07-23 09:18:06.663968129 +0000 UTC m=+75.203246708" Jul 23 09:18:06.693726 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:06.693691 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:06.867798 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:06.867713 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:06.970639 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:06.970585 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:07.159087 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:07.159047 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:09.157686 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:09.157643 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:09.256061 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:09.256029 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:09.979779 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:09.979761 2576 scope.go:117] "RemoveContainer" containerID="28bf8b78eaff551a1b5c0d969b593fe4b88c837f24a83c81a35eb3d64871a906" Jul 23 09:18:09.979942 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:18:09.979923 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-565b7b569d-5szxg_openshift-console-operator(4908aff3-196e-44fa-b8e6-c0e81b476433)\"" pod="openshift-console-operator/console-operator-565b7b569d-5szxg" podUID="4908aff3-196e-44fa-b8e6-c0e81b476433" Jul 23 09:18:10.501450 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:10.501413 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kwb49" event={"ID":"81ee471f-b215-43bd-a279-9ee79d67cd2b","Type":"ContainerStarted","Data":"86f4f71319d13967e4f11dfc3de41bf2264b5332f25623327484de6da9ce35a6"} Jul 23 09:18:10.522887 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:10.522837 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-kwb49" podStartSLOduration=17.052297951 podStartE2EDuration="33.522823738s" totalImagesPullingTime="16.470525787s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 09:17:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:18:10.521739956 +0000 UTC m=+79.061018533" watchObservedRunningTime="2026-07-23 09:18:10.522823738 +0000 UTC m=+79.062102336" Jul 23 09:18:10.523106 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:10.523070 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-b7fmg" podStartSLOduration=75.817781109 podStartE2EDuration="1m17.523058146s" totalImagesPullingTime="1.705277037s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:16:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:18:06.704479659 +0000 UTC m=+75.243758237" watchObservedRunningTime="2026-07-23 09:18:10.523058146 +0000 UTC m=+79.062336724" Jul 23 09:18:10.692015 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:10.691981 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:10.795742 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:10.795652 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:11.911892 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:11.911858 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:11.982707 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:11.982676 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:14.159289 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:14.159157 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:14.927191 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:14.927159 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:15.021528 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:15.021499 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:15.764387 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:15.764356 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:15.863757 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:15.863725 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:15.974260 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:15.974220 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:16.058971 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:16.058900 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-jvqzl"] Jul 23 09:18:16.061596 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:16.061579 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jvqzl" Jul 23 09:18:16.067417 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:16.067393 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 23 09:18:16.068533 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:16.068510 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 23 09:18:16.068664 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:16.068645 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 23 09:18:16.068719 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:16.068645 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 23 09:18:16.069059 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:16.069045 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-dcjpd\"" Jul 23 09:18:16.089895 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:16.089864 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:16.116153 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:16.116123 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a40defcf-809b-41d8-bae4-72fb87268217-root\") pod \"node-exporter-jvqzl\" (UID: \"a40defcf-809b-41d8-bae4-72fb87268217\") " pod="openshift-monitoring/node-exporter-jvqzl" Jul 23 09:18:16.116153 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:16.116153 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a40defcf-809b-41d8-bae4-72fb87268217-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jvqzl\" (UID: \"a40defcf-809b-41d8-bae4-72fb87268217\") " pod="openshift-monitoring/node-exporter-jvqzl" Jul 23 09:18:16.116382 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:16.116173 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a40defcf-809b-41d8-bae4-72fb87268217-sys\") pod \"node-exporter-jvqzl\" (UID: \"a40defcf-809b-41d8-bae4-72fb87268217\") " pod="openshift-monitoring/node-exporter-jvqzl" Jul 23 09:18:16.116382 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:16.116192 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a40defcf-809b-41d8-bae4-72fb87268217-node-exporter-tls\") pod \"node-exporter-jvqzl\" (UID: \"a40defcf-809b-41d8-bae4-72fb87268217\") " pod="openshift-monitoring/node-exporter-jvqzl" Jul 23 09:18:16.116382 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:16.116261 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a40defcf-809b-41d8-bae4-72fb87268217-node-exporter-accelerators-collector-config\") pod \"node-exporter-jvqzl\" (UID: \"a40defcf-809b-41d8-bae4-72fb87268217\") " pod="openshift-monitoring/node-exporter-jvqzl" Jul 23 09:18:16.116382 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:16.116320 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrl52\" (UniqueName: \"kubernetes.io/projected/a40defcf-809b-41d8-bae4-72fb87268217-kube-api-access-wrl52\") pod \"node-exporter-jvqzl\" (UID: \"a40defcf-809b-41d8-bae4-72fb87268217\") " pod="openshift-monitoring/node-exporter-jvqzl" Jul 23 09:18:16.116382 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:16.116354 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a40defcf-809b-41d8-bae4-72fb87268217-node-exporter-wtmp\") pod \"node-exporter-jvqzl\" (UID: \"a40defcf-809b-41d8-bae4-72fb87268217\") " pod="openshift-monitoring/node-exporter-jvqzl" Jul 23 09:18:16.116382 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:16.116374 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a40defcf-809b-41d8-bae4-72fb87268217-node-exporter-textfile\") pod \"node-exporter-jvqzl\" (UID: \"a40defcf-809b-41d8-bae4-72fb87268217\") " pod="openshift-monitoring/node-exporter-jvqzl" Jul 23 09:18:16.116695 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:16.116393 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a40defcf-809b-41d8-bae4-72fb87268217-metrics-client-ca\") pod \"node-exporter-jvqzl\" (UID: \"a40defcf-809b-41d8-bae4-72fb87268217\") " pod="openshift-monitoring/node-exporter-jvqzl" Jul 23 09:18:16.184379 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:16.184345 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:16.217466 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:16.217437 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a40defcf-809b-41d8-bae4-72fb87268217-node-exporter-accelerators-collector-config\") pod \"node-exporter-jvqzl\" (UID: \"a40defcf-809b-41d8-bae4-72fb87268217\") " pod="openshift-monitoring/node-exporter-jvqzl" Jul 23 09:18:16.217618 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:16.217475 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrl52\" (UniqueName: \"kubernetes.io/projected/a40defcf-809b-41d8-bae4-72fb87268217-kube-api-access-wrl52\") pod \"node-exporter-jvqzl\" (UID: \"a40defcf-809b-41d8-bae4-72fb87268217\") " pod="openshift-monitoring/node-exporter-jvqzl" Jul 23 09:18:16.217618 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:16.217506 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a40defcf-809b-41d8-bae4-72fb87268217-node-exporter-wtmp\") pod \"node-exporter-jvqzl\" (UID: \"a40defcf-809b-41d8-bae4-72fb87268217\") " pod="openshift-monitoring/node-exporter-jvqzl" Jul 23 09:18:16.217733 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:16.217622 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a40defcf-809b-41d8-bae4-72fb87268217-node-exporter-textfile\") pod \"node-exporter-jvqzl\" (UID: \"a40defcf-809b-41d8-bae4-72fb87268217\") " pod="openshift-monitoring/node-exporter-jvqzl" Jul 23 09:18:16.217733 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:16.217659 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a40defcf-809b-41d8-bae4-72fb87268217-metrics-client-ca\") pod \"node-exporter-jvqzl\" (UID: \"a40defcf-809b-41d8-bae4-72fb87268217\") " pod="openshift-monitoring/node-exporter-jvqzl" Jul 23 09:18:16.217733 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:16.217625 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a40defcf-809b-41d8-bae4-72fb87268217-node-exporter-wtmp\") pod \"node-exporter-jvqzl\" (UID: \"a40defcf-809b-41d8-bae4-72fb87268217\") " pod="openshift-monitoring/node-exporter-jvqzl" Jul 23 09:18:16.217882 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:16.217774 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a40defcf-809b-41d8-bae4-72fb87268217-root\") pod \"node-exporter-jvqzl\" (UID: \"a40defcf-809b-41d8-bae4-72fb87268217\") " pod="openshift-monitoring/node-exporter-jvqzl" Jul 23 09:18:16.217882 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:16.217806 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a40defcf-809b-41d8-bae4-72fb87268217-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jvqzl\" (UID: \"a40defcf-809b-41d8-bae4-72fb87268217\") " pod="openshift-monitoring/node-exporter-jvqzl" Jul 23 09:18:16.217882 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:16.217842 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a40defcf-809b-41d8-bae4-72fb87268217-sys\") pod \"node-exporter-jvqzl\" (UID: \"a40defcf-809b-41d8-bae4-72fb87268217\") " pod="openshift-monitoring/node-exporter-jvqzl" Jul 23 09:18:16.217882 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:16.217874 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a40defcf-809b-41d8-bae4-72fb87268217-node-exporter-tls\") pod \"node-exporter-jvqzl\" (UID: \"a40defcf-809b-41d8-bae4-72fb87268217\") " pod="openshift-monitoring/node-exporter-jvqzl" Jul 23 09:18:16.217882 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:16.217877 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a40defcf-809b-41d8-bae4-72fb87268217-sys\") pod \"node-exporter-jvqzl\" (UID: \"a40defcf-809b-41d8-bae4-72fb87268217\") " pod="openshift-monitoring/node-exporter-jvqzl" Jul 23 09:18:16.218082 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:16.217843 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a40defcf-809b-41d8-bae4-72fb87268217-root\") pod \"node-exporter-jvqzl\" (UID: \"a40defcf-809b-41d8-bae4-72fb87268217\") " pod="openshift-monitoring/node-exporter-jvqzl" Jul 23 09:18:16.218082 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:16.217932 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a40defcf-809b-41d8-bae4-72fb87268217-node-exporter-textfile\") pod \"node-exporter-jvqzl\" (UID: \"a40defcf-809b-41d8-bae4-72fb87268217\") " pod="openshift-monitoring/node-exporter-jvqzl" Jul 23 09:18:16.218082 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:16.218032 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a40defcf-809b-41d8-bae4-72fb87268217-node-exporter-accelerators-collector-config\") pod \"node-exporter-jvqzl\" (UID: \"a40defcf-809b-41d8-bae4-72fb87268217\") " pod="openshift-monitoring/node-exporter-jvqzl" Jul 23 09:18:16.218185 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:16.218135 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a40defcf-809b-41d8-bae4-72fb87268217-metrics-client-ca\") pod \"node-exporter-jvqzl\" (UID: \"a40defcf-809b-41d8-bae4-72fb87268217\") " pod="openshift-monitoring/node-exporter-jvqzl" Jul 23 09:18:16.220077 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:16.220058 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a40defcf-809b-41d8-bae4-72fb87268217-node-exporter-tls\") pod \"node-exporter-jvqzl\" (UID: \"a40defcf-809b-41d8-bae4-72fb87268217\") " pod="openshift-monitoring/node-exporter-jvqzl" Jul 23 09:18:16.220135 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:16.220066 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a40defcf-809b-41d8-bae4-72fb87268217-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jvqzl\" (UID: \"a40defcf-809b-41d8-bae4-72fb87268217\") " pod="openshift-monitoring/node-exporter-jvqzl" Jul 23 09:18:16.227009 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:16.226989 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrl52\" (UniqueName: \"kubernetes.io/projected/a40defcf-809b-41d8-bae4-72fb87268217-kube-api-access-wrl52\") pod \"node-exporter-jvqzl\" (UID: \"a40defcf-809b-41d8-bae4-72fb87268217\") " pod="openshift-monitoring/node-exporter-jvqzl" Jul 23 09:18:16.371347 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:16.371262 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jvqzl" Jul 23 09:18:16.380516 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:16.380486 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:16.381467 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:18:16.381440 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda40defcf_809b_41d8_bae4_72fb87268217.slice/crio-a5fcaa5b3e7230dccab0fe20c34ceb11b371eee9e9840f07f37dec0424318306 WatchSource:0}: Error finding container a5fcaa5b3e7230dccab0fe20c34ceb11b371eee9e9840f07f37dec0424318306: Status 404 returned error can't find the container with id a5fcaa5b3e7230dccab0fe20c34ceb11b371eee9e9840f07f37dec0424318306 Jul 23 09:18:16.488046 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:16.488011 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:16.488897 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:16.488873 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-mgk4m" Jul 23 09:18:16.525536 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:16.525509 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jvqzl" event={"ID":"a40defcf-809b-41d8-bae4-72fb87268217","Type":"ContainerStarted","Data":"a5fcaa5b3e7230dccab0fe20c34ceb11b371eee9e9840f07f37dec0424318306"} Jul 23 09:18:16.596004 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:16.595977 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:16.732232 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:16.732201 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:17.118442 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:17.118332 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:17.233677 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:17.233644 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:17.238034 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:17.238013 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 23 09:18:17.240502 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:17.240484 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:18:17.244397 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:17.244381 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 23 09:18:17.244601 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:17.244578 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 23 09:18:17.244601 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:17.244593 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 23 09:18:17.244793 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:17.244641 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 23 09:18:17.244793 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:17.244669 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 23 09:18:17.244911 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:17.244884 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 23 09:18:17.245119 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:17.245102 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-nh6sj\"" Jul 23 09:18:17.245235 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:17.245184 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 23 09:18:17.245532 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:17.245516 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 23 09:18:17.257205 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:17.257188 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 23 09:18:17.265932 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:17.265913 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 23 09:18:17.327274 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:17.327232 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d3c507db-91e8-4eb1-94bb-96dd05b53eac-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d3c507db-91e8-4eb1-94bb-96dd05b53eac\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:18:17.327426 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:17.327295 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d3c507db-91e8-4eb1-94bb-96dd05b53eac-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d3c507db-91e8-4eb1-94bb-96dd05b53eac\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:18:17.327426 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:17.327364 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d3c507db-91e8-4eb1-94bb-96dd05b53eac-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d3c507db-91e8-4eb1-94bb-96dd05b53eac\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:18:17.327426 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:17.327405 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3c507db-91e8-4eb1-94bb-96dd05b53eac-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d3c507db-91e8-4eb1-94bb-96dd05b53eac\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:18:17.327586 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:17.327438 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d3c507db-91e8-4eb1-94bb-96dd05b53eac-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d3c507db-91e8-4eb1-94bb-96dd05b53eac\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:18:17.327586 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:17.327470 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d3c507db-91e8-4eb1-94bb-96dd05b53eac-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d3c507db-91e8-4eb1-94bb-96dd05b53eac\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:18:17.327586 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:17.327495 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d3c507db-91e8-4eb1-94bb-96dd05b53eac-config-out\") pod \"alertmanager-main-0\" (UID: \"d3c507db-91e8-4eb1-94bb-96dd05b53eac\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:18:17.327685 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:17.327582 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d3c507db-91e8-4eb1-94bb-96dd05b53eac-config-volume\") pod \"alertmanager-main-0\" (UID: \"d3c507db-91e8-4eb1-94bb-96dd05b53eac\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:18:17.327685 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:17.327611 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d3c507db-91e8-4eb1-94bb-96dd05b53eac-web-config\") pod \"alertmanager-main-0\" (UID: \"d3c507db-91e8-4eb1-94bb-96dd05b53eac\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:18:17.327685 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:17.327638 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d3c507db-91e8-4eb1-94bb-96dd05b53eac-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d3c507db-91e8-4eb1-94bb-96dd05b53eac\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:18:17.327685 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:17.327666 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d3c507db-91e8-4eb1-94bb-96dd05b53eac-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d3c507db-91e8-4eb1-94bb-96dd05b53eac\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:18:17.327864 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:17.327722 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d3c507db-91e8-4eb1-94bb-96dd05b53eac-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d3c507db-91e8-4eb1-94bb-96dd05b53eac\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:18:17.327864 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:17.327778 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lpxw\" (UniqueName: \"kubernetes.io/projected/d3c507db-91e8-4eb1-94bb-96dd05b53eac-kube-api-access-6lpxw\") pod \"alertmanager-main-0\" (UID: \"d3c507db-91e8-4eb1-94bb-96dd05b53eac\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:18:17.428321 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:17.428290 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d3c507db-91e8-4eb1-94bb-96dd05b53eac-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d3c507db-91e8-4eb1-94bb-96dd05b53eac\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:18:17.428479 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:17.428347 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3c507db-91e8-4eb1-94bb-96dd05b53eac-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d3c507db-91e8-4eb1-94bb-96dd05b53eac\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:18:17.428479 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:17.428380 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d3c507db-91e8-4eb1-94bb-96dd05b53eac-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d3c507db-91e8-4eb1-94bb-96dd05b53eac\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:18:17.428479 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:17.428408 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d3c507db-91e8-4eb1-94bb-96dd05b53eac-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d3c507db-91e8-4eb1-94bb-96dd05b53eac\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:18:17.428479 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:17.428433 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d3c507db-91e8-4eb1-94bb-96dd05b53eac-config-out\") pod \"alertmanager-main-0\" (UID: \"d3c507db-91e8-4eb1-94bb-96dd05b53eac\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:18:17.428690 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:17.428480 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d3c507db-91e8-4eb1-94bb-96dd05b53eac-config-volume\") pod \"alertmanager-main-0\" (UID: \"d3c507db-91e8-4eb1-94bb-96dd05b53eac\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:18:17.428690 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:17.428507 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d3c507db-91e8-4eb1-94bb-96dd05b53eac-web-config\") pod \"alertmanager-main-0\" (UID: \"d3c507db-91e8-4eb1-94bb-96dd05b53eac\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:18:17.428690 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:17.428534 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d3c507db-91e8-4eb1-94bb-96dd05b53eac-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d3c507db-91e8-4eb1-94bb-96dd05b53eac\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:18:17.428690 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:18:17.428558 2576 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 23 09:18:17.428690 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:18:17.428631 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3c507db-91e8-4eb1-94bb-96dd05b53eac-secret-alertmanager-main-tls podName:d3c507db-91e8-4eb1-94bb-96dd05b53eac nodeName:}" failed. No retries permitted until 2026-07-23 09:18:17.928606782 +0000 UTC m=+86.467885368 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/d3c507db-91e8-4eb1-94bb-96dd05b53eac-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "d3c507db-91e8-4eb1-94bb-96dd05b53eac") : secret "alertmanager-main-tls" not found Jul 23 09:18:17.428935 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:17.428741 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d3c507db-91e8-4eb1-94bb-96dd05b53eac-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d3c507db-91e8-4eb1-94bb-96dd05b53eac\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:18:17.428935 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:17.428564 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d3c507db-91e8-4eb1-94bb-96dd05b53eac-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d3c507db-91e8-4eb1-94bb-96dd05b53eac\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:18:17.428935 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:17.428806 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d3c507db-91e8-4eb1-94bb-96dd05b53eac-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d3c507db-91e8-4eb1-94bb-96dd05b53eac\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:18:17.428935 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:17.428860 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lpxw\" (UniqueName: \"kubernetes.io/projected/d3c507db-91e8-4eb1-94bb-96dd05b53eac-kube-api-access-6lpxw\") pod \"alertmanager-main-0\" (UID: \"d3c507db-91e8-4eb1-94bb-96dd05b53eac\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:18:17.428935 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:17.428900 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d3c507db-91e8-4eb1-94bb-96dd05b53eac-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d3c507db-91e8-4eb1-94bb-96dd05b53eac\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:18:17.429173 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:17.428934 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d3c507db-91e8-4eb1-94bb-96dd05b53eac-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d3c507db-91e8-4eb1-94bb-96dd05b53eac\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:18:17.429743 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:17.429356 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3c507db-91e8-4eb1-94bb-96dd05b53eac-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d3c507db-91e8-4eb1-94bb-96dd05b53eac\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:18:17.430606 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:17.430538 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d3c507db-91e8-4eb1-94bb-96dd05b53eac-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d3c507db-91e8-4eb1-94bb-96dd05b53eac\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:18:17.431569 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:17.431546 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d3c507db-91e8-4eb1-94bb-96dd05b53eac-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d3c507db-91e8-4eb1-94bb-96dd05b53eac\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:18:17.431569 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:17.431560 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d3c507db-91e8-4eb1-94bb-96dd05b53eac-config-out\") pod \"alertmanager-main-0\" (UID: \"d3c507db-91e8-4eb1-94bb-96dd05b53eac\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:18:17.431991 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:17.431944 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d3c507db-91e8-4eb1-94bb-96dd05b53eac-config-volume\") pod \"alertmanager-main-0\" (UID: \"d3c507db-91e8-4eb1-94bb-96dd05b53eac\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:18:17.431991 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:17.431984 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d3c507db-91e8-4eb1-94bb-96dd05b53eac-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d3c507db-91e8-4eb1-94bb-96dd05b53eac\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:18:17.432136 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:17.432057 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d3c507db-91e8-4eb1-94bb-96dd05b53eac-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d3c507db-91e8-4eb1-94bb-96dd05b53eac\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:18:17.432136 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:17.432116 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d3c507db-91e8-4eb1-94bb-96dd05b53eac-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d3c507db-91e8-4eb1-94bb-96dd05b53eac\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:18:17.433334 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:17.433315 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d3c507db-91e8-4eb1-94bb-96dd05b53eac-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d3c507db-91e8-4eb1-94bb-96dd05b53eac\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:18:17.433334 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:17.433328 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d3c507db-91e8-4eb1-94bb-96dd05b53eac-web-config\") pod \"alertmanager-main-0\" (UID: \"d3c507db-91e8-4eb1-94bb-96dd05b53eac\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:18:17.442834 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:17.442811 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lpxw\" (UniqueName: \"kubernetes.io/projected/d3c507db-91e8-4eb1-94bb-96dd05b53eac-kube-api-access-6lpxw\") pod \"alertmanager-main-0\" (UID: \"d3c507db-91e8-4eb1-94bb-96dd05b53eac\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:18:17.531362 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:17.531335 2576 generic.go:356] "Generic (PLEG): container finished" podID="a40defcf-809b-41d8-bae4-72fb87268217" containerID="decb9776c393230c73cf7a855ae2eb173780d75f1ca5b9134d734c4d571ab654" exitCode=0 Jul 23 09:18:17.531475 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:17.531425 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jvqzl" event={"ID":"a40defcf-809b-41d8-bae4-72fb87268217","Type":"ContainerDied","Data":"decb9776c393230c73cf7a855ae2eb173780d75f1ca5b9134d734c4d571ab654"} Jul 23 09:18:17.650550 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:17.650520 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:17.933654 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:17.933568 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d3c507db-91e8-4eb1-94bb-96dd05b53eac-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d3c507db-91e8-4eb1-94bb-96dd05b53eac\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:18:17.935927 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:17.935899 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d3c507db-91e8-4eb1-94bb-96dd05b53eac-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d3c507db-91e8-4eb1-94bb-96dd05b53eac\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:18:18.151380 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:18.151346 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:18:18.245837 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:18.245805 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:18.304661 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:18.304632 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 23 09:18:18.305776 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:18:18.305752 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3c507db_91e8_4eb1_94bb_96dd05b53eac.slice/crio-52237837a1b1b89c883bac406b35428ba37c5b55f2971fd020bf53965df89791 WatchSource:0}: Error finding container 52237837a1b1b89c883bac406b35428ba37c5b55f2971fd020bf53965df89791: Status 404 returned error can't find the container with id 52237837a1b1b89c883bac406b35428ba37c5b55f2971fd020bf53965df89791 Jul 23 09:18:18.333363 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:18.333337 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:18.536806 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:18.536718 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jvqzl" event={"ID":"a40defcf-809b-41d8-bae4-72fb87268217","Type":"ContainerStarted","Data":"e6f35edde13ebd1ad5755518a3f071f9c1e9116e6ea6c92cd0ebab0e7e67d4a4"} Jul 23 09:18:18.536806 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:18.536782 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jvqzl" event={"ID":"a40defcf-809b-41d8-bae4-72fb87268217","Type":"ContainerStarted","Data":"fff148ba51017650c7a2c1b66bd58a4d79daf96c963462ff2cc53e3873e14b37"} Jul 23 09:18:18.537811 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:18.537782 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d3c507db-91e8-4eb1-94bb-96dd05b53eac","Type":"ContainerStarted","Data":"52237837a1b1b89c883bac406b35428ba37c5b55f2971fd020bf53965df89791"} Jul 23 09:18:18.558537 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:18.558496 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-jvqzl" podStartSLOduration=1.635943375 podStartE2EDuration="2.558481976s" totalImagesPullingTime="856.533256ms" totalInitContainerRuntime="66.005345ms" isStatefulPod=true podCreationTimestamp="2026-07-23 09:18:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:18:18.557523859 +0000 UTC m=+87.096802447" watchObservedRunningTime="2026-07-23 09:18:18.558481976 +0000 UTC m=+87.097760554" Jul 23 09:18:19.024478 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:19.024440 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:19.107337 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:19.107301 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:19.177187 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:19.177154 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:19.427443 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:19.427421 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:19.533682 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:19.533653 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:19.543432 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:19.543406 2576 generic.go:356] "Generic (PLEG): container finished" podID="d3c507db-91e8-4eb1-94bb-96dd05b53eac" containerID="f0b814f4809098312a191fde862386c052a2d25754fd03de5d2e3d235444de94" exitCode=0 Jul 23 09:18:19.543543 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:19.543493 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d3c507db-91e8-4eb1-94bb-96dd05b53eac","Type":"ContainerDied","Data":"f0b814f4809098312a191fde862386c052a2d25754fd03de5d2e3d235444de94"} Jul 23 09:18:19.633596 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:19.633569 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:19.705578 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:19.705515 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:20.484511 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:20.484474 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-84c8fd6f77-4txkh"] Jul 23 09:18:20.486634 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:20.486617 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-84c8fd6f77-4txkh" Jul 23 09:18:20.489349 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:20.489329 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-3ubhjd1thb7un\"" Jul 23 09:18:20.489463 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:20.489334 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 23 09:18:20.490195 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:20.490177 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 23 09:18:20.490305 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:20.490233 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-kw2wv\"" Jul 23 09:18:20.490358 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:20.490328 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 23 09:18:20.490412 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:20.490369 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 23 09:18:20.498604 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:20.498585 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-84c8fd6f77-4txkh"] Jul 23 09:18:20.557086 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:20.557056 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/74395092-8e7f-4941-8d00-ac34b51838fc-audit-log\") pod \"metrics-server-84c8fd6f77-4txkh\" (UID: \"74395092-8e7f-4941-8d00-ac34b51838fc\") " pod="openshift-monitoring/metrics-server-84c8fd6f77-4txkh" Jul 23 09:18:20.557264 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:20.557102 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2b9v\" (UniqueName: \"kubernetes.io/projected/74395092-8e7f-4941-8d00-ac34b51838fc-kube-api-access-n2b9v\") pod \"metrics-server-84c8fd6f77-4txkh\" (UID: \"74395092-8e7f-4941-8d00-ac34b51838fc\") " pod="openshift-monitoring/metrics-server-84c8fd6f77-4txkh" Jul 23 09:18:20.557264 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:20.557123 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/74395092-8e7f-4941-8d00-ac34b51838fc-secret-metrics-server-tls\") pod \"metrics-server-84c8fd6f77-4txkh\" (UID: \"74395092-8e7f-4941-8d00-ac34b51838fc\") " pod="openshift-monitoring/metrics-server-84c8fd6f77-4txkh" Jul 23 09:18:20.557264 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:20.557150 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/74395092-8e7f-4941-8d00-ac34b51838fc-secret-metrics-server-client-certs\") pod \"metrics-server-84c8fd6f77-4txkh\" (UID: \"74395092-8e7f-4941-8d00-ac34b51838fc\") " pod="openshift-monitoring/metrics-server-84c8fd6f77-4txkh" Jul 23 09:18:20.557264 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:20.557188 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74395092-8e7f-4941-8d00-ac34b51838fc-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-84c8fd6f77-4txkh\" (UID: \"74395092-8e7f-4941-8d00-ac34b51838fc\") " pod="openshift-monitoring/metrics-server-84c8fd6f77-4txkh" Jul 23 09:18:20.557264 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:20.557209 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/74395092-8e7f-4941-8d00-ac34b51838fc-metrics-server-audit-profiles\") pod \"metrics-server-84c8fd6f77-4txkh\" (UID: \"74395092-8e7f-4941-8d00-ac34b51838fc\") " pod="openshift-monitoring/metrics-server-84c8fd6f77-4txkh" Jul 23 09:18:20.557264 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:20.557232 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/74395092-8e7f-4941-8d00-ac34b51838fc-client-ca-bundle\") pod \"metrics-server-84c8fd6f77-4txkh\" (UID: \"74395092-8e7f-4941-8d00-ac34b51838fc\") " pod="openshift-monitoring/metrics-server-84c8fd6f77-4txkh" Jul 23 09:18:20.658549 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:20.658504 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/74395092-8e7f-4941-8d00-ac34b51838fc-audit-log\") pod \"metrics-server-84c8fd6f77-4txkh\" (UID: \"74395092-8e7f-4941-8d00-ac34b51838fc\") " pod="openshift-monitoring/metrics-server-84c8fd6f77-4txkh" Jul 23 09:18:20.658724 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:20.658586 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2b9v\" (UniqueName: \"kubernetes.io/projected/74395092-8e7f-4941-8d00-ac34b51838fc-kube-api-access-n2b9v\") pod \"metrics-server-84c8fd6f77-4txkh\" (UID: \"74395092-8e7f-4941-8d00-ac34b51838fc\") " pod="openshift-monitoring/metrics-server-84c8fd6f77-4txkh" Jul 23 09:18:20.658724 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:20.658617 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/74395092-8e7f-4941-8d00-ac34b51838fc-secret-metrics-server-tls\") pod \"metrics-server-84c8fd6f77-4txkh\" (UID: \"74395092-8e7f-4941-8d00-ac34b51838fc\") " pod="openshift-monitoring/metrics-server-84c8fd6f77-4txkh" Jul 23 09:18:20.658724 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:20.658645 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/74395092-8e7f-4941-8d00-ac34b51838fc-secret-metrics-server-client-certs\") pod \"metrics-server-84c8fd6f77-4txkh\" (UID: \"74395092-8e7f-4941-8d00-ac34b51838fc\") " pod="openshift-monitoring/metrics-server-84c8fd6f77-4txkh" Jul 23 09:18:20.658724 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:20.658675 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74395092-8e7f-4941-8d00-ac34b51838fc-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-84c8fd6f77-4txkh\" (UID: \"74395092-8e7f-4941-8d00-ac34b51838fc\") " pod="openshift-monitoring/metrics-server-84c8fd6f77-4txkh" Jul 23 09:18:20.658724 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:20.658710 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/74395092-8e7f-4941-8d00-ac34b51838fc-metrics-server-audit-profiles\") pod \"metrics-server-84c8fd6f77-4txkh\" (UID: \"74395092-8e7f-4941-8d00-ac34b51838fc\") " pod="openshift-monitoring/metrics-server-84c8fd6f77-4txkh" Jul 23 09:18:20.659074 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:20.658744 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/74395092-8e7f-4941-8d00-ac34b51838fc-client-ca-bundle\") pod \"metrics-server-84c8fd6f77-4txkh\" (UID: \"74395092-8e7f-4941-8d00-ac34b51838fc\") " pod="openshift-monitoring/metrics-server-84c8fd6f77-4txkh" Jul 23 09:18:20.659074 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:20.658969 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/74395092-8e7f-4941-8d00-ac34b51838fc-audit-log\") pod \"metrics-server-84c8fd6f77-4txkh\" (UID: \"74395092-8e7f-4941-8d00-ac34b51838fc\") " pod="openshift-monitoring/metrics-server-84c8fd6f77-4txkh" Jul 23 09:18:20.659604 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:20.659557 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74395092-8e7f-4941-8d00-ac34b51838fc-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-84c8fd6f77-4txkh\" (UID: \"74395092-8e7f-4941-8d00-ac34b51838fc\") " pod="openshift-monitoring/metrics-server-84c8fd6f77-4txkh" Jul 23 09:18:20.659999 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:20.659971 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/74395092-8e7f-4941-8d00-ac34b51838fc-metrics-server-audit-profiles\") pod \"metrics-server-84c8fd6f77-4txkh\" (UID: \"74395092-8e7f-4941-8d00-ac34b51838fc\") " pod="openshift-monitoring/metrics-server-84c8fd6f77-4txkh" Jul 23 09:18:20.661673 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:20.661646 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/74395092-8e7f-4941-8d00-ac34b51838fc-client-ca-bundle\") pod \"metrics-server-84c8fd6f77-4txkh\" (UID: \"74395092-8e7f-4941-8d00-ac34b51838fc\") " pod="openshift-monitoring/metrics-server-84c8fd6f77-4txkh" Jul 23 09:18:20.661781 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:20.661662 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/74395092-8e7f-4941-8d00-ac34b51838fc-secret-metrics-server-client-certs\") pod \"metrics-server-84c8fd6f77-4txkh\" (UID: \"74395092-8e7f-4941-8d00-ac34b51838fc\") " pod="openshift-monitoring/metrics-server-84c8fd6f77-4txkh" Jul 23 09:18:20.661781 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:20.661748 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/74395092-8e7f-4941-8d00-ac34b51838fc-secret-metrics-server-tls\") pod \"metrics-server-84c8fd6f77-4txkh\" (UID: \"74395092-8e7f-4941-8d00-ac34b51838fc\") " pod="openshift-monitoring/metrics-server-84c8fd6f77-4txkh" Jul 23 09:18:20.667171 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:20.667145 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2b9v\" (UniqueName: \"kubernetes.io/projected/74395092-8e7f-4941-8d00-ac34b51838fc-kube-api-access-n2b9v\") pod \"metrics-server-84c8fd6f77-4txkh\" (UID: \"74395092-8e7f-4941-8d00-ac34b51838fc\") " pod="openshift-monitoring/metrics-server-84c8fd6f77-4txkh" Jul 23 09:18:20.783170 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:20.783080 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:20.795183 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:20.795055 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-84c8fd6f77-4txkh" Jul 23 09:18:20.932569 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:20.932537 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-84c8fd6f77-4txkh"] Jul 23 09:18:20.935568 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:18:20.935536 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74395092_8e7f_4941_8d00_ac34b51838fc.slice/crio-737b257ae611c797b9091762663b88fde2a15c1f3425301a0735ee37771d878e WatchSource:0}: Error finding container 737b257ae611c797b9091762663b88fde2a15c1f3425301a0735ee37771d878e: Status 404 returned error can't find the container with id 737b257ae611c797b9091762663b88fde2a15c1f3425301a0735ee37771d878e Jul 23 09:18:20.994279 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:20.994229 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:21.322771 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:21.322729 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-9998cfcd5-b7c92"] Jul 23 09:18:21.325658 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:21.325634 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-9998cfcd5-b7c92" Jul 23 09:18:21.329491 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:21.329468 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 23 09:18:21.329598 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:21.329524 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-s5mdd\"" Jul 23 09:18:21.329658 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:21.329596 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 23 09:18:21.330085 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:21.329866 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 23 09:18:21.330085 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:21.329945 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 23 09:18:21.330085 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:21.329973 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 23 09:18:21.335717 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:21.335691 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 23 09:18:21.339768 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:21.339744 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-9998cfcd5-b7c92"] Jul 23 09:18:21.365154 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:21.365129 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a05745d8-9561-497d-8e07-d0b9c1ee6ba3-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-9998cfcd5-b7c92\" (UID: \"a05745d8-9561-497d-8e07-d0b9c1ee6ba3\") " pod="openshift-monitoring/telemeter-client-9998cfcd5-b7c92" Jul 23 09:18:21.365285 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:21.365176 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a05745d8-9561-497d-8e07-d0b9c1ee6ba3-federate-client-tls\") pod \"telemeter-client-9998cfcd5-b7c92\" (UID: \"a05745d8-9561-497d-8e07-d0b9c1ee6ba3\") " pod="openshift-monitoring/telemeter-client-9998cfcd5-b7c92" Jul 23 09:18:21.365339 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:21.365279 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a05745d8-9561-497d-8e07-d0b9c1ee6ba3-telemeter-trusted-ca-bundle\") pod \"telemeter-client-9998cfcd5-b7c92\" (UID: \"a05745d8-9561-497d-8e07-d0b9c1ee6ba3\") " pod="openshift-monitoring/telemeter-client-9998cfcd5-b7c92" Jul 23 09:18:21.365396 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:21.365356 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a05745d8-9561-497d-8e07-d0b9c1ee6ba3-telemeter-client-tls\") pod \"telemeter-client-9998cfcd5-b7c92\" (UID: \"a05745d8-9561-497d-8e07-d0b9c1ee6ba3\") " pod="openshift-monitoring/telemeter-client-9998cfcd5-b7c92" Jul 23 09:18:21.365396 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:21.365388 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a05745d8-9561-497d-8e07-d0b9c1ee6ba3-metrics-client-ca\") pod \"telemeter-client-9998cfcd5-b7c92\" (UID: \"a05745d8-9561-497d-8e07-d0b9c1ee6ba3\") " pod="openshift-monitoring/telemeter-client-9998cfcd5-b7c92" Jul 23 09:18:21.365517 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:21.365499 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a05745d8-9561-497d-8e07-d0b9c1ee6ba3-secret-telemeter-client\") pod \"telemeter-client-9998cfcd5-b7c92\" (UID: \"a05745d8-9561-497d-8e07-d0b9c1ee6ba3\") " pod="openshift-monitoring/telemeter-client-9998cfcd5-b7c92" Jul 23 09:18:21.365574 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:21.365536 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x96mv\" (UniqueName: \"kubernetes.io/projected/a05745d8-9561-497d-8e07-d0b9c1ee6ba3-kube-api-access-x96mv\") pod \"telemeter-client-9998cfcd5-b7c92\" (UID: \"a05745d8-9561-497d-8e07-d0b9c1ee6ba3\") " pod="openshift-monitoring/telemeter-client-9998cfcd5-b7c92" Jul 23 09:18:21.365624 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:21.365570 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a05745d8-9561-497d-8e07-d0b9c1ee6ba3-serving-certs-ca-bundle\") pod \"telemeter-client-9998cfcd5-b7c92\" (UID: \"a05745d8-9561-497d-8e07-d0b9c1ee6ba3\") " pod="openshift-monitoring/telemeter-client-9998cfcd5-b7c92" Jul 23 09:18:21.454352 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:21.454321 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7989bd97d7-g2rdd" Jul 23 09:18:21.466192 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:21.466145 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a05745d8-9561-497d-8e07-d0b9c1ee6ba3-telemeter-client-tls\") pod \"telemeter-client-9998cfcd5-b7c92\" (UID: \"a05745d8-9561-497d-8e07-d0b9c1ee6ba3\") " pod="openshift-monitoring/telemeter-client-9998cfcd5-b7c92" Jul 23 09:18:21.466192 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:21.466186 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a05745d8-9561-497d-8e07-d0b9c1ee6ba3-metrics-client-ca\") pod \"telemeter-client-9998cfcd5-b7c92\" (UID: \"a05745d8-9561-497d-8e07-d0b9c1ee6ba3\") " pod="openshift-monitoring/telemeter-client-9998cfcd5-b7c92" Jul 23 09:18:21.466423 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:21.466222 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a05745d8-9561-497d-8e07-d0b9c1ee6ba3-secret-telemeter-client\") pod \"telemeter-client-9998cfcd5-b7c92\" (UID: \"a05745d8-9561-497d-8e07-d0b9c1ee6ba3\") " pod="openshift-monitoring/telemeter-client-9998cfcd5-b7c92" Jul 23 09:18:21.466423 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:21.466261 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x96mv\" (UniqueName: \"kubernetes.io/projected/a05745d8-9561-497d-8e07-d0b9c1ee6ba3-kube-api-access-x96mv\") pod \"telemeter-client-9998cfcd5-b7c92\" (UID: \"a05745d8-9561-497d-8e07-d0b9c1ee6ba3\") " pod="openshift-monitoring/telemeter-client-9998cfcd5-b7c92" Jul 23 09:18:21.466423 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:21.466285 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a05745d8-9561-497d-8e07-d0b9c1ee6ba3-serving-certs-ca-bundle\") pod \"telemeter-client-9998cfcd5-b7c92\" (UID: \"a05745d8-9561-497d-8e07-d0b9c1ee6ba3\") " pod="openshift-monitoring/telemeter-client-9998cfcd5-b7c92" Jul 23 09:18:21.466423 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:21.466325 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a05745d8-9561-497d-8e07-d0b9c1ee6ba3-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-9998cfcd5-b7c92\" (UID: \"a05745d8-9561-497d-8e07-d0b9c1ee6ba3\") " pod="openshift-monitoring/telemeter-client-9998cfcd5-b7c92" Jul 23 09:18:21.466423 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:21.466368 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a05745d8-9561-497d-8e07-d0b9c1ee6ba3-federate-client-tls\") pod \"telemeter-client-9998cfcd5-b7c92\" (UID: \"a05745d8-9561-497d-8e07-d0b9c1ee6ba3\") " pod="openshift-monitoring/telemeter-client-9998cfcd5-b7c92" Jul 23 09:18:21.466423 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:21.466423 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a05745d8-9561-497d-8e07-d0b9c1ee6ba3-telemeter-trusted-ca-bundle\") pod \"telemeter-client-9998cfcd5-b7c92\" (UID: \"a05745d8-9561-497d-8e07-d0b9c1ee6ba3\") " pod="openshift-monitoring/telemeter-client-9998cfcd5-b7c92" Jul 23 09:18:21.467233 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:21.467180 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a05745d8-9561-497d-8e07-d0b9c1ee6ba3-metrics-client-ca\") pod \"telemeter-client-9998cfcd5-b7c92\" (UID: \"a05745d8-9561-497d-8e07-d0b9c1ee6ba3\") " pod="openshift-monitoring/telemeter-client-9998cfcd5-b7c92" Jul 23 09:18:21.467381 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:21.467345 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a05745d8-9561-497d-8e07-d0b9c1ee6ba3-telemeter-trusted-ca-bundle\") pod \"telemeter-client-9998cfcd5-b7c92\" (UID: \"a05745d8-9561-497d-8e07-d0b9c1ee6ba3\") " pod="openshift-monitoring/telemeter-client-9998cfcd5-b7c92" Jul 23 09:18:21.468014 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:21.467986 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a05745d8-9561-497d-8e07-d0b9c1ee6ba3-serving-certs-ca-bundle\") pod \"telemeter-client-9998cfcd5-b7c92\" (UID: \"a05745d8-9561-497d-8e07-d0b9c1ee6ba3\") " pod="openshift-monitoring/telemeter-client-9998cfcd5-b7c92" Jul 23 09:18:21.469057 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:21.469035 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a05745d8-9561-497d-8e07-d0b9c1ee6ba3-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-9998cfcd5-b7c92\" (UID: \"a05745d8-9561-497d-8e07-d0b9c1ee6ba3\") " pod="openshift-monitoring/telemeter-client-9998cfcd5-b7c92" Jul 23 09:18:21.469311 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:21.469288 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a05745d8-9561-497d-8e07-d0b9c1ee6ba3-secret-telemeter-client\") pod \"telemeter-client-9998cfcd5-b7c92\" (UID: \"a05745d8-9561-497d-8e07-d0b9c1ee6ba3\") " pod="openshift-monitoring/telemeter-client-9998cfcd5-b7c92" Jul 23 09:18:21.469420 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:21.469340 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a05745d8-9561-497d-8e07-d0b9c1ee6ba3-telemeter-client-tls\") pod \"telemeter-client-9998cfcd5-b7c92\" (UID: \"a05745d8-9561-497d-8e07-d0b9c1ee6ba3\") " pod="openshift-monitoring/telemeter-client-9998cfcd5-b7c92" Jul 23 09:18:21.469729 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:21.469709 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a05745d8-9561-497d-8e07-d0b9c1ee6ba3-federate-client-tls\") pod \"telemeter-client-9998cfcd5-b7c92\" (UID: \"a05745d8-9561-497d-8e07-d0b9c1ee6ba3\") " pod="openshift-monitoring/telemeter-client-9998cfcd5-b7c92" Jul 23 09:18:21.475740 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:21.475721 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x96mv\" (UniqueName: \"kubernetes.io/projected/a05745d8-9561-497d-8e07-d0b9c1ee6ba3-kube-api-access-x96mv\") pod \"telemeter-client-9998cfcd5-b7c92\" (UID: \"a05745d8-9561-497d-8e07-d0b9c1ee6ba3\") " pod="openshift-monitoring/telemeter-client-9998cfcd5-b7c92" Jul 23 09:18:21.553297 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:21.553262 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-84c8fd6f77-4txkh" event={"ID":"74395092-8e7f-4941-8d00-ac34b51838fc","Type":"ContainerStarted","Data":"737b257ae611c797b9091762663b88fde2a15c1f3425301a0735ee37771d878e"} Jul 23 09:18:21.637518 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:21.637493 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-9998cfcd5-b7c92" Jul 23 09:18:21.651111 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:21.651088 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:21.771729 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:21.771697 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:21.843367 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:21.842709 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-9998cfcd5-b7c92"] Jul 23 09:18:21.844709 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:18:21.844672 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda05745d8_9561_497d_8e07_d0b9c1ee6ba3.slice/crio-2749b9370f812ae3d3436fee27ae115c47e5b82b4aa0a56edebf8f5a5ac364f2 WatchSource:0}: Error finding container 2749b9370f812ae3d3436fee27ae115c47e5b82b4aa0a56edebf8f5a5ac364f2: Status 404 returned error can't find the container with id 2749b9370f812ae3d3436fee27ae115c47e5b82b4aa0a56edebf8f5a5ac364f2 Jul 23 09:18:22.340686 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:22.340651 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:22.560075 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:22.560050 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d3c507db-91e8-4eb1-94bb-96dd05b53eac","Type":"ContainerStarted","Data":"4505e914ce83900d19afb88d4b8b6fb9e7ce9e0d985bf7b7c1a0c9a68db83b59"} Jul 23 09:18:22.560433 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:22.560126 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d3c507db-91e8-4eb1-94bb-96dd05b53eac","Type":"ContainerStarted","Data":"4399f6c4bd60f1d8562862c68ff034ddacd38fa3f30a32954ce5b355bd7e4f0e"} Jul 23 09:18:22.560433 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:22.560162 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d3c507db-91e8-4eb1-94bb-96dd05b53eac","Type":"ContainerStarted","Data":"02afea67ff99cccd3c74c1cb11317a9fba222f6e961cf5c74e2349eba5c26056"} Jul 23 09:18:22.560433 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:22.560189 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d3c507db-91e8-4eb1-94bb-96dd05b53eac","Type":"ContainerStarted","Data":"7b426a2b8c29b42471a3c8abd3edc6803bf74965794f3fe430bd53de90f02bf8"} Jul 23 09:18:22.560433 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:22.560220 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d3c507db-91e8-4eb1-94bb-96dd05b53eac","Type":"ContainerStarted","Data":"322ee8b69a743b8a3e29861ad366fde0f351322fb168d7ef4ec03ca153fd2174"} Jul 23 09:18:22.561086 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:22.561061 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-9998cfcd5-b7c92" event={"ID":"a05745d8-9561-497d-8e07-d0b9c1ee6ba3","Type":"ContainerStarted","Data":"2749b9370f812ae3d3436fee27ae115c47e5b82b4aa0a56edebf8f5a5ac364f2"} Jul 23 09:18:22.656944 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:22.656891 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:22.749727 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:22.749669 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:23.569141 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:23.569099 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-84c8fd6f77-4txkh" event={"ID":"74395092-8e7f-4941-8d00-ac34b51838fc","Type":"ContainerStarted","Data":"bdffe10fa2382217449f2c9a21907460c06df60ba15713a6109fde21522afd3a"} Jul 23 09:18:23.573060 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:23.573027 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d3c507db-91e8-4eb1-94bb-96dd05b53eac","Type":"ContainerStarted","Data":"fd1df8ed4eff28fa6e4c147213cd1c1ed3641fd66419624baca0cfa9936d3b9b"} Jul 23 09:18:23.594751 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:23.594701 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-84c8fd6f77-4txkh" podStartSLOduration=1.979535158 podStartE2EDuration="3.594682466s" totalImagesPullingTime="1.615147308s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:18:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:18:23.592200863 +0000 UTC m=+92.131479480" watchObservedRunningTime="2026-07-23 09:18:23.594682466 +0000 UTC m=+92.133961044" Jul 23 09:18:23.629864 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:23.629805 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.532328514 podStartE2EDuration="6.629786715s" totalImagesPullingTime="4.081736888s" totalInitContainerRuntime="15.721313ms" isStatefulPod=false podCreationTimestamp="2026-07-23 09:18:17 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:18:23.628481473 +0000 UTC m=+92.167760050" watchObservedRunningTime="2026-07-23 09:18:23.629786715 +0000 UTC m=+92.169065293" Jul 23 09:18:24.579020 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:24.578987 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-9998cfcd5-b7c92" event={"ID":"a05745d8-9561-497d-8e07-d0b9c1ee6ba3","Type":"ContainerStarted","Data":"1a258930b456e01a52076bee67e5f362d24e66c7e46aa2703cc7c141b373fe70"} Jul 23 09:18:24.579429 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:24.579042 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-9998cfcd5-b7c92" event={"ID":"a05745d8-9561-497d-8e07-d0b9c1ee6ba3","Type":"ContainerStarted","Data":"f0c52d8183b8b77460508d3f7d8d1307bff109fa3fec9c68317849d551c32264"} Jul 23 09:18:24.579429 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:24.579062 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-9998cfcd5-b7c92" event={"ID":"a05745d8-9561-497d-8e07-d0b9c1ee6ba3","Type":"ContainerStarted","Data":"600c2dee4ff0547f3f856c5ea1bbd7ad6379ba29af5e1198265053ca6f7ed632"} Jul 23 09:18:24.979995 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:24.979963 2576 scope.go:117] "RemoveContainer" containerID="28bf8b78eaff551a1b5c0d969b593fe4b88c837f24a83c81a35eb3d64871a906" Jul 23 09:18:25.292887 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:25.292777 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-9998cfcd5-b7c92" podStartSLOduration=2.1466367 podStartE2EDuration="4.29275861s" totalImagesPullingTime="2.14612191s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:18:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:18:24.618044936 +0000 UTC m=+93.157323526" watchObservedRunningTime="2026-07-23 09:18:25.29275861 +0000 UTC m=+93.832037186" Jul 23 09:18:25.293425 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:25.293402 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7989bd97d7-g2rdd"] Jul 23 09:18:25.465008 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:25.464981 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:25.584950 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:25.584871 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-5szxg_4908aff3-196e-44fa-b8e6-c0e81b476433/console-operator/2.log" Jul 23 09:18:25.585401 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:25.585017 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-5szxg" event={"ID":"4908aff3-196e-44fa-b8e6-c0e81b476433","Type":"ContainerStarted","Data":"77f88b082c627b745ec92a7f866c39a33a544cec85822d0c30d3c6395a25c29e"} Jul 23 09:18:25.615171 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:25.615111 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-565b7b569d-5szxg" podStartSLOduration=54.206070362 podStartE2EDuration="1m1.615096106s" totalImagesPullingTime="7.409025744s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:17:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:18:25.614313671 +0000 UTC m=+94.153592248" watchObservedRunningTime="2026-07-23 09:18:25.615096106 +0000 UTC m=+94.154374683" Jul 23 09:18:25.619945 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:25.619920 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:25.740220 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:25.740184 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:25.761428 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:25.761400 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-5bc67dcd46-28qcl"] Jul 23 09:18:25.763613 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:25.763595 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-28qcl" Jul 23 09:18:25.767366 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:25.767335 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 23 09:18:25.767579 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:25.767562 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-8hzr8\"" Jul 23 09:18:25.768098 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:25.768083 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 23 09:18:25.785357 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:25.785336 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-28qcl"] Jul 23 09:18:25.805292 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:25.805269 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkln9\" (UniqueName: \"kubernetes.io/projected/5d5c84db-7503-4aec-8ef8-81ef0c6b6a9b-kube-api-access-wkln9\") pod \"downloads-5bc67dcd46-28qcl\" (UID: \"5d5c84db-7503-4aec-8ef8-81ef0c6b6a9b\") " pod="openshift-console/downloads-5bc67dcd46-28qcl" Jul 23 09:18:25.835157 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:25.835079 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:25.906514 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:25.906458 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkln9\" (UniqueName: \"kubernetes.io/projected/5d5c84db-7503-4aec-8ef8-81ef0c6b6a9b-kube-api-access-wkln9\") pod \"downloads-5bc67dcd46-28qcl\" (UID: \"5d5c84db-7503-4aec-8ef8-81ef0c6b6a9b\") " pod="openshift-console/downloads-5bc67dcd46-28qcl" Jul 23 09:18:25.914723 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:25.914698 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkln9\" (UniqueName: \"kubernetes.io/projected/5d5c84db-7503-4aec-8ef8-81ef0c6b6a9b-kube-api-access-wkln9\") pod \"downloads-5bc67dcd46-28qcl\" (UID: \"5d5c84db-7503-4aec-8ef8-81ef0c6b6a9b\") " pod="openshift-console/downloads-5bc67dcd46-28qcl" Jul 23 09:18:25.998074 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:25.998042 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:26.073039 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:26.073002 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-28qcl" Jul 23 09:18:26.099619 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:26.099555 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:26.220570 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:26.220420 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-28qcl"] Jul 23 09:18:26.223207 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:18:26.223178 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d5c84db_7503_4aec_8ef8_81ef0c6b6a9b.slice/crio-d9511d2a5c34fc0cc67d651f4aacdf6f10e725c92f880c54154746f924e1d182 WatchSource:0}: Error finding container d9511d2a5c34fc0cc67d651f4aacdf6f10e725c92f880c54154746f924e1d182: Status 404 returned error can't find the container with id d9511d2a5c34fc0cc67d651f4aacdf6f10e725c92f880c54154746f924e1d182 Jul 23 09:18:26.277090 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:26.277062 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:26.390716 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:26.390686 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:26.491647 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:26.491607 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:26.589810 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:26.589779 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-28qcl" event={"ID":"5d5c84db-7503-4aec-8ef8-81ef0c6b6a9b","Type":"ContainerStarted","Data":"d9511d2a5c34fc0cc67d651f4aacdf6f10e725c92f880c54154746f924e1d182"} Jul 23 09:18:26.788152 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:26.788077 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:26.890367 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:26.890338 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:27.241203 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:27.241171 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:27.476061 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:27.476023 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:27.663189 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:27.663155 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:28.043663 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:28.043509 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:28.144475 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:28.144427 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:29.429538 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:29.429504 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-b24sn" Jul 23 09:18:31.174353 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:31.174324 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:31.297530 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:31.297496 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:31.716588 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:31.716559 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6f55bc48bc-9gzff"] Jul 23 09:18:31.829458 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:31.829421 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f55bc48bc-9gzff"] Jul 23 09:18:31.829621 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:31.829581 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f55bc48bc-9gzff" Jul 23 09:18:31.832382 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:31.832350 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-r6hr8\"" Jul 23 09:18:31.832577 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:31.832347 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 23 09:18:31.832826 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:31.832805 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 23 09:18:31.832950 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:31.832846 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 23 09:18:31.832950 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:31.832857 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 23 09:18:31.832950 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:31.832811 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 23 09:18:31.960197 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:31.960153 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0cf9f282-d8c9-4bed-9ca3-2400ef455c62-console-serving-cert\") pod \"console-6f55bc48bc-9gzff\" (UID: \"0cf9f282-d8c9-4bed-9ca3-2400ef455c62\") " pod="openshift-console/console-6f55bc48bc-9gzff" Jul 23 09:18:31.960375 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:31.960225 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0cf9f282-d8c9-4bed-9ca3-2400ef455c62-service-ca\") pod \"console-6f55bc48bc-9gzff\" (UID: \"0cf9f282-d8c9-4bed-9ca3-2400ef455c62\") " pod="openshift-console/console-6f55bc48bc-9gzff" Jul 23 09:18:31.960375 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:31.960287 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0cf9f282-d8c9-4bed-9ca3-2400ef455c62-console-oauth-config\") pod \"console-6f55bc48bc-9gzff\" (UID: \"0cf9f282-d8c9-4bed-9ca3-2400ef455c62\") " pod="openshift-console/console-6f55bc48bc-9gzff" Jul 23 09:18:31.960475 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:31.960361 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0cf9f282-d8c9-4bed-9ca3-2400ef455c62-oauth-serving-cert\") pod \"console-6f55bc48bc-9gzff\" (UID: \"0cf9f282-d8c9-4bed-9ca3-2400ef455c62\") " pod="openshift-console/console-6f55bc48bc-9gzff" Jul 23 09:18:31.960526 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:31.960464 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0cf9f282-d8c9-4bed-9ca3-2400ef455c62-console-config\") pod \"console-6f55bc48bc-9gzff\" (UID: \"0cf9f282-d8c9-4bed-9ca3-2400ef455c62\") " pod="openshift-console/console-6f55bc48bc-9gzff" Jul 23 09:18:31.960526 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:31.960502 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d47g\" (UniqueName: \"kubernetes.io/projected/0cf9f282-d8c9-4bed-9ca3-2400ef455c62-kube-api-access-8d47g\") pod \"console-6f55bc48bc-9gzff\" (UID: \"0cf9f282-d8c9-4bed-9ca3-2400ef455c62\") " pod="openshift-console/console-6f55bc48bc-9gzff" Jul 23 09:18:32.061801 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:32.061710 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0cf9f282-d8c9-4bed-9ca3-2400ef455c62-oauth-serving-cert\") pod \"console-6f55bc48bc-9gzff\" (UID: \"0cf9f282-d8c9-4bed-9ca3-2400ef455c62\") " pod="openshift-console/console-6f55bc48bc-9gzff" Jul 23 09:18:32.061960 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:32.061878 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0cf9f282-d8c9-4bed-9ca3-2400ef455c62-console-config\") pod \"console-6f55bc48bc-9gzff\" (UID: \"0cf9f282-d8c9-4bed-9ca3-2400ef455c62\") " pod="openshift-console/console-6f55bc48bc-9gzff" Jul 23 09:18:32.061960 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:32.061916 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d47g\" (UniqueName: \"kubernetes.io/projected/0cf9f282-d8c9-4bed-9ca3-2400ef455c62-kube-api-access-8d47g\") pod \"console-6f55bc48bc-9gzff\" (UID: \"0cf9f282-d8c9-4bed-9ca3-2400ef455c62\") " pod="openshift-console/console-6f55bc48bc-9gzff" Jul 23 09:18:32.061960 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:32.061947 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0cf9f282-d8c9-4bed-9ca3-2400ef455c62-console-serving-cert\") pod \"console-6f55bc48bc-9gzff\" (UID: \"0cf9f282-d8c9-4bed-9ca3-2400ef455c62\") " pod="openshift-console/console-6f55bc48bc-9gzff" Jul 23 09:18:32.062118 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:32.062001 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0cf9f282-d8c9-4bed-9ca3-2400ef455c62-service-ca\") pod \"console-6f55bc48bc-9gzff\" (UID: \"0cf9f282-d8c9-4bed-9ca3-2400ef455c62\") " pod="openshift-console/console-6f55bc48bc-9gzff" Jul 23 09:18:32.062118 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:32.062053 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0cf9f282-d8c9-4bed-9ca3-2400ef455c62-console-oauth-config\") pod \"console-6f55bc48bc-9gzff\" (UID: \"0cf9f282-d8c9-4bed-9ca3-2400ef455c62\") " pod="openshift-console/console-6f55bc48bc-9gzff" Jul 23 09:18:32.062656 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:32.062595 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0cf9f282-d8c9-4bed-9ca3-2400ef455c62-oauth-serving-cert\") pod \"console-6f55bc48bc-9gzff\" (UID: \"0cf9f282-d8c9-4bed-9ca3-2400ef455c62\") " pod="openshift-console/console-6f55bc48bc-9gzff" Jul 23 09:18:32.062656 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:32.062637 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0cf9f282-d8c9-4bed-9ca3-2400ef455c62-console-config\") pod \"console-6f55bc48bc-9gzff\" (UID: \"0cf9f282-d8c9-4bed-9ca3-2400ef455c62\") " pod="openshift-console/console-6f55bc48bc-9gzff" Jul 23 09:18:32.062933 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:32.062890 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0cf9f282-d8c9-4bed-9ca3-2400ef455c62-service-ca\") pod \"console-6f55bc48bc-9gzff\" (UID: \"0cf9f282-d8c9-4bed-9ca3-2400ef455c62\") " pod="openshift-console/console-6f55bc48bc-9gzff" Jul 23 09:18:32.064897 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:32.064871 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0cf9f282-d8c9-4bed-9ca3-2400ef455c62-console-oauth-config\") pod \"console-6f55bc48bc-9gzff\" (UID: \"0cf9f282-d8c9-4bed-9ca3-2400ef455c62\") " pod="openshift-console/console-6f55bc48bc-9gzff" Jul 23 09:18:32.065209 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:32.065189 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0cf9f282-d8c9-4bed-9ca3-2400ef455c62-console-serving-cert\") pod \"console-6f55bc48bc-9gzff\" (UID: \"0cf9f282-d8c9-4bed-9ca3-2400ef455c62\") " pod="openshift-console/console-6f55bc48bc-9gzff" Jul 23 09:18:32.073042 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:32.073018 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d47g\" (UniqueName: \"kubernetes.io/projected/0cf9f282-d8c9-4bed-9ca3-2400ef455c62-kube-api-access-8d47g\") pod \"console-6f55bc48bc-9gzff\" (UID: \"0cf9f282-d8c9-4bed-9ca3-2400ef455c62\") " pod="openshift-console/console-6f55bc48bc-9gzff" Jul 23 09:18:32.142838 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:32.142793 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f55bc48bc-9gzff" Jul 23 09:18:32.349215 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:32.349140 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f55bc48bc-9gzff"] Jul 23 09:18:32.351963 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:18:32.351921 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cf9f282_d8c9_4bed_9ca3_2400ef455c62.slice/crio-7c176e77abf6c836384caa6fc8e2edf3f11dc282478feae1e6d7aa2fe17f23b3 WatchSource:0}: Error finding container 7c176e77abf6c836384caa6fc8e2edf3f11dc282478feae1e6d7aa2fe17f23b3: Status 404 returned error can't find the container with id 7c176e77abf6c836384caa6fc8e2edf3f11dc282478feae1e6d7aa2fe17f23b3 Jul 23 09:18:32.617512 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:32.617437 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f55bc48bc-9gzff" event={"ID":"0cf9f282-d8c9-4bed-9ca3-2400ef455c62","Type":"ContainerStarted","Data":"7c176e77abf6c836384caa6fc8e2edf3f11dc282478feae1e6d7aa2fe17f23b3"} Jul 23 09:18:34.636469 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:34.636381 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:34.696012 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:34.695966 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:35.585619 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:35.585579 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-5szxg" Jul 23 09:18:35.590098 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:35.590067 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-565b7b569d-5szxg" Jul 23 09:18:36.639370 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:36.639329 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f55bc48bc-9gzff" event={"ID":"0cf9f282-d8c9-4bed-9ca3-2400ef455c62","Type":"ContainerStarted","Data":"3eef79576259be7ab34bca19f4cf734b82e3a69a049d205ecf63da391bfc6aac"} Jul 23 09:18:36.660571 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:36.660524 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6f55bc48bc-9gzff" podStartSLOduration=2.003216358 podStartE2EDuration="5.660506748s" totalImagesPullingTime="3.65729039s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:18:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:18:36.658676739 +0000 UTC m=+105.197955318" watchObservedRunningTime="2026-07-23 09:18:36.660506748 +0000 UTC m=+105.199785324" Jul 23 09:18:38.024736 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:38.024697 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:39.448888 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:39.448852 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:40.239341 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:40.239312 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-595f97f64c-5zqx9"] Jul 23 09:18:40.251812 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:40.251786 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-595f97f64c-5zqx9" Jul 23 09:18:40.251959 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:40.251926 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-595f97f64c-5zqx9"] Jul 23 09:18:40.260419 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:40.260388 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 23 09:18:40.341202 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:40.341157 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1cf71e65-e876-45f1-b242-81a6b329dad8-console-serving-cert\") pod \"console-595f97f64c-5zqx9\" (UID: \"1cf71e65-e876-45f1-b242-81a6b329dad8\") " pod="openshift-console/console-595f97f64c-5zqx9" Jul 23 09:18:40.341409 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:40.341219 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1cf71e65-e876-45f1-b242-81a6b329dad8-oauth-serving-cert\") pod \"console-595f97f64c-5zqx9\" (UID: \"1cf71e65-e876-45f1-b242-81a6b329dad8\") " pod="openshift-console/console-595f97f64c-5zqx9" Jul 23 09:18:40.341409 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:40.341276 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x26q\" (UniqueName: \"kubernetes.io/projected/1cf71e65-e876-45f1-b242-81a6b329dad8-kube-api-access-6x26q\") pod \"console-595f97f64c-5zqx9\" (UID: \"1cf71e65-e876-45f1-b242-81a6b329dad8\") " pod="openshift-console/console-595f97f64c-5zqx9" Jul 23 09:18:40.341409 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:40.341398 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1cf71e65-e876-45f1-b242-81a6b329dad8-console-oauth-config\") pod \"console-595f97f64c-5zqx9\" (UID: \"1cf71e65-e876-45f1-b242-81a6b329dad8\") " pod="openshift-console/console-595f97f64c-5zqx9" Jul 23 09:18:40.341561 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:40.341441 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1cf71e65-e876-45f1-b242-81a6b329dad8-console-config\") pod \"console-595f97f64c-5zqx9\" (UID: \"1cf71e65-e876-45f1-b242-81a6b329dad8\") " pod="openshift-console/console-595f97f64c-5zqx9" Jul 23 09:18:40.341561 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:40.341470 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1cf71e65-e876-45f1-b242-81a6b329dad8-service-ca\") pod \"console-595f97f64c-5zqx9\" (UID: \"1cf71e65-e876-45f1-b242-81a6b329dad8\") " pod="openshift-console/console-595f97f64c-5zqx9" Jul 23 09:18:40.341561 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:40.341525 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1cf71e65-e876-45f1-b242-81a6b329dad8-trusted-ca-bundle\") pod \"console-595f97f64c-5zqx9\" (UID: \"1cf71e65-e876-45f1-b242-81a6b329dad8\") " pod="openshift-console/console-595f97f64c-5zqx9" Jul 23 09:18:40.442678 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:40.442635 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1cf71e65-e876-45f1-b242-81a6b329dad8-oauth-serving-cert\") pod \"console-595f97f64c-5zqx9\" (UID: \"1cf71e65-e876-45f1-b242-81a6b329dad8\") " pod="openshift-console/console-595f97f64c-5zqx9" Jul 23 09:18:40.442857 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:40.442688 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x26q\" (UniqueName: \"kubernetes.io/projected/1cf71e65-e876-45f1-b242-81a6b329dad8-kube-api-access-6x26q\") pod \"console-595f97f64c-5zqx9\" (UID: \"1cf71e65-e876-45f1-b242-81a6b329dad8\") " pod="openshift-console/console-595f97f64c-5zqx9" Jul 23 09:18:40.442857 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:40.442758 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1cf71e65-e876-45f1-b242-81a6b329dad8-console-oauth-config\") pod \"console-595f97f64c-5zqx9\" (UID: \"1cf71e65-e876-45f1-b242-81a6b329dad8\") " pod="openshift-console/console-595f97f64c-5zqx9" Jul 23 09:18:40.442857 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:40.442791 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1cf71e65-e876-45f1-b242-81a6b329dad8-console-config\") pod \"console-595f97f64c-5zqx9\" (UID: \"1cf71e65-e876-45f1-b242-81a6b329dad8\") " pod="openshift-console/console-595f97f64c-5zqx9" Jul 23 09:18:40.442857 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:40.442820 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1cf71e65-e876-45f1-b242-81a6b329dad8-service-ca\") pod \"console-595f97f64c-5zqx9\" (UID: \"1cf71e65-e876-45f1-b242-81a6b329dad8\") " pod="openshift-console/console-595f97f64c-5zqx9" Jul 23 09:18:40.443074 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:40.442875 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1cf71e65-e876-45f1-b242-81a6b329dad8-trusted-ca-bundle\") pod \"console-595f97f64c-5zqx9\" (UID: \"1cf71e65-e876-45f1-b242-81a6b329dad8\") " pod="openshift-console/console-595f97f64c-5zqx9" Jul 23 09:18:40.443074 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:40.442909 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1cf71e65-e876-45f1-b242-81a6b329dad8-console-serving-cert\") pod \"console-595f97f64c-5zqx9\" (UID: \"1cf71e65-e876-45f1-b242-81a6b329dad8\") " pod="openshift-console/console-595f97f64c-5zqx9" Jul 23 09:18:40.443615 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:40.443552 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1cf71e65-e876-45f1-b242-81a6b329dad8-oauth-serving-cert\") pod \"console-595f97f64c-5zqx9\" (UID: \"1cf71e65-e876-45f1-b242-81a6b329dad8\") " pod="openshift-console/console-595f97f64c-5zqx9" Jul 23 09:18:40.443615 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:40.443599 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1cf71e65-e876-45f1-b242-81a6b329dad8-service-ca\") pod \"console-595f97f64c-5zqx9\" (UID: \"1cf71e65-e876-45f1-b242-81a6b329dad8\") " pod="openshift-console/console-595f97f64c-5zqx9" Jul 23 09:18:40.444040 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:40.443972 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1cf71e65-e876-45f1-b242-81a6b329dad8-trusted-ca-bundle\") pod \"console-595f97f64c-5zqx9\" (UID: \"1cf71e65-e876-45f1-b242-81a6b329dad8\") " pod="openshift-console/console-595f97f64c-5zqx9" Jul 23 09:18:40.444175 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:40.444150 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1cf71e65-e876-45f1-b242-81a6b329dad8-console-config\") pod \"console-595f97f64c-5zqx9\" (UID: \"1cf71e65-e876-45f1-b242-81a6b329dad8\") " pod="openshift-console/console-595f97f64c-5zqx9" Jul 23 09:18:40.445894 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:40.445870 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1cf71e65-e876-45f1-b242-81a6b329dad8-console-serving-cert\") pod \"console-595f97f64c-5zqx9\" (UID: \"1cf71e65-e876-45f1-b242-81a6b329dad8\") " pod="openshift-console/console-595f97f64c-5zqx9" Jul 23 09:18:40.446045 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:40.446022 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1cf71e65-e876-45f1-b242-81a6b329dad8-console-oauth-config\") pod \"console-595f97f64c-5zqx9\" (UID: \"1cf71e65-e876-45f1-b242-81a6b329dad8\") " pod="openshift-console/console-595f97f64c-5zqx9" Jul 23 09:18:40.451185 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:40.451160 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x26q\" (UniqueName: \"kubernetes.io/projected/1cf71e65-e876-45f1-b242-81a6b329dad8-kube-api-access-6x26q\") pod \"console-595f97f64c-5zqx9\" (UID: \"1cf71e65-e876-45f1-b242-81a6b329dad8\") " pod="openshift-console/console-595f97f64c-5zqx9" Jul 23 09:18:40.565067 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:40.564979 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-595f97f64c-5zqx9" Jul 23 09:18:40.652398 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:40.652365 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:40.712454 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:40.712419 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:40.796206 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:40.796170 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-84c8fd6f77-4txkh" Jul 23 09:18:40.796397 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:40.796250 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-84c8fd6f77-4txkh" Jul 23 09:18:42.143194 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:42.143150 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6f55bc48bc-9gzff" Jul 23 09:18:42.143673 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:42.143266 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6f55bc48bc-9gzff" Jul 23 09:18:42.148881 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:42.148858 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6f55bc48bc-9gzff" Jul 23 09:18:42.669671 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:42.669640 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6f55bc48bc-9gzff" Jul 23 09:18:42.844349 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:42.844311 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:44.468730 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:44.468704 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-595f97f64c-5zqx9"] Jul 23 09:18:44.481070 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:18:44.481042 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cf71e65_e876_45f1_b242_81a6b329dad8.slice/crio-166fa64e652e99705c1ca97482451217097087f72d826a9c24ef622395e00659 WatchSource:0}: Error finding container 166fa64e652e99705c1ca97482451217097087f72d826a9c24ef622395e00659: Status 404 returned error can't find the container with id 166fa64e652e99705c1ca97482451217097087f72d826a9c24ef622395e00659 Jul 23 09:18:44.676010 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:44.675970 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-595f97f64c-5zqx9" event={"ID":"1cf71e65-e876-45f1-b242-81a6b329dad8","Type":"ContainerStarted","Data":"563e252c11ef79f1d2ba1bf72289a46ed120791566d194d9da8f5ad1a51ef4bd"} Jul 23 09:18:44.676210 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:44.676040 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-595f97f64c-5zqx9" event={"ID":"1cf71e65-e876-45f1-b242-81a6b329dad8","Type":"ContainerStarted","Data":"166fa64e652e99705c1ca97482451217097087f72d826a9c24ef622395e00659"} Jul 23 09:18:44.677802 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:44.677745 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-28qcl" event={"ID":"5d5c84db-7503-4aec-8ef8-81ef0c6b6a9b","Type":"ContainerStarted","Data":"4a4be001948cec4a54a18c4f20a69decf5dce1e82816900ec594200a94c64b19"} Jul 23 09:18:44.677916 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:44.677849 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-5bc67dcd46-28qcl" Jul 23 09:18:44.684621 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:44.684601 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-5bc67dcd46-28qcl" Jul 23 09:18:44.697482 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:44.697444 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-595f97f64c-5zqx9" podStartSLOduration=4.697430534 podStartE2EDuration="4.697430534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:18:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:18:44.696608482 +0000 UTC m=+113.235887093" watchObservedRunningTime="2026-07-23 09:18:44.697430534 +0000 UTC m=+113.236709133" Jul 23 09:18:44.719083 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:44.719037 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-5bc67dcd46-28qcl" podStartSLOduration=1.5187670770000001 podStartE2EDuration="19.719020383s" totalImagesPullingTime="18.200253306s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:18:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:18:44.717803726 +0000 UTC m=+113.257082304" watchObservedRunningTime="2026-07-23 09:18:44.719020383 +0000 UTC m=+113.258298959" Jul 23 09:18:46.057595 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:46.057541 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:46.143353 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:46.143320 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:49.835376 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:49.835336 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:49.890265 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:49.890208 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:50.313461 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:50.313419 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7989bd97d7-g2rdd" podUID="1260ceaa-4aa5-4130-9ea7-ff8212c2d751" containerName="registry" containerID="cri-o://cf22fe53a4e6e3c9cf57fb73bfbe576fcb984058e856fddabfd27bb839bffe16" gracePeriod=30 Jul 23 09:18:50.565863 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:50.565784 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-595f97f64c-5zqx9" Jul 23 09:18:50.566018 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:50.565885 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-595f97f64c-5zqx9" Jul 23 09:18:50.571534 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:50.571384 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-595f97f64c-5zqx9" Jul 23 09:18:50.610305 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:50.610272 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7989bd97d7-g2rdd" Jul 23 09:18:50.707155 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:50.707113 2576 generic.go:356] "Generic (PLEG): container finished" podID="1260ceaa-4aa5-4130-9ea7-ff8212c2d751" containerID="cf22fe53a4e6e3c9cf57fb73bfbe576fcb984058e856fddabfd27bb839bffe16" exitCode=0 Jul 23 09:18:50.707342 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:50.707166 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7989bd97d7-g2rdd" Jul 23 09:18:50.707342 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:50.707208 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7989bd97d7-g2rdd" event={"ID":"1260ceaa-4aa5-4130-9ea7-ff8212c2d751","Type":"ContainerDied","Data":"cf22fe53a4e6e3c9cf57fb73bfbe576fcb984058e856fddabfd27bb839bffe16"} Jul 23 09:18:50.707342 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:50.707291 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7989bd97d7-g2rdd" event={"ID":"1260ceaa-4aa5-4130-9ea7-ff8212c2d751","Type":"ContainerDied","Data":"702643edcea8ddfea1795989dbb6e483ddfb7e535d604d10f5102df266c78926"} Jul 23 09:18:50.707342 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:50.707321 2576 scope.go:117] "RemoveContainer" containerID="cf22fe53a4e6e3c9cf57fb73bfbe576fcb984058e856fddabfd27bb839bffe16" Jul 23 09:18:50.712298 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:50.712275 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-595f97f64c-5zqx9" Jul 23 09:18:50.717734 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:50.717714 2576 scope.go:117] "RemoveContainer" containerID="cf22fe53a4e6e3c9cf57fb73bfbe576fcb984058e856fddabfd27bb839bffe16" Jul 23 09:18:50.718023 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:18:50.718001 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf22fe53a4e6e3c9cf57fb73bfbe576fcb984058e856fddabfd27bb839bffe16\": container with ID starting with cf22fe53a4e6e3c9cf57fb73bfbe576fcb984058e856fddabfd27bb839bffe16 not found: ID does not exist" containerID="cf22fe53a4e6e3c9cf57fb73bfbe576fcb984058e856fddabfd27bb839bffe16" Jul 23 09:18:50.718106 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:50.718034 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf22fe53a4e6e3c9cf57fb73bfbe576fcb984058e856fddabfd27bb839bffe16"} err="failed to get container status \"cf22fe53a4e6e3c9cf57fb73bfbe576fcb984058e856fddabfd27bb839bffe16\": rpc error: code = NotFound desc = could not find container \"cf22fe53a4e6e3c9cf57fb73bfbe576fcb984058e856fddabfd27bb839bffe16\": container with ID starting with cf22fe53a4e6e3c9cf57fb73bfbe576fcb984058e856fddabfd27bb839bffe16 not found: ID does not exist" Jul 23 09:18:50.748583 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:50.748556 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1260ceaa-4aa5-4130-9ea7-ff8212c2d751-registry-certificates\") pod \"1260ceaa-4aa5-4130-9ea7-ff8212c2d751\" (UID: \"1260ceaa-4aa5-4130-9ea7-ff8212c2d751\") " Jul 23 09:18:50.748737 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:50.748602 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6vvmp\" (UniqueName: \"kubernetes.io/projected/1260ceaa-4aa5-4130-9ea7-ff8212c2d751-kube-api-access-6vvmp\") pod \"1260ceaa-4aa5-4130-9ea7-ff8212c2d751\" (UID: \"1260ceaa-4aa5-4130-9ea7-ff8212c2d751\") " Jul 23 09:18:50.748737 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:50.748645 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1260ceaa-4aa5-4130-9ea7-ff8212c2d751-image-registry-private-configuration\") pod \"1260ceaa-4aa5-4130-9ea7-ff8212c2d751\" (UID: \"1260ceaa-4aa5-4130-9ea7-ff8212c2d751\") " Jul 23 09:18:50.748737 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:50.748684 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1260ceaa-4aa5-4130-9ea7-ff8212c2d751-bound-sa-token\") pod \"1260ceaa-4aa5-4130-9ea7-ff8212c2d751\" (UID: \"1260ceaa-4aa5-4130-9ea7-ff8212c2d751\") " Jul 23 09:18:50.748920 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:50.748768 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1260ceaa-4aa5-4130-9ea7-ff8212c2d751-ca-trust-extracted\") pod \"1260ceaa-4aa5-4130-9ea7-ff8212c2d751\" (UID: \"1260ceaa-4aa5-4130-9ea7-ff8212c2d751\") " Jul 23 09:18:50.748920 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:50.748801 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1260ceaa-4aa5-4130-9ea7-ff8212c2d751-trusted-ca\") pod \"1260ceaa-4aa5-4130-9ea7-ff8212c2d751\" (UID: \"1260ceaa-4aa5-4130-9ea7-ff8212c2d751\") " Jul 23 09:18:50.748920 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:50.748853 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1260ceaa-4aa5-4130-9ea7-ff8212c2d751-registry-tls\") pod \"1260ceaa-4aa5-4130-9ea7-ff8212c2d751\" (UID: \"1260ceaa-4aa5-4130-9ea7-ff8212c2d751\") " Jul 23 09:18:50.748920 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:50.748882 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1260ceaa-4aa5-4130-9ea7-ff8212c2d751-installation-pull-secrets\") pod \"1260ceaa-4aa5-4130-9ea7-ff8212c2d751\" (UID: \"1260ceaa-4aa5-4130-9ea7-ff8212c2d751\") " Jul 23 09:18:50.749115 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:50.748980 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1260ceaa-4aa5-4130-9ea7-ff8212c2d751-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "1260ceaa-4aa5-4130-9ea7-ff8212c2d751" (UID: "1260ceaa-4aa5-4130-9ea7-ff8212c2d751"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 09:18:50.749277 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:50.749208 2576 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1260ceaa-4aa5-4130-9ea7-ff8212c2d751-registry-certificates\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:18:50.749376 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:50.749356 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1260ceaa-4aa5-4130-9ea7-ff8212c2d751-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "1260ceaa-4aa5-4130-9ea7-ff8212c2d751" (UID: "1260ceaa-4aa5-4130-9ea7-ff8212c2d751"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 09:18:50.763357 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:50.763322 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1260ceaa-4aa5-4130-9ea7-ff8212c2d751-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "1260ceaa-4aa5-4130-9ea7-ff8212c2d751" (UID: "1260ceaa-4aa5-4130-9ea7-ff8212c2d751"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 09:18:50.764048 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:50.764020 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f55bc48bc-9gzff"] Jul 23 09:18:50.766840 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:50.766801 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1260ceaa-4aa5-4130-9ea7-ff8212c2d751-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "1260ceaa-4aa5-4130-9ea7-ff8212c2d751" (UID: "1260ceaa-4aa5-4130-9ea7-ff8212c2d751"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 09:18:50.767277 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:50.767248 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1260ceaa-4aa5-4130-9ea7-ff8212c2d751-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "1260ceaa-4aa5-4130-9ea7-ff8212c2d751" (UID: "1260ceaa-4aa5-4130-9ea7-ff8212c2d751"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 09:18:50.768081 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:50.768037 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1260ceaa-4aa5-4130-9ea7-ff8212c2d751-kube-api-access-6vvmp" (OuterVolumeSpecName: "kube-api-access-6vvmp") pod "1260ceaa-4aa5-4130-9ea7-ff8212c2d751" (UID: "1260ceaa-4aa5-4130-9ea7-ff8212c2d751"). InnerVolumeSpecName "kube-api-access-6vvmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 09:18:50.768979 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:50.768946 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1260ceaa-4aa5-4130-9ea7-ff8212c2d751-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "1260ceaa-4aa5-4130-9ea7-ff8212c2d751" (UID: "1260ceaa-4aa5-4130-9ea7-ff8212c2d751"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 09:18:50.770771 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:50.770731 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1260ceaa-4aa5-4130-9ea7-ff8212c2d751-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "1260ceaa-4aa5-4130-9ea7-ff8212c2d751" (UID: "1260ceaa-4aa5-4130-9ea7-ff8212c2d751"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 09:18:50.850531 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:50.850442 2576 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1260ceaa-4aa5-4130-9ea7-ff8212c2d751-ca-trust-extracted\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:18:50.850531 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:50.850471 2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1260ceaa-4aa5-4130-9ea7-ff8212c2d751-trusted-ca\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:18:50.850531 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:50.850484 2576 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1260ceaa-4aa5-4130-9ea7-ff8212c2d751-registry-tls\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:18:50.850531 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:50.850494 2576 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1260ceaa-4aa5-4130-9ea7-ff8212c2d751-installation-pull-secrets\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:18:50.850531 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:50.850508 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vvmp\" (UniqueName: \"kubernetes.io/projected/1260ceaa-4aa5-4130-9ea7-ff8212c2d751-kube-api-access-6vvmp\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:18:50.850531 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:50.850520 2576 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1260ceaa-4aa5-4130-9ea7-ff8212c2d751-image-registry-private-configuration\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:18:50.850531 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:50.850530 2576 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1260ceaa-4aa5-4130-9ea7-ff8212c2d751-bound-sa-token\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:18:51.024404 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:51.024369 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7989bd97d7-g2rdd"] Jul 23 09:18:51.030328 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:51.030300 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7989bd97d7-g2rdd"] Jul 23 09:18:51.052850 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:51.052827 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:51.113109 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:51.113042 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:52.720093 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:52.720052 2576 generic.go:356] "Generic (PLEG): container finished" podID="989625f0-3f97-4f66-8d67-2e8f47a13a6c" containerID="950301a33c82b5d5b3316bfd0c420dffe3d528f0ce62f801a8a6d18ff1ded372" exitCode=0 Jul 23 09:18:52.720557 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:52.720139 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-pxlst" event={"ID":"989625f0-3f97-4f66-8d67-2e8f47a13a6c","Type":"ContainerDied","Data":"950301a33c82b5d5b3316bfd0c420dffe3d528f0ce62f801a8a6d18ff1ded372"} Jul 23 09:18:52.720684 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:52.720659 2576 scope.go:117] "RemoveContainer" containerID="950301a33c82b5d5b3316bfd0c420dffe3d528f0ce62f801a8a6d18ff1ded372" Jul 23 09:18:52.986842 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:52.986759 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1260ceaa-4aa5-4130-9ea7-ff8212c2d751" path="/var/lib/kubelet/pods/1260ceaa-4aa5-4130-9ea7-ff8212c2d751/volumes" Jul 23 09:18:53.727024 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:53.726988 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-pxlst" event={"ID":"989625f0-3f97-4f66-8d67-2e8f47a13a6c","Type":"ContainerStarted","Data":"aaca9f99b8c039de9bb4723d9e64c91bb7abe2050954e0262cb668c100411a7a"} Jul 23 09:18:55.673806 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:55.673762 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:18:55.742158 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:18:55.742124 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:19:00.801114 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:00.801086 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-84c8fd6f77-4txkh" Jul 23 09:19:00.804919 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:00.804899 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-84c8fd6f77-4txkh" Jul 23 09:19:02.767074 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:02.767039 2576 generic.go:356] "Generic (PLEG): container finished" podID="ef3505b4-c02c-487a-8314-f6793087ad3a" containerID="3f2f1f1a0b9799ea38e9b88cf3926d880b58c666ff994e48992234b54d8281bf" exitCode=0 Jul 23 09:19:02.767462 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:02.767115 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-nfv7r" event={"ID":"ef3505b4-c02c-487a-8314-f6793087ad3a","Type":"ContainerDied","Data":"3f2f1f1a0b9799ea38e9b88cf3926d880b58c666ff994e48992234b54d8281bf"} Jul 23 09:19:02.767462 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:02.767449 2576 scope.go:117] "RemoveContainer" containerID="3f2f1f1a0b9799ea38e9b88cf3926d880b58c666ff994e48992234b54d8281bf" Jul 23 09:19:03.772532 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:03.772498 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-nfv7r" event={"ID":"ef3505b4-c02c-487a-8314-f6793087ad3a","Type":"ContainerStarted","Data":"d946825f033d8452b91970a43dcbcb2db5949c3d897d655014e636019a60e51e"} Jul 23 09:19:07.794069 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:07.794035 2576 generic.go:356] "Generic (PLEG): container finished" podID="b5c91d72-2e17-44e7-83ab-4b9ce3a89b43" containerID="6a69790c16310ccc73f0eff984cf246790eec31b639f512ab59121de140d602b" exitCode=0 Jul 23 09:19:07.794489 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:07.794115 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-sp78t" event={"ID":"b5c91d72-2e17-44e7-83ab-4b9ce3a89b43","Type":"ContainerDied","Data":"6a69790c16310ccc73f0eff984cf246790eec31b639f512ab59121de140d602b"} Jul 23 09:19:07.794555 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:07.794539 2576 scope.go:117] "RemoveContainer" containerID="6a69790c16310ccc73f0eff984cf246790eec31b639f512ab59121de140d602b" Jul 23 09:19:08.800043 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:08.800004 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-sp78t" event={"ID":"b5c91d72-2e17-44e7-83ab-4b9ce3a89b43","Type":"ContainerStarted","Data":"85a45f11df636f412b6d8b8cb29c0baf13573748a19349c183bee2cdb63b8761"} Jul 23 09:19:13.787136 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:13.787104 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:19:15.792898 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:15.792830 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6f55bc48bc-9gzff" podUID="0cf9f282-d8c9-4bed-9ca3-2400ef455c62" containerName="console" containerID="cri-o://3eef79576259be7ab34bca19f4cf734b82e3a69a049d205ecf63da391bfc6aac" gracePeriod=15 Jul 23 09:19:16.066494 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:16.066433 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f55bc48bc-9gzff" Jul 23 09:19:16.169051 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:16.169022 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0cf9f282-d8c9-4bed-9ca3-2400ef455c62-console-serving-cert\") pod \"0cf9f282-d8c9-4bed-9ca3-2400ef455c62\" (UID: \"0cf9f282-d8c9-4bed-9ca3-2400ef455c62\") " Jul 23 09:19:16.169224 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:16.169096 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0cf9f282-d8c9-4bed-9ca3-2400ef455c62-service-ca\") pod \"0cf9f282-d8c9-4bed-9ca3-2400ef455c62\" (UID: \"0cf9f282-d8c9-4bed-9ca3-2400ef455c62\") " Jul 23 09:19:16.169224 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:16.169122 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0cf9f282-d8c9-4bed-9ca3-2400ef455c62-console-oauth-config\") pod \"0cf9f282-d8c9-4bed-9ca3-2400ef455c62\" (UID: \"0cf9f282-d8c9-4bed-9ca3-2400ef455c62\") " Jul 23 09:19:16.169224 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:16.169154 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8d47g\" (UniqueName: \"kubernetes.io/projected/0cf9f282-d8c9-4bed-9ca3-2400ef455c62-kube-api-access-8d47g\") pod \"0cf9f282-d8c9-4bed-9ca3-2400ef455c62\" (UID: \"0cf9f282-d8c9-4bed-9ca3-2400ef455c62\") " Jul 23 09:19:16.169224 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:16.169177 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0cf9f282-d8c9-4bed-9ca3-2400ef455c62-console-config\") pod \"0cf9f282-d8c9-4bed-9ca3-2400ef455c62\" (UID: \"0cf9f282-d8c9-4bed-9ca3-2400ef455c62\") " Jul 23 09:19:16.169224 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:16.169202 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0cf9f282-d8c9-4bed-9ca3-2400ef455c62-oauth-serving-cert\") pod \"0cf9f282-d8c9-4bed-9ca3-2400ef455c62\" (UID: \"0cf9f282-d8c9-4bed-9ca3-2400ef455c62\") " Jul 23 09:19:16.169616 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:16.169587 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cf9f282-d8c9-4bed-9ca3-2400ef455c62-service-ca" (OuterVolumeSpecName: "service-ca") pod "0cf9f282-d8c9-4bed-9ca3-2400ef455c62" (UID: "0cf9f282-d8c9-4bed-9ca3-2400ef455c62"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 09:19:16.169711 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:16.169601 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cf9f282-d8c9-4bed-9ca3-2400ef455c62-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "0cf9f282-d8c9-4bed-9ca3-2400ef455c62" (UID: "0cf9f282-d8c9-4bed-9ca3-2400ef455c62"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 09:19:16.169711 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:16.169589 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cf9f282-d8c9-4bed-9ca3-2400ef455c62-console-config" (OuterVolumeSpecName: "console-config") pod "0cf9f282-d8c9-4bed-9ca3-2400ef455c62" (UID: "0cf9f282-d8c9-4bed-9ca3-2400ef455c62"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 09:19:16.171459 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:16.171430 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cf9f282-d8c9-4bed-9ca3-2400ef455c62-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "0cf9f282-d8c9-4bed-9ca3-2400ef455c62" (UID: "0cf9f282-d8c9-4bed-9ca3-2400ef455c62"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 09:19:16.171549 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:16.171485 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cf9f282-d8c9-4bed-9ca3-2400ef455c62-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "0cf9f282-d8c9-4bed-9ca3-2400ef455c62" (UID: "0cf9f282-d8c9-4bed-9ca3-2400ef455c62"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 09:19:16.171549 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:16.171528 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cf9f282-d8c9-4bed-9ca3-2400ef455c62-kube-api-access-8d47g" (OuterVolumeSpecName: "kube-api-access-8d47g") pod "0cf9f282-d8c9-4bed-9ca3-2400ef455c62" (UID: "0cf9f282-d8c9-4bed-9ca3-2400ef455c62"). InnerVolumeSpecName "kube-api-access-8d47g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 09:19:16.270802 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:16.270772 2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0cf9f282-d8c9-4bed-9ca3-2400ef455c62-service-ca\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:19:16.270802 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:16.270799 2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0cf9f282-d8c9-4bed-9ca3-2400ef455c62-console-oauth-config\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:19:16.270802 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:16.270810 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8d47g\" (UniqueName: \"kubernetes.io/projected/0cf9f282-d8c9-4bed-9ca3-2400ef455c62-kube-api-access-8d47g\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:19:16.271026 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:16.270820 2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0cf9f282-d8c9-4bed-9ca3-2400ef455c62-console-config\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:19:16.271026 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:16.270833 2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0cf9f282-d8c9-4bed-9ca3-2400ef455c62-oauth-serving-cert\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:19:16.271026 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:16.270844 2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0cf9f282-d8c9-4bed-9ca3-2400ef455c62-console-serving-cert\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:19:16.832140 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:16.832103 2576 generic.go:356] "Generic (PLEG): container finished" podID="0cf9f282-d8c9-4bed-9ca3-2400ef455c62" containerID="3eef79576259be7ab34bca19f4cf734b82e3a69a049d205ecf63da391bfc6aac" exitCode=0 Jul 23 09:19:16.832553 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:16.832148 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f55bc48bc-9gzff" Jul 23 09:19:16.832553 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:16.832169 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f55bc48bc-9gzff" event={"ID":"0cf9f282-d8c9-4bed-9ca3-2400ef455c62","Type":"ContainerDied","Data":"3eef79576259be7ab34bca19f4cf734b82e3a69a049d205ecf63da391bfc6aac"} Jul 23 09:19:16.832553 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:16.832220 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f55bc48bc-9gzff" event={"ID":"0cf9f282-d8c9-4bed-9ca3-2400ef455c62","Type":"ContainerDied","Data":"7c176e77abf6c836384caa6fc8e2edf3f11dc282478feae1e6d7aa2fe17f23b3"} Jul 23 09:19:16.832553 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:16.832258 2576 scope.go:117] "RemoveContainer" containerID="3eef79576259be7ab34bca19f4cf734b82e3a69a049d205ecf63da391bfc6aac" Jul 23 09:19:16.840680 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:16.840664 2576 scope.go:117] "RemoveContainer" containerID="3eef79576259be7ab34bca19f4cf734b82e3a69a049d205ecf63da391bfc6aac" Jul 23 09:19:16.840914 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:19:16.840891 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3eef79576259be7ab34bca19f4cf734b82e3a69a049d205ecf63da391bfc6aac\": container with ID starting with 3eef79576259be7ab34bca19f4cf734b82e3a69a049d205ecf63da391bfc6aac not found: ID does not exist" containerID="3eef79576259be7ab34bca19f4cf734b82e3a69a049d205ecf63da391bfc6aac" Jul 23 09:19:16.841007 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:16.840919 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3eef79576259be7ab34bca19f4cf734b82e3a69a049d205ecf63da391bfc6aac"} err="failed to get container status \"3eef79576259be7ab34bca19f4cf734b82e3a69a049d205ecf63da391bfc6aac\": rpc error: code = NotFound desc = could not find container \"3eef79576259be7ab34bca19f4cf734b82e3a69a049d205ecf63da391bfc6aac\": container with ID starting with 3eef79576259be7ab34bca19f4cf734b82e3a69a049d205ecf63da391bfc6aac not found: ID does not exist" Jul 23 09:19:16.859440 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:16.859419 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f55bc48bc-9gzff"] Jul 23 09:19:16.863533 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:16.863509 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6f55bc48bc-9gzff"] Jul 23 09:19:16.984377 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:16.984341 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0cf9f282-d8c9-4bed-9ca3-2400ef455c62" path="/var/lib/kubelet/pods/0cf9f282-d8c9-4bed-9ca3-2400ef455c62/volumes" Jul 23 09:19:32.258961 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:32.258920 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:19:32.348080 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:32.348041 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:19:36.644227 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:36.644190 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 23 09:19:36.644779 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:36.644493 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d3c507db-91e8-4eb1-94bb-96dd05b53eac" containerName="alertmanager" containerID="cri-o://322ee8b69a743b8a3e29861ad366fde0f351322fb168d7ef4ec03ca153fd2174" gracePeriod=120 Jul 23 09:19:36.644779 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:36.644553 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d3c507db-91e8-4eb1-94bb-96dd05b53eac" containerName="kube-rbac-proxy-metric" containerID="cri-o://4505e914ce83900d19afb88d4b8b6fb9e7ce9e0d985bf7b7c1a0c9a68db83b59" gracePeriod=120 Jul 23 09:19:36.644779 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:36.644572 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d3c507db-91e8-4eb1-94bb-96dd05b53eac" containerName="kube-rbac-proxy-web" containerID="cri-o://02afea67ff99cccd3c74c1cb11317a9fba222f6e961cf5c74e2349eba5c26056" gracePeriod=120 Jul 23 09:19:36.644779 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:36.644585 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d3c507db-91e8-4eb1-94bb-96dd05b53eac" containerName="config-reloader" containerID="cri-o://7b426a2b8c29b42471a3c8abd3edc6803bf74965794f3fe430bd53de90f02bf8" gracePeriod=120 Jul 23 09:19:36.644779 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:36.644626 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d3c507db-91e8-4eb1-94bb-96dd05b53eac" containerName="kube-rbac-proxy" containerID="cri-o://4399f6c4bd60f1d8562862c68ff034ddacd38fa3f30a32954ce5b355bd7e4f0e" gracePeriod=120 Jul 23 09:19:36.644779 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:36.644684 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d3c507db-91e8-4eb1-94bb-96dd05b53eac" containerName="prom-label-proxy" containerID="cri-o://fd1df8ed4eff28fa6e4c147213cd1c1ed3641fd66419624baca0cfa9936d3b9b" gracePeriod=120 Jul 23 09:19:36.911026 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:36.910942 2576 generic.go:356] "Generic (PLEG): container finished" podID="d3c507db-91e8-4eb1-94bb-96dd05b53eac" containerID="fd1df8ed4eff28fa6e4c147213cd1c1ed3641fd66419624baca0cfa9936d3b9b" exitCode=0 Jul 23 09:19:36.911026 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:36.910967 2576 generic.go:356] "Generic (PLEG): container finished" podID="d3c507db-91e8-4eb1-94bb-96dd05b53eac" containerID="4399f6c4bd60f1d8562862c68ff034ddacd38fa3f30a32954ce5b355bd7e4f0e" exitCode=0 Jul 23 09:19:36.911026 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:36.910973 2576 generic.go:356] "Generic (PLEG): container finished" podID="d3c507db-91e8-4eb1-94bb-96dd05b53eac" containerID="7b426a2b8c29b42471a3c8abd3edc6803bf74965794f3fe430bd53de90f02bf8" exitCode=0 Jul 23 09:19:36.911026 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:36.910979 2576 generic.go:356] "Generic (PLEG): container finished" podID="d3c507db-91e8-4eb1-94bb-96dd05b53eac" containerID="322ee8b69a743b8a3e29861ad366fde0f351322fb168d7ef4ec03ca153fd2174" exitCode=0 Jul 23 09:19:36.911364 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:36.911019 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d3c507db-91e8-4eb1-94bb-96dd05b53eac","Type":"ContainerDied","Data":"fd1df8ed4eff28fa6e4c147213cd1c1ed3641fd66419624baca0cfa9936d3b9b"} Jul 23 09:19:36.911364 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:36.911100 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d3c507db-91e8-4eb1-94bb-96dd05b53eac","Type":"ContainerDied","Data":"4399f6c4bd60f1d8562862c68ff034ddacd38fa3f30a32954ce5b355bd7e4f0e"} Jul 23 09:19:36.911364 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:36.911141 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d3c507db-91e8-4eb1-94bb-96dd05b53eac","Type":"ContainerDied","Data":"7b426a2b8c29b42471a3c8abd3edc6803bf74965794f3fe430bd53de90f02bf8"} Jul 23 09:19:36.911364 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:36.911177 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d3c507db-91e8-4eb1-94bb-96dd05b53eac","Type":"ContainerDied","Data":"322ee8b69a743b8a3e29861ad366fde0f351322fb168d7ef4ec03ca153fd2174"} Jul 23 09:19:37.894548 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:37.894522 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:19:37.917783 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:37.917754 2576 generic.go:356] "Generic (PLEG): container finished" podID="d3c507db-91e8-4eb1-94bb-96dd05b53eac" containerID="4505e914ce83900d19afb88d4b8b6fb9e7ce9e0d985bf7b7c1a0c9a68db83b59" exitCode=0 Jul 23 09:19:37.917783 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:37.917776 2576 generic.go:356] "Generic (PLEG): container finished" podID="d3c507db-91e8-4eb1-94bb-96dd05b53eac" containerID="02afea67ff99cccd3c74c1cb11317a9fba222f6e961cf5c74e2349eba5c26056" exitCode=0 Jul 23 09:19:37.917963 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:37.917835 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:19:37.917963 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:37.917842 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d3c507db-91e8-4eb1-94bb-96dd05b53eac","Type":"ContainerDied","Data":"4505e914ce83900d19afb88d4b8b6fb9e7ce9e0d985bf7b7c1a0c9a68db83b59"} Jul 23 09:19:37.917963 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:37.917928 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d3c507db-91e8-4eb1-94bb-96dd05b53eac","Type":"ContainerDied","Data":"02afea67ff99cccd3c74c1cb11317a9fba222f6e961cf5c74e2349eba5c26056"} Jul 23 09:19:37.918076 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:37.917972 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d3c507db-91e8-4eb1-94bb-96dd05b53eac","Type":"ContainerDied","Data":"52237837a1b1b89c883bac406b35428ba37c5b55f2971fd020bf53965df89791"} Jul 23 09:19:37.918076 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:37.917991 2576 scope.go:117] "RemoveContainer" containerID="fd1df8ed4eff28fa6e4c147213cd1c1ed3641fd66419624baca0cfa9936d3b9b" Jul 23 09:19:37.925369 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:37.925349 2576 scope.go:117] "RemoveContainer" containerID="4505e914ce83900d19afb88d4b8b6fb9e7ce9e0d985bf7b7c1a0c9a68db83b59" Jul 23 09:19:37.932864 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:37.932837 2576 scope.go:117] "RemoveContainer" containerID="4399f6c4bd60f1d8562862c68ff034ddacd38fa3f30a32954ce5b355bd7e4f0e" Jul 23 09:19:37.940193 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:37.940173 2576 scope.go:117] "RemoveContainer" containerID="02afea67ff99cccd3c74c1cb11317a9fba222f6e961cf5c74e2349eba5c26056" Jul 23 09:19:37.947629 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:37.947610 2576 scope.go:117] "RemoveContainer" containerID="7b426a2b8c29b42471a3c8abd3edc6803bf74965794f3fe430bd53de90f02bf8" Jul 23 09:19:37.954621 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:37.954590 2576 scope.go:117] "RemoveContainer" containerID="322ee8b69a743b8a3e29861ad366fde0f351322fb168d7ef4ec03ca153fd2174" Jul 23 09:19:37.961712 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:37.961693 2576 scope.go:117] "RemoveContainer" containerID="f0b814f4809098312a191fde862386c052a2d25754fd03de5d2e3d235444de94" Jul 23 09:19:37.969534 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:37.969517 2576 scope.go:117] "RemoveContainer" containerID="fd1df8ed4eff28fa6e4c147213cd1c1ed3641fd66419624baca0cfa9936d3b9b" Jul 23 09:19:37.969806 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:19:37.969782 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd1df8ed4eff28fa6e4c147213cd1c1ed3641fd66419624baca0cfa9936d3b9b\": container with ID starting with fd1df8ed4eff28fa6e4c147213cd1c1ed3641fd66419624baca0cfa9936d3b9b not found: ID does not exist" containerID="fd1df8ed4eff28fa6e4c147213cd1c1ed3641fd66419624baca0cfa9936d3b9b" Jul 23 09:19:37.969906 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:37.969812 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd1df8ed4eff28fa6e4c147213cd1c1ed3641fd66419624baca0cfa9936d3b9b"} err="failed to get container status \"fd1df8ed4eff28fa6e4c147213cd1c1ed3641fd66419624baca0cfa9936d3b9b\": rpc error: code = NotFound desc = could not find container \"fd1df8ed4eff28fa6e4c147213cd1c1ed3641fd66419624baca0cfa9936d3b9b\": container with ID starting with fd1df8ed4eff28fa6e4c147213cd1c1ed3641fd66419624baca0cfa9936d3b9b not found: ID does not exist" Jul 23 09:19:37.969906 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:37.969832 2576 scope.go:117] "RemoveContainer" containerID="4505e914ce83900d19afb88d4b8b6fb9e7ce9e0d985bf7b7c1a0c9a68db83b59" Jul 23 09:19:37.970124 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:19:37.970088 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4505e914ce83900d19afb88d4b8b6fb9e7ce9e0d985bf7b7c1a0c9a68db83b59\": container with ID starting with 4505e914ce83900d19afb88d4b8b6fb9e7ce9e0d985bf7b7c1a0c9a68db83b59 not found: ID does not exist" containerID="4505e914ce83900d19afb88d4b8b6fb9e7ce9e0d985bf7b7c1a0c9a68db83b59" Jul 23 09:19:37.970206 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:37.970138 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4505e914ce83900d19afb88d4b8b6fb9e7ce9e0d985bf7b7c1a0c9a68db83b59"} err="failed to get container status \"4505e914ce83900d19afb88d4b8b6fb9e7ce9e0d985bf7b7c1a0c9a68db83b59\": rpc error: code = NotFound desc = could not find container \"4505e914ce83900d19afb88d4b8b6fb9e7ce9e0d985bf7b7c1a0c9a68db83b59\": container with ID starting with 4505e914ce83900d19afb88d4b8b6fb9e7ce9e0d985bf7b7c1a0c9a68db83b59 not found: ID does not exist" Jul 23 09:19:37.970206 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:37.970163 2576 scope.go:117] "RemoveContainer" containerID="4399f6c4bd60f1d8562862c68ff034ddacd38fa3f30a32954ce5b355bd7e4f0e" Jul 23 09:19:37.970501 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:19:37.970484 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4399f6c4bd60f1d8562862c68ff034ddacd38fa3f30a32954ce5b355bd7e4f0e\": container with ID starting with 4399f6c4bd60f1d8562862c68ff034ddacd38fa3f30a32954ce5b355bd7e4f0e not found: ID does not exist" containerID="4399f6c4bd60f1d8562862c68ff034ddacd38fa3f30a32954ce5b355bd7e4f0e" Jul 23 09:19:37.970567 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:37.970508 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4399f6c4bd60f1d8562862c68ff034ddacd38fa3f30a32954ce5b355bd7e4f0e"} err="failed to get container status \"4399f6c4bd60f1d8562862c68ff034ddacd38fa3f30a32954ce5b355bd7e4f0e\": rpc error: code = NotFound desc = could not find container \"4399f6c4bd60f1d8562862c68ff034ddacd38fa3f30a32954ce5b355bd7e4f0e\": container with ID starting with 4399f6c4bd60f1d8562862c68ff034ddacd38fa3f30a32954ce5b355bd7e4f0e not found: ID does not exist" Jul 23 09:19:37.970567 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:37.970523 2576 scope.go:117] "RemoveContainer" containerID="02afea67ff99cccd3c74c1cb11317a9fba222f6e961cf5c74e2349eba5c26056" Jul 23 09:19:37.970769 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:19:37.970748 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02afea67ff99cccd3c74c1cb11317a9fba222f6e961cf5c74e2349eba5c26056\": container with ID starting with 02afea67ff99cccd3c74c1cb11317a9fba222f6e961cf5c74e2349eba5c26056 not found: ID does not exist" containerID="02afea67ff99cccd3c74c1cb11317a9fba222f6e961cf5c74e2349eba5c26056" Jul 23 09:19:37.970808 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:37.970777 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02afea67ff99cccd3c74c1cb11317a9fba222f6e961cf5c74e2349eba5c26056"} err="failed to get container status \"02afea67ff99cccd3c74c1cb11317a9fba222f6e961cf5c74e2349eba5c26056\": rpc error: code = NotFound desc = could not find container \"02afea67ff99cccd3c74c1cb11317a9fba222f6e961cf5c74e2349eba5c26056\": container with ID starting with 02afea67ff99cccd3c74c1cb11317a9fba222f6e961cf5c74e2349eba5c26056 not found: ID does not exist" Jul 23 09:19:37.970808 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:37.970799 2576 scope.go:117] "RemoveContainer" containerID="7b426a2b8c29b42471a3c8abd3edc6803bf74965794f3fe430bd53de90f02bf8" Jul 23 09:19:37.971016 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:19:37.971000 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b426a2b8c29b42471a3c8abd3edc6803bf74965794f3fe430bd53de90f02bf8\": container with ID starting with 7b426a2b8c29b42471a3c8abd3edc6803bf74965794f3fe430bd53de90f02bf8 not found: ID does not exist" containerID="7b426a2b8c29b42471a3c8abd3edc6803bf74965794f3fe430bd53de90f02bf8" Jul 23 09:19:37.971060 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:37.971019 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b426a2b8c29b42471a3c8abd3edc6803bf74965794f3fe430bd53de90f02bf8"} err="failed to get container status \"7b426a2b8c29b42471a3c8abd3edc6803bf74965794f3fe430bd53de90f02bf8\": rpc error: code = NotFound desc = could not find container \"7b426a2b8c29b42471a3c8abd3edc6803bf74965794f3fe430bd53de90f02bf8\": container with ID starting with 7b426a2b8c29b42471a3c8abd3edc6803bf74965794f3fe430bd53de90f02bf8 not found: ID does not exist" Jul 23 09:19:37.971060 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:37.971040 2576 scope.go:117] "RemoveContainer" containerID="322ee8b69a743b8a3e29861ad366fde0f351322fb168d7ef4ec03ca153fd2174" Jul 23 09:19:37.971285 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:19:37.971270 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"322ee8b69a743b8a3e29861ad366fde0f351322fb168d7ef4ec03ca153fd2174\": container with ID starting with 322ee8b69a743b8a3e29861ad366fde0f351322fb168d7ef4ec03ca153fd2174 not found: ID does not exist" containerID="322ee8b69a743b8a3e29861ad366fde0f351322fb168d7ef4ec03ca153fd2174" Jul 23 09:19:37.971330 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:37.971289 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"322ee8b69a743b8a3e29861ad366fde0f351322fb168d7ef4ec03ca153fd2174"} err="failed to get container status \"322ee8b69a743b8a3e29861ad366fde0f351322fb168d7ef4ec03ca153fd2174\": rpc error: code = NotFound desc = could not find container \"322ee8b69a743b8a3e29861ad366fde0f351322fb168d7ef4ec03ca153fd2174\": container with ID starting with 322ee8b69a743b8a3e29861ad366fde0f351322fb168d7ef4ec03ca153fd2174 not found: ID does not exist" Jul 23 09:19:37.971330 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:37.971300 2576 scope.go:117] "RemoveContainer" containerID="f0b814f4809098312a191fde862386c052a2d25754fd03de5d2e3d235444de94" Jul 23 09:19:37.971540 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:19:37.971526 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0b814f4809098312a191fde862386c052a2d25754fd03de5d2e3d235444de94\": container with ID starting with f0b814f4809098312a191fde862386c052a2d25754fd03de5d2e3d235444de94 not found: ID does not exist" containerID="f0b814f4809098312a191fde862386c052a2d25754fd03de5d2e3d235444de94" Jul 23 09:19:37.971581 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:37.971542 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0b814f4809098312a191fde862386c052a2d25754fd03de5d2e3d235444de94"} err="failed to get container status \"f0b814f4809098312a191fde862386c052a2d25754fd03de5d2e3d235444de94\": rpc error: code = NotFound desc = could not find container \"f0b814f4809098312a191fde862386c052a2d25754fd03de5d2e3d235444de94\": container with ID starting with f0b814f4809098312a191fde862386c052a2d25754fd03de5d2e3d235444de94 not found: ID does not exist" Jul 23 09:19:37.971581 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:37.971554 2576 scope.go:117] "RemoveContainer" containerID="fd1df8ed4eff28fa6e4c147213cd1c1ed3641fd66419624baca0cfa9936d3b9b" Jul 23 09:19:37.971783 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:37.971765 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd1df8ed4eff28fa6e4c147213cd1c1ed3641fd66419624baca0cfa9936d3b9b"} err="failed to get container status \"fd1df8ed4eff28fa6e4c147213cd1c1ed3641fd66419624baca0cfa9936d3b9b\": rpc error: code = NotFound desc = could not find container \"fd1df8ed4eff28fa6e4c147213cd1c1ed3641fd66419624baca0cfa9936d3b9b\": container with ID starting with fd1df8ed4eff28fa6e4c147213cd1c1ed3641fd66419624baca0cfa9936d3b9b not found: ID does not exist" Jul 23 09:19:37.971783 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:37.971783 2576 scope.go:117] "RemoveContainer" containerID="4505e914ce83900d19afb88d4b8b6fb9e7ce9e0d985bf7b7c1a0c9a68db83b59" Jul 23 09:19:37.972010 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:37.971991 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4505e914ce83900d19afb88d4b8b6fb9e7ce9e0d985bf7b7c1a0c9a68db83b59"} err="failed to get container status \"4505e914ce83900d19afb88d4b8b6fb9e7ce9e0d985bf7b7c1a0c9a68db83b59\": rpc error: code = NotFound desc = could not find container \"4505e914ce83900d19afb88d4b8b6fb9e7ce9e0d985bf7b7c1a0c9a68db83b59\": container with ID starting with 4505e914ce83900d19afb88d4b8b6fb9e7ce9e0d985bf7b7c1a0c9a68db83b59 not found: ID does not exist" Jul 23 09:19:37.972056 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:37.972012 2576 scope.go:117] "RemoveContainer" containerID="4399f6c4bd60f1d8562862c68ff034ddacd38fa3f30a32954ce5b355bd7e4f0e" Jul 23 09:19:37.972308 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:37.972287 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4399f6c4bd60f1d8562862c68ff034ddacd38fa3f30a32954ce5b355bd7e4f0e"} err="failed to get container status \"4399f6c4bd60f1d8562862c68ff034ddacd38fa3f30a32954ce5b355bd7e4f0e\": rpc error: code = NotFound desc = could not find container \"4399f6c4bd60f1d8562862c68ff034ddacd38fa3f30a32954ce5b355bd7e4f0e\": container with ID starting with 4399f6c4bd60f1d8562862c68ff034ddacd38fa3f30a32954ce5b355bd7e4f0e not found: ID does not exist" Jul 23 09:19:37.972361 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:37.972309 2576 scope.go:117] "RemoveContainer" containerID="02afea67ff99cccd3c74c1cb11317a9fba222f6e961cf5c74e2349eba5c26056" Jul 23 09:19:37.972558 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:37.972542 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02afea67ff99cccd3c74c1cb11317a9fba222f6e961cf5c74e2349eba5c26056"} err="failed to get container status \"02afea67ff99cccd3c74c1cb11317a9fba222f6e961cf5c74e2349eba5c26056\": rpc error: code = NotFound desc = could not find container \"02afea67ff99cccd3c74c1cb11317a9fba222f6e961cf5c74e2349eba5c26056\": container with ID starting with 02afea67ff99cccd3c74c1cb11317a9fba222f6e961cf5c74e2349eba5c26056 not found: ID does not exist" Jul 23 09:19:37.972608 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:37.972558 2576 scope.go:117] "RemoveContainer" containerID="7b426a2b8c29b42471a3c8abd3edc6803bf74965794f3fe430bd53de90f02bf8" Jul 23 09:19:37.972778 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:37.972764 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b426a2b8c29b42471a3c8abd3edc6803bf74965794f3fe430bd53de90f02bf8"} err="failed to get container status \"7b426a2b8c29b42471a3c8abd3edc6803bf74965794f3fe430bd53de90f02bf8\": rpc error: code = NotFound desc = could not find container \"7b426a2b8c29b42471a3c8abd3edc6803bf74965794f3fe430bd53de90f02bf8\": container with ID starting with 7b426a2b8c29b42471a3c8abd3edc6803bf74965794f3fe430bd53de90f02bf8 not found: ID does not exist" Jul 23 09:19:37.972819 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:37.972779 2576 scope.go:117] "RemoveContainer" containerID="322ee8b69a743b8a3e29861ad366fde0f351322fb168d7ef4ec03ca153fd2174" Jul 23 09:19:37.973025 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:37.973006 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"322ee8b69a743b8a3e29861ad366fde0f351322fb168d7ef4ec03ca153fd2174"} err="failed to get container status \"322ee8b69a743b8a3e29861ad366fde0f351322fb168d7ef4ec03ca153fd2174\": rpc error: code = NotFound desc = could not find container \"322ee8b69a743b8a3e29861ad366fde0f351322fb168d7ef4ec03ca153fd2174\": container with ID starting with 322ee8b69a743b8a3e29861ad366fde0f351322fb168d7ef4ec03ca153fd2174 not found: ID does not exist" Jul 23 09:19:37.973064 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:37.973026 2576 scope.go:117] "RemoveContainer" containerID="f0b814f4809098312a191fde862386c052a2d25754fd03de5d2e3d235444de94" Jul 23 09:19:37.973267 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:37.973234 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0b814f4809098312a191fde862386c052a2d25754fd03de5d2e3d235444de94"} err="failed to get container status \"f0b814f4809098312a191fde862386c052a2d25754fd03de5d2e3d235444de94\": rpc error: code = NotFound desc = could not find container \"f0b814f4809098312a191fde862386c052a2d25754fd03de5d2e3d235444de94\": container with ID starting with f0b814f4809098312a191fde862386c052a2d25754fd03de5d2e3d235444de94 not found: ID does not exist" Jul 23 09:19:38.070950 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.070917 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d3c507db-91e8-4eb1-94bb-96dd05b53eac-secret-alertmanager-main-tls\") pod \"d3c507db-91e8-4eb1-94bb-96dd05b53eac\" (UID: \"d3c507db-91e8-4eb1-94bb-96dd05b53eac\") " Jul 23 09:19:38.070950 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.070958 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d3c507db-91e8-4eb1-94bb-96dd05b53eac-alertmanager-main-db\") pod \"d3c507db-91e8-4eb1-94bb-96dd05b53eac\" (UID: \"d3c507db-91e8-4eb1-94bb-96dd05b53eac\") " Jul 23 09:19:38.071205 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.070977 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6lpxw\" (UniqueName: \"kubernetes.io/projected/d3c507db-91e8-4eb1-94bb-96dd05b53eac-kube-api-access-6lpxw\") pod \"d3c507db-91e8-4eb1-94bb-96dd05b53eac\" (UID: \"d3c507db-91e8-4eb1-94bb-96dd05b53eac\") " Jul 23 09:19:38.071205 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.071003 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d3c507db-91e8-4eb1-94bb-96dd05b53eac-metrics-client-ca\") pod \"d3c507db-91e8-4eb1-94bb-96dd05b53eac\" (UID: \"d3c507db-91e8-4eb1-94bb-96dd05b53eac\") " Jul 23 09:19:38.071205 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.071126 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d3c507db-91e8-4eb1-94bb-96dd05b53eac-web-config\") pod \"d3c507db-91e8-4eb1-94bb-96dd05b53eac\" (UID: \"d3c507db-91e8-4eb1-94bb-96dd05b53eac\") " Jul 23 09:19:38.071205 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.071194 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d3c507db-91e8-4eb1-94bb-96dd05b53eac-secret-alertmanager-kube-rbac-proxy\") pod \"d3c507db-91e8-4eb1-94bb-96dd05b53eac\" (UID: \"d3c507db-91e8-4eb1-94bb-96dd05b53eac\") " Jul 23 09:19:38.071456 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.071232 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3c507db-91e8-4eb1-94bb-96dd05b53eac-alertmanager-trusted-ca-bundle\") pod \"d3c507db-91e8-4eb1-94bb-96dd05b53eac\" (UID: \"d3c507db-91e8-4eb1-94bb-96dd05b53eac\") " Jul 23 09:19:38.071456 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.071290 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d3c507db-91e8-4eb1-94bb-96dd05b53eac-config-volume\") pod \"d3c507db-91e8-4eb1-94bb-96dd05b53eac\" (UID: \"d3c507db-91e8-4eb1-94bb-96dd05b53eac\") " Jul 23 09:19:38.071456 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.071294 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3c507db-91e8-4eb1-94bb-96dd05b53eac-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "d3c507db-91e8-4eb1-94bb-96dd05b53eac" (UID: "d3c507db-91e8-4eb1-94bb-96dd05b53eac"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 09:19:38.071456 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.071323 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d3c507db-91e8-4eb1-94bb-96dd05b53eac-cluster-tls-config\") pod \"d3c507db-91e8-4eb1-94bb-96dd05b53eac\" (UID: \"d3c507db-91e8-4eb1-94bb-96dd05b53eac\") " Jul 23 09:19:38.071456 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.071341 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3c507db-91e8-4eb1-94bb-96dd05b53eac-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "d3c507db-91e8-4eb1-94bb-96dd05b53eac" (UID: "d3c507db-91e8-4eb1-94bb-96dd05b53eac"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 09:19:38.071456 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.071391 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d3c507db-91e8-4eb1-94bb-96dd05b53eac-secret-alertmanager-kube-rbac-proxy-web\") pod \"d3c507db-91e8-4eb1-94bb-96dd05b53eac\" (UID: \"d3c507db-91e8-4eb1-94bb-96dd05b53eac\") " Jul 23 09:19:38.071456 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.071436 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d3c507db-91e8-4eb1-94bb-96dd05b53eac-secret-alertmanager-kube-rbac-proxy-metric\") pod \"d3c507db-91e8-4eb1-94bb-96dd05b53eac\" (UID: \"d3c507db-91e8-4eb1-94bb-96dd05b53eac\") " Jul 23 09:19:38.071843 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.071487 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d3c507db-91e8-4eb1-94bb-96dd05b53eac-config-out\") pod \"d3c507db-91e8-4eb1-94bb-96dd05b53eac\" (UID: \"d3c507db-91e8-4eb1-94bb-96dd05b53eac\") " Jul 23 09:19:38.071843 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.071513 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d3c507db-91e8-4eb1-94bb-96dd05b53eac-tls-assets\") pod \"d3c507db-91e8-4eb1-94bb-96dd05b53eac\" (UID: \"d3c507db-91e8-4eb1-94bb-96dd05b53eac\") " Jul 23 09:19:38.071843 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.071697 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3c507db-91e8-4eb1-94bb-96dd05b53eac-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "d3c507db-91e8-4eb1-94bb-96dd05b53eac" (UID: "d3c507db-91e8-4eb1-94bb-96dd05b53eac"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 09:19:38.072021 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.072001 2576 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d3c507db-91e8-4eb1-94bb-96dd05b53eac-alertmanager-main-db\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:19:38.072081 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.072026 2576 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d3c507db-91e8-4eb1-94bb-96dd05b53eac-metrics-client-ca\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:19:38.072081 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.072043 2576 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3c507db-91e8-4eb1-94bb-96dd05b53eac-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:19:38.073646 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.073613 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3c507db-91e8-4eb1-94bb-96dd05b53eac-kube-api-access-6lpxw" (OuterVolumeSpecName: "kube-api-access-6lpxw") pod "d3c507db-91e8-4eb1-94bb-96dd05b53eac" (UID: "d3c507db-91e8-4eb1-94bb-96dd05b53eac"). InnerVolumeSpecName "kube-api-access-6lpxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 09:19:38.073761 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.073688 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3c507db-91e8-4eb1-94bb-96dd05b53eac-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "d3c507db-91e8-4eb1-94bb-96dd05b53eac" (UID: "d3c507db-91e8-4eb1-94bb-96dd05b53eac"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 09:19:38.074114 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.074085 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3c507db-91e8-4eb1-94bb-96dd05b53eac-config-volume" (OuterVolumeSpecName: "config-volume") pod "d3c507db-91e8-4eb1-94bb-96dd05b53eac" (UID: "d3c507db-91e8-4eb1-94bb-96dd05b53eac"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 09:19:38.074228 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.074134 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3c507db-91e8-4eb1-94bb-96dd05b53eac-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "d3c507db-91e8-4eb1-94bb-96dd05b53eac" (UID: "d3c507db-91e8-4eb1-94bb-96dd05b53eac"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 09:19:38.074761 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.074736 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3c507db-91e8-4eb1-94bb-96dd05b53eac-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "d3c507db-91e8-4eb1-94bb-96dd05b53eac" (UID: "d3c507db-91e8-4eb1-94bb-96dd05b53eac"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 09:19:38.075043 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.075024 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3c507db-91e8-4eb1-94bb-96dd05b53eac-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "d3c507db-91e8-4eb1-94bb-96dd05b53eac" (UID: "d3c507db-91e8-4eb1-94bb-96dd05b53eac"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 09:19:38.075852 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.075834 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3c507db-91e8-4eb1-94bb-96dd05b53eac-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "d3c507db-91e8-4eb1-94bb-96dd05b53eac" (UID: "d3c507db-91e8-4eb1-94bb-96dd05b53eac"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 09:19:38.076252 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.076223 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3c507db-91e8-4eb1-94bb-96dd05b53eac-config-out" (OuterVolumeSpecName: "config-out") pod "d3c507db-91e8-4eb1-94bb-96dd05b53eac" (UID: "d3c507db-91e8-4eb1-94bb-96dd05b53eac"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 09:19:38.079619 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.079588 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3c507db-91e8-4eb1-94bb-96dd05b53eac-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "d3c507db-91e8-4eb1-94bb-96dd05b53eac" (UID: "d3c507db-91e8-4eb1-94bb-96dd05b53eac"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 09:19:38.085196 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.085175 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3c507db-91e8-4eb1-94bb-96dd05b53eac-web-config" (OuterVolumeSpecName: "web-config") pod "d3c507db-91e8-4eb1-94bb-96dd05b53eac" (UID: "d3c507db-91e8-4eb1-94bb-96dd05b53eac"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 09:19:38.172694 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.172657 2576 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d3c507db-91e8-4eb1-94bb-96dd05b53eac-tls-assets\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:19:38.172694 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.172689 2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d3c507db-91e8-4eb1-94bb-96dd05b53eac-secret-alertmanager-main-tls\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:19:38.172694 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.172702 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lpxw\" (UniqueName: \"kubernetes.io/projected/d3c507db-91e8-4eb1-94bb-96dd05b53eac-kube-api-access-6lpxw\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:19:38.172899 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.172710 2576 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d3c507db-91e8-4eb1-94bb-96dd05b53eac-web-config\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:19:38.172899 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.172719 2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d3c507db-91e8-4eb1-94bb-96dd05b53eac-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:19:38.172899 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.172728 2576 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d3c507db-91e8-4eb1-94bb-96dd05b53eac-config-volume\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:19:38.172899 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.172737 2576 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d3c507db-91e8-4eb1-94bb-96dd05b53eac-cluster-tls-config\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:19:38.172899 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.172745 2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d3c507db-91e8-4eb1-94bb-96dd05b53eac-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:19:38.172899 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.172754 2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d3c507db-91e8-4eb1-94bb-96dd05b53eac-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:19:38.172899 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.172763 2576 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d3c507db-91e8-4eb1-94bb-96dd05b53eac-config-out\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:19:38.245485 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.245458 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 23 09:19:38.253849 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.253822 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 23 09:19:38.293023 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.292984 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 23 09:19:38.293621 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.293586 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3c507db-91e8-4eb1-94bb-96dd05b53eac" containerName="kube-rbac-proxy-metric" Jul 23 09:19:38.293621 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.293625 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3c507db-91e8-4eb1-94bb-96dd05b53eac" containerName="kube-rbac-proxy-metric" Jul 23 09:19:38.293784 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.293649 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cf9f282-d8c9-4bed-9ca3-2400ef455c62" containerName="console" Jul 23 09:19:38.293784 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.293658 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cf9f282-d8c9-4bed-9ca3-2400ef455c62" containerName="console" Jul 23 09:19:38.293784 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.293676 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3c507db-91e8-4eb1-94bb-96dd05b53eac" containerName="kube-rbac-proxy-web" Jul 23 09:19:38.293784 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.293686 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3c507db-91e8-4eb1-94bb-96dd05b53eac" containerName="kube-rbac-proxy-web" Jul 23 09:19:38.293784 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.293709 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3c507db-91e8-4eb1-94bb-96dd05b53eac" containerName="config-reloader" Jul 23 09:19:38.293784 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.293717 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3c507db-91e8-4eb1-94bb-96dd05b53eac" containerName="config-reloader" Jul 23 09:19:38.293784 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.293726 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3c507db-91e8-4eb1-94bb-96dd05b53eac" containerName="kube-rbac-proxy" Jul 23 09:19:38.293784 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.293734 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3c507db-91e8-4eb1-94bb-96dd05b53eac" containerName="kube-rbac-proxy" Jul 23 09:19:38.293784 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.293752 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3c507db-91e8-4eb1-94bb-96dd05b53eac" containerName="init-config-reloader" Jul 23 09:19:38.293784 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.293760 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3c507db-91e8-4eb1-94bb-96dd05b53eac" containerName="init-config-reloader" Jul 23 09:19:38.293784 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.293776 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1260ceaa-4aa5-4130-9ea7-ff8212c2d751" containerName="registry" Jul 23 09:19:38.293784 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.293784 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1260ceaa-4aa5-4130-9ea7-ff8212c2d751" containerName="registry" Jul 23 09:19:38.294203 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.293809 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3c507db-91e8-4eb1-94bb-96dd05b53eac" containerName="prom-label-proxy" Jul 23 09:19:38.294203 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.293817 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3c507db-91e8-4eb1-94bb-96dd05b53eac" containerName="prom-label-proxy" Jul 23 09:19:38.294203 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.293833 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3c507db-91e8-4eb1-94bb-96dd05b53eac" containerName="alertmanager" Jul 23 09:19:38.294203 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.293847 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3c507db-91e8-4eb1-94bb-96dd05b53eac" containerName="alertmanager" Jul 23 09:19:38.294203 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.293970 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3c507db-91e8-4eb1-94bb-96dd05b53eac" containerName="kube-rbac-proxy" Jul 23 09:19:38.294203 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.294018 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3c507db-91e8-4eb1-94bb-96dd05b53eac" containerName="kube-rbac-proxy-metric" Jul 23 09:19:38.294203 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.294175 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cf9f282-d8c9-4bed-9ca3-2400ef455c62" containerName="console" Jul 23 09:19:38.294203 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.294196 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3c507db-91e8-4eb1-94bb-96dd05b53eac" containerName="alertmanager" Jul 23 09:19:38.294458 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.294210 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1260ceaa-4aa5-4130-9ea7-ff8212c2d751" containerName="registry" Jul 23 09:19:38.294458 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.294220 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3c507db-91e8-4eb1-94bb-96dd05b53eac" containerName="config-reloader" Jul 23 09:19:38.294458 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.294434 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3c507db-91e8-4eb1-94bb-96dd05b53eac" containerName="prom-label-proxy" Jul 23 09:19:38.294458 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.294443 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3c507db-91e8-4eb1-94bb-96dd05b53eac" containerName="kube-rbac-proxy-web" Jul 23 09:19:38.297890 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.297873 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:19:38.300896 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.300878 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 23 09:19:38.301510 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.301494 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 23 09:19:38.301792 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.301779 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 23 09:19:38.302537 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.302524 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 23 09:19:38.303311 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.303295 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 23 09:19:38.304126 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.304101 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 23 09:19:38.304540 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.304521 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 23 09:19:38.304684 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.304665 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-nh6sj\"" Jul 23 09:19:38.304933 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.304920 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 23 09:19:38.315558 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.315531 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 23 09:19:38.318999 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.318975 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 23 09:19:38.476212 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.476129 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtznp\" (UniqueName: \"kubernetes.io/projected/89311462-b7ab-4bc4-bd39-49da1d67d18d-kube-api-access-rtznp\") pod \"alertmanager-main-0\" (UID: \"89311462-b7ab-4bc4-bd39-49da1d67d18d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:19:38.476212 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.476171 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/89311462-b7ab-4bc4-bd39-49da1d67d18d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"89311462-b7ab-4bc4-bd39-49da1d67d18d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:19:38.476212 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.476192 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/89311462-b7ab-4bc4-bd39-49da1d67d18d-web-config\") pod \"alertmanager-main-0\" (UID: \"89311462-b7ab-4bc4-bd39-49da1d67d18d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:19:38.476212 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.476210 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/89311462-b7ab-4bc4-bd39-49da1d67d18d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"89311462-b7ab-4bc4-bd39-49da1d67d18d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:19:38.476551 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.476286 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/89311462-b7ab-4bc4-bd39-49da1d67d18d-config-out\") pod \"alertmanager-main-0\" (UID: \"89311462-b7ab-4bc4-bd39-49da1d67d18d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:19:38.476551 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.476307 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/89311462-b7ab-4bc4-bd39-49da1d67d18d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"89311462-b7ab-4bc4-bd39-49da1d67d18d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:19:38.476551 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.476326 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/89311462-b7ab-4bc4-bd39-49da1d67d18d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"89311462-b7ab-4bc4-bd39-49da1d67d18d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:19:38.476551 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.476354 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/89311462-b7ab-4bc4-bd39-49da1d67d18d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"89311462-b7ab-4bc4-bd39-49da1d67d18d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:19:38.476551 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.476431 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/89311462-b7ab-4bc4-bd39-49da1d67d18d-config-volume\") pod \"alertmanager-main-0\" (UID: \"89311462-b7ab-4bc4-bd39-49da1d67d18d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:19:38.476551 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.476482 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/89311462-b7ab-4bc4-bd39-49da1d67d18d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"89311462-b7ab-4bc4-bd39-49da1d67d18d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:19:38.476551 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.476518 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/89311462-b7ab-4bc4-bd39-49da1d67d18d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"89311462-b7ab-4bc4-bd39-49da1d67d18d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:19:38.476551 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.476548 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89311462-b7ab-4bc4-bd39-49da1d67d18d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"89311462-b7ab-4bc4-bd39-49da1d67d18d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:19:38.476820 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.476576 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/89311462-b7ab-4bc4-bd39-49da1d67d18d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"89311462-b7ab-4bc4-bd39-49da1d67d18d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:19:38.577881 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.577845 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/89311462-b7ab-4bc4-bd39-49da1d67d18d-config-out\") pod \"alertmanager-main-0\" (UID: \"89311462-b7ab-4bc4-bd39-49da1d67d18d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:19:38.577881 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.577887 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/89311462-b7ab-4bc4-bd39-49da1d67d18d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"89311462-b7ab-4bc4-bd39-49da1d67d18d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:19:38.578086 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.577915 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/89311462-b7ab-4bc4-bd39-49da1d67d18d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"89311462-b7ab-4bc4-bd39-49da1d67d18d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:19:38.578086 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.577955 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/89311462-b7ab-4bc4-bd39-49da1d67d18d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"89311462-b7ab-4bc4-bd39-49da1d67d18d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:19:38.578086 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.577984 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/89311462-b7ab-4bc4-bd39-49da1d67d18d-config-volume\") pod \"alertmanager-main-0\" (UID: \"89311462-b7ab-4bc4-bd39-49da1d67d18d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:19:38.578086 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.578011 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/89311462-b7ab-4bc4-bd39-49da1d67d18d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"89311462-b7ab-4bc4-bd39-49da1d67d18d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:19:38.578086 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.578038 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/89311462-b7ab-4bc4-bd39-49da1d67d18d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"89311462-b7ab-4bc4-bd39-49da1d67d18d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:19:38.578086 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.578061 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89311462-b7ab-4bc4-bd39-49da1d67d18d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"89311462-b7ab-4bc4-bd39-49da1d67d18d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:19:38.578086 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.578086 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/89311462-b7ab-4bc4-bd39-49da1d67d18d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"89311462-b7ab-4bc4-bd39-49da1d67d18d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:19:38.578479 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.578116 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtznp\" (UniqueName: \"kubernetes.io/projected/89311462-b7ab-4bc4-bd39-49da1d67d18d-kube-api-access-rtznp\") pod \"alertmanager-main-0\" (UID: \"89311462-b7ab-4bc4-bd39-49da1d67d18d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:19:38.578479 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.578149 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/89311462-b7ab-4bc4-bd39-49da1d67d18d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"89311462-b7ab-4bc4-bd39-49da1d67d18d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:19:38.578479 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.578172 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/89311462-b7ab-4bc4-bd39-49da1d67d18d-web-config\") pod \"alertmanager-main-0\" (UID: \"89311462-b7ab-4bc4-bd39-49da1d67d18d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:19:38.578479 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.578199 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/89311462-b7ab-4bc4-bd39-49da1d67d18d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"89311462-b7ab-4bc4-bd39-49da1d67d18d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:19:38.578479 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.578421 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/89311462-b7ab-4bc4-bd39-49da1d67d18d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"89311462-b7ab-4bc4-bd39-49da1d67d18d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:19:38.579143 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.578826 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/89311462-b7ab-4bc4-bd39-49da1d67d18d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"89311462-b7ab-4bc4-bd39-49da1d67d18d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:19:38.580827 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.580800 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/89311462-b7ab-4bc4-bd39-49da1d67d18d-config-out\") pod \"alertmanager-main-0\" (UID: \"89311462-b7ab-4bc4-bd39-49da1d67d18d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:19:38.580988 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.580967 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/89311462-b7ab-4bc4-bd39-49da1d67d18d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"89311462-b7ab-4bc4-bd39-49da1d67d18d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:19:38.581065 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.581016 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/89311462-b7ab-4bc4-bd39-49da1d67d18d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"89311462-b7ab-4bc4-bd39-49da1d67d18d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:19:38.581162 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.581137 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/89311462-b7ab-4bc4-bd39-49da1d67d18d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"89311462-b7ab-4bc4-bd39-49da1d67d18d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:19:38.581222 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.581191 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/89311462-b7ab-4bc4-bd39-49da1d67d18d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"89311462-b7ab-4bc4-bd39-49da1d67d18d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:19:38.581305 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.581222 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/89311462-b7ab-4bc4-bd39-49da1d67d18d-config-volume\") pod \"alertmanager-main-0\" (UID: \"89311462-b7ab-4bc4-bd39-49da1d67d18d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:19:38.581364 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.581319 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/89311462-b7ab-4bc4-bd39-49da1d67d18d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"89311462-b7ab-4bc4-bd39-49da1d67d18d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:19:38.581639 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.581617 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89311462-b7ab-4bc4-bd39-49da1d67d18d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"89311462-b7ab-4bc4-bd39-49da1d67d18d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:19:38.581842 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.581817 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/89311462-b7ab-4bc4-bd39-49da1d67d18d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"89311462-b7ab-4bc4-bd39-49da1d67d18d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:19:38.582801 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.582784 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/89311462-b7ab-4bc4-bd39-49da1d67d18d-web-config\") pod \"alertmanager-main-0\" (UID: \"89311462-b7ab-4bc4-bd39-49da1d67d18d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:19:38.591227 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.591202 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtznp\" (UniqueName: \"kubernetes.io/projected/89311462-b7ab-4bc4-bd39-49da1d67d18d-kube-api-access-rtznp\") pod \"alertmanager-main-0\" (UID: \"89311462-b7ab-4bc4-bd39-49da1d67d18d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:19:38.607142 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.607121 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 23 09:19:38.759512 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.759436 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 23 09:19:38.765935 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:19:38.765905 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89311462_b7ab_4bc4_bd39_49da1d67d18d.slice/crio-b4185fc1904d17f9af3deb8c4d3ca04262a72a8b6817d400f2791e89e3439a77 WatchSource:0}: Error finding container b4185fc1904d17f9af3deb8c4d3ca04262a72a8b6817d400f2791e89e3439a77: Status 404 returned error can't find the container with id b4185fc1904d17f9af3deb8c4d3ca04262a72a8b6817d400f2791e89e3439a77 Jul 23 09:19:38.924155 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.924119 2576 generic.go:356] "Generic (PLEG): container finished" podID="89311462-b7ab-4bc4-bd39-49da1d67d18d" containerID="8f8a3d1c3ab147d509b830d39cf6aae2e281d5e9e4a46ad07dd7d594673daefc" exitCode=0 Jul 23 09:19:38.924483 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.924181 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"89311462-b7ab-4bc4-bd39-49da1d67d18d","Type":"ContainerDied","Data":"8f8a3d1c3ab147d509b830d39cf6aae2e281d5e9e4a46ad07dd7d594673daefc"} Jul 23 09:19:38.924483 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.924275 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"89311462-b7ab-4bc4-bd39-49da1d67d18d","Type":"ContainerStarted","Data":"b4185fc1904d17f9af3deb8c4d3ca04262a72a8b6817d400f2791e89e3439a77"} Jul 23 09:19:38.986080 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:38.986050 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3c507db-91e8-4eb1-94bb-96dd05b53eac" path="/var/lib/kubelet/pods/d3c507db-91e8-4eb1-94bb-96dd05b53eac/volumes" Jul 23 09:19:39.445599 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:39.445564 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7457b69c65-wfpk6"] Jul 23 09:19:39.447901 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:39.447884 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7457b69c65-wfpk6" Jul 23 09:19:39.481459 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:39.481430 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7457b69c65-wfpk6"] Jul 23 09:19:39.528144 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:39.528110 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:19:39.589119 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:39.589089 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28242908-b96c-4c04-b469-f36acdf6e5f3-trusted-ca-bundle\") pod \"console-7457b69c65-wfpk6\" (UID: \"28242908-b96c-4c04-b469-f36acdf6e5f3\") " pod="openshift-console/console-7457b69c65-wfpk6" Jul 23 09:19:39.589288 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:39.589124 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/28242908-b96c-4c04-b469-f36acdf6e5f3-console-oauth-config\") pod \"console-7457b69c65-wfpk6\" (UID: \"28242908-b96c-4c04-b469-f36acdf6e5f3\") " pod="openshift-console/console-7457b69c65-wfpk6" Jul 23 09:19:39.589288 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:39.589146 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/28242908-b96c-4c04-b469-f36acdf6e5f3-oauth-serving-cert\") pod \"console-7457b69c65-wfpk6\" (UID: \"28242908-b96c-4c04-b469-f36acdf6e5f3\") " pod="openshift-console/console-7457b69c65-wfpk6" Jul 23 09:19:39.589288 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:39.589170 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgj8k\" (UniqueName: \"kubernetes.io/projected/28242908-b96c-4c04-b469-f36acdf6e5f3-kube-api-access-vgj8k\") pod \"console-7457b69c65-wfpk6\" (UID: \"28242908-b96c-4c04-b469-f36acdf6e5f3\") " pod="openshift-console/console-7457b69c65-wfpk6" Jul 23 09:19:39.589288 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:39.589209 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/28242908-b96c-4c04-b469-f36acdf6e5f3-console-config\") pod \"console-7457b69c65-wfpk6\" (UID: \"28242908-b96c-4c04-b469-f36acdf6e5f3\") " pod="openshift-console/console-7457b69c65-wfpk6" Jul 23 09:19:39.589427 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:39.589289 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/28242908-b96c-4c04-b469-f36acdf6e5f3-console-serving-cert\") pod \"console-7457b69c65-wfpk6\" (UID: \"28242908-b96c-4c04-b469-f36acdf6e5f3\") " pod="openshift-console/console-7457b69c65-wfpk6" Jul 23 09:19:39.589427 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:39.589343 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/28242908-b96c-4c04-b469-f36acdf6e5f3-service-ca\") pod \"console-7457b69c65-wfpk6\" (UID: \"28242908-b96c-4c04-b469-f36acdf6e5f3\") " pod="openshift-console/console-7457b69c65-wfpk6" Jul 23 09:19:39.670809 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:39.670780 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:19:39.695286 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:39.695254 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/28242908-b96c-4c04-b469-f36acdf6e5f3-console-oauth-config\") pod \"console-7457b69c65-wfpk6\" (UID: \"28242908-b96c-4c04-b469-f36acdf6e5f3\") " pod="openshift-console/console-7457b69c65-wfpk6" Jul 23 09:19:39.695427 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:39.695306 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/28242908-b96c-4c04-b469-f36acdf6e5f3-oauth-serving-cert\") pod \"console-7457b69c65-wfpk6\" (UID: \"28242908-b96c-4c04-b469-f36acdf6e5f3\") " pod="openshift-console/console-7457b69c65-wfpk6" Jul 23 09:19:39.695427 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:39.695333 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgj8k\" (UniqueName: \"kubernetes.io/projected/28242908-b96c-4c04-b469-f36acdf6e5f3-kube-api-access-vgj8k\") pod \"console-7457b69c65-wfpk6\" (UID: \"28242908-b96c-4c04-b469-f36acdf6e5f3\") " pod="openshift-console/console-7457b69c65-wfpk6" Jul 23 09:19:39.695427 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:39.695364 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/28242908-b96c-4c04-b469-f36acdf6e5f3-console-config\") pod \"console-7457b69c65-wfpk6\" (UID: \"28242908-b96c-4c04-b469-f36acdf6e5f3\") " pod="openshift-console/console-7457b69c65-wfpk6" Jul 23 09:19:39.695587 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:39.695448 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/28242908-b96c-4c04-b469-f36acdf6e5f3-console-serving-cert\") pod \"console-7457b69c65-wfpk6\" (UID: \"28242908-b96c-4c04-b469-f36acdf6e5f3\") " pod="openshift-console/console-7457b69c65-wfpk6" Jul 23 09:19:39.695587 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:39.695484 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/28242908-b96c-4c04-b469-f36acdf6e5f3-service-ca\") pod \"console-7457b69c65-wfpk6\" (UID: \"28242908-b96c-4c04-b469-f36acdf6e5f3\") " pod="openshift-console/console-7457b69c65-wfpk6" Jul 23 09:19:39.695587 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:39.695538 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28242908-b96c-4c04-b469-f36acdf6e5f3-trusted-ca-bundle\") pod \"console-7457b69c65-wfpk6\" (UID: \"28242908-b96c-4c04-b469-f36acdf6e5f3\") " pod="openshift-console/console-7457b69c65-wfpk6" Jul 23 09:19:39.696371 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:39.696235 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/28242908-b96c-4c04-b469-f36acdf6e5f3-oauth-serving-cert\") pod \"console-7457b69c65-wfpk6\" (UID: \"28242908-b96c-4c04-b469-f36acdf6e5f3\") " pod="openshift-console/console-7457b69c65-wfpk6" Jul 23 09:19:39.696371 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:39.696324 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/28242908-b96c-4c04-b469-f36acdf6e5f3-service-ca\") pod \"console-7457b69c65-wfpk6\" (UID: \"28242908-b96c-4c04-b469-f36acdf6e5f3\") " pod="openshift-console/console-7457b69c65-wfpk6" Jul 23 09:19:39.696561 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:39.696536 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28242908-b96c-4c04-b469-f36acdf6e5f3-trusted-ca-bundle\") pod \"console-7457b69c65-wfpk6\" (UID: \"28242908-b96c-4c04-b469-f36acdf6e5f3\") " pod="openshift-console/console-7457b69c65-wfpk6" Jul 23 09:19:39.696806 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:39.696782 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/28242908-b96c-4c04-b469-f36acdf6e5f3-console-config\") pod \"console-7457b69c65-wfpk6\" (UID: \"28242908-b96c-4c04-b469-f36acdf6e5f3\") " pod="openshift-console/console-7457b69c65-wfpk6" Jul 23 09:19:39.698060 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:39.698034 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/28242908-b96c-4c04-b469-f36acdf6e5f3-console-oauth-config\") pod \"console-7457b69c65-wfpk6\" (UID: \"28242908-b96c-4c04-b469-f36acdf6e5f3\") " pod="openshift-console/console-7457b69c65-wfpk6" Jul 23 09:19:39.698289 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:39.698267 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/28242908-b96c-4c04-b469-f36acdf6e5f3-console-serving-cert\") pod \"console-7457b69c65-wfpk6\" (UID: \"28242908-b96c-4c04-b469-f36acdf6e5f3\") " pod="openshift-console/console-7457b69c65-wfpk6" Jul 23 09:19:39.709438 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:39.709419 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgj8k\" (UniqueName: \"kubernetes.io/projected/28242908-b96c-4c04-b469-f36acdf6e5f3-kube-api-access-vgj8k\") pod \"console-7457b69c65-wfpk6\" (UID: \"28242908-b96c-4c04-b469-f36acdf6e5f3\") " pod="openshift-console/console-7457b69c65-wfpk6" Jul 23 09:19:39.757147 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:39.757120 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7457b69c65-wfpk6" Jul 23 09:19:39.891857 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:39.891835 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7457b69c65-wfpk6"] Jul 23 09:19:39.894011 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:19:39.893983 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28242908_b96c_4c04_b469_f36acdf6e5f3.slice/crio-7819cb4af7d7737372d66eb30b3aa551833f8ee5063616b5699c7bf3f6ba31c6 WatchSource:0}: Error finding container 7819cb4af7d7737372d66eb30b3aa551833f8ee5063616b5699c7bf3f6ba31c6: Status 404 returned error can't find the container with id 7819cb4af7d7737372d66eb30b3aa551833f8ee5063616b5699c7bf3f6ba31c6 Jul 23 09:19:39.930744 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:39.930716 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7457b69c65-wfpk6" event={"ID":"28242908-b96c-4c04-b469-f36acdf6e5f3","Type":"ContainerStarted","Data":"7819cb4af7d7737372d66eb30b3aa551833f8ee5063616b5699c7bf3f6ba31c6"} Jul 23 09:19:39.933633 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:39.933609 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"89311462-b7ab-4bc4-bd39-49da1d67d18d","Type":"ContainerStarted","Data":"2e9549018b479a8d0d09cd343024ca75563c2c257559fa4072c80f197546faf2"} Jul 23 09:19:39.933729 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:39.933683 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"89311462-b7ab-4bc4-bd39-49da1d67d18d","Type":"ContainerStarted","Data":"28f7648ada56498521d1b0a0f8f783053d42cca22e3ae0602cd87f664322541a"} Jul 23 09:19:39.933729 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:39.933722 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"89311462-b7ab-4bc4-bd39-49da1d67d18d","Type":"ContainerStarted","Data":"3bf13c20c8c4fa4a9797ab1df537c68a508c7c6148af481b79006a6f04ada681"} Jul 23 09:19:39.933795 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:39.933770 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"89311462-b7ab-4bc4-bd39-49da1d67d18d","Type":"ContainerStarted","Data":"e915d36f20ef91d3dd22d6a47edacccd22d8c2e12116b3757c1d068d9e18d4ec"} Jul 23 09:19:39.933827 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:39.933807 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"89311462-b7ab-4bc4-bd39-49da1d67d18d","Type":"ContainerStarted","Data":"bc4b705730a28905ae27b4cfa40a11bf2842a3c2c55076f63711b1f537eb270b"} Jul 23 09:19:39.933859 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:39.933843 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"89311462-b7ab-4bc4-bd39-49da1d67d18d","Type":"ContainerStarted","Data":"3e1fc595031dd74cd31aadc6df1e0d2a289fd22059ed31e9568f2d8c3ab3c589"} Jul 23 09:19:39.972653 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:39.972602 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=1.956723651 podStartE2EDuration="1.972581535s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.857884ms" isStatefulPod=false podCreationTimestamp="2026-07-23 09:19:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:19:39.970525927 +0000 UTC m=+168.509804504" watchObservedRunningTime="2026-07-23 09:19:39.972581535 +0000 UTC m=+168.511860115" Jul 23 09:19:40.940933 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:40.940896 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7457b69c65-wfpk6" event={"ID":"28242908-b96c-4c04-b469-f36acdf6e5f3","Type":"ContainerStarted","Data":"af20805fdca3a6ab696e794382fedadc79e8d4288c2283c3b28034b5e897e96e"} Jul 23 09:19:40.966975 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:40.966926 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7457b69c65-wfpk6" podStartSLOduration=1.966910644 podStartE2EDuration="1.966910644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:19:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:19:40.964708416 +0000 UTC m=+169.503986993" watchObservedRunningTime="2026-07-23 09:19:40.966910644 +0000 UTC m=+169.506189221" Jul 23 09:19:44.130511 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:44.130437 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:19:44.257864 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:44.257831 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:19:49.758078 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:49.758046 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7457b69c65-wfpk6" Jul 23 09:19:49.758476 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:49.758097 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7457b69c65-wfpk6" Jul 23 09:19:49.762894 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:49.762871 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7457b69c65-wfpk6" Jul 23 09:19:49.982486 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:49.982457 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7457b69c65-wfpk6" Jul 23 09:19:50.028361 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:50.028271 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-595f97f64c-5zqx9"] Jul 23 09:19:50.136454 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:50.136425 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:19:50.198299 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:50.198268 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:19:50.257156 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:50.257131 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:19:55.723972 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:55.723933 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:19:55.811439 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:19:55.811412 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:20:15.056430 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:20:15.056364 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-595f97f64c-5zqx9" podUID="1cf71e65-e876-45f1-b242-81a6b329dad8" containerName="console" containerID="cri-o://563e252c11ef79f1d2ba1bf72289a46ed120791566d194d9da8f5ad1a51ef4bd" gracePeriod=15 Jul 23 09:20:15.286899 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:20:15.286869 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-595f97f64c-5zqx9" Jul 23 09:20:15.385485 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:20:15.385453 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1cf71e65-e876-45f1-b242-81a6b329dad8-oauth-serving-cert\") pod \"1cf71e65-e876-45f1-b242-81a6b329dad8\" (UID: \"1cf71e65-e876-45f1-b242-81a6b329dad8\") " Jul 23 09:20:15.385652 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:20:15.385508 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1cf71e65-e876-45f1-b242-81a6b329dad8-console-config\") pod \"1cf71e65-e876-45f1-b242-81a6b329dad8\" (UID: \"1cf71e65-e876-45f1-b242-81a6b329dad8\") " Jul 23 09:20:15.385652 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:20:15.385569 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1cf71e65-e876-45f1-b242-81a6b329dad8-console-serving-cert\") pod \"1cf71e65-e876-45f1-b242-81a6b329dad8\" (UID: \"1cf71e65-e876-45f1-b242-81a6b329dad8\") " Jul 23 09:20:15.385652 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:20:15.385624 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6x26q\" (UniqueName: \"kubernetes.io/projected/1cf71e65-e876-45f1-b242-81a6b329dad8-kube-api-access-6x26q\") pod \"1cf71e65-e876-45f1-b242-81a6b329dad8\" (UID: \"1cf71e65-e876-45f1-b242-81a6b329dad8\") " Jul 23 09:20:15.385819 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:20:15.385663 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1cf71e65-e876-45f1-b242-81a6b329dad8-console-oauth-config\") pod \"1cf71e65-e876-45f1-b242-81a6b329dad8\" (UID: \"1cf71e65-e876-45f1-b242-81a6b329dad8\") " Jul 23 09:20:15.385819 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:20:15.385755 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1cf71e65-e876-45f1-b242-81a6b329dad8-service-ca\") pod \"1cf71e65-e876-45f1-b242-81a6b329dad8\" (UID: \"1cf71e65-e876-45f1-b242-81a6b329dad8\") " Jul 23 09:20:15.385819 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:20:15.385795 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1cf71e65-e876-45f1-b242-81a6b329dad8-trusted-ca-bundle\") pod \"1cf71e65-e876-45f1-b242-81a6b329dad8\" (UID: \"1cf71e65-e876-45f1-b242-81a6b329dad8\") " Jul 23 09:20:15.385976 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:20:15.385878 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cf71e65-e876-45f1-b242-81a6b329dad8-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "1cf71e65-e876-45f1-b242-81a6b329dad8" (UID: "1cf71e65-e876-45f1-b242-81a6b329dad8"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 09:20:15.385976 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:20:15.385942 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cf71e65-e876-45f1-b242-81a6b329dad8-console-config" (OuterVolumeSpecName: "console-config") pod "1cf71e65-e876-45f1-b242-81a6b329dad8" (UID: "1cf71e65-e876-45f1-b242-81a6b329dad8"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 09:20:15.386184 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:20:15.386163 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cf71e65-e876-45f1-b242-81a6b329dad8-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "1cf71e65-e876-45f1-b242-81a6b329dad8" (UID: "1cf71e65-e876-45f1-b242-81a6b329dad8"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 09:20:15.386184 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:20:15.386162 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cf71e65-e876-45f1-b242-81a6b329dad8-service-ca" (OuterVolumeSpecName: "service-ca") pod "1cf71e65-e876-45f1-b242-81a6b329dad8" (UID: "1cf71e65-e876-45f1-b242-81a6b329dad8"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 09:20:15.386334 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:20:15.386182 2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1cf71e65-e876-45f1-b242-81a6b329dad8-oauth-serving-cert\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:20:15.386334 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:20:15.386203 2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1cf71e65-e876-45f1-b242-81a6b329dad8-console-config\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:20:15.387842 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:20:15.387817 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cf71e65-e876-45f1-b242-81a6b329dad8-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "1cf71e65-e876-45f1-b242-81a6b329dad8" (UID: "1cf71e65-e876-45f1-b242-81a6b329dad8"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 09:20:15.387942 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:20:15.387854 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cf71e65-e876-45f1-b242-81a6b329dad8-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "1cf71e65-e876-45f1-b242-81a6b329dad8" (UID: "1cf71e65-e876-45f1-b242-81a6b329dad8"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 09:20:15.387942 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:20:15.387876 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cf71e65-e876-45f1-b242-81a6b329dad8-kube-api-access-6x26q" (OuterVolumeSpecName: "kube-api-access-6x26q") pod "1cf71e65-e876-45f1-b242-81a6b329dad8" (UID: "1cf71e65-e876-45f1-b242-81a6b329dad8"). InnerVolumeSpecName "kube-api-access-6x26q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 09:20:15.487503 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:20:15.487478 2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1cf71e65-e876-45f1-b242-81a6b329dad8-console-serving-cert\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:20:15.487503 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:20:15.487501 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6x26q\" (UniqueName: \"kubernetes.io/projected/1cf71e65-e876-45f1-b242-81a6b329dad8-kube-api-access-6x26q\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:20:15.487655 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:20:15.487511 2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1cf71e65-e876-45f1-b242-81a6b329dad8-console-oauth-config\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:20:15.487655 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:20:15.487520 2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1cf71e65-e876-45f1-b242-81a6b329dad8-service-ca\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:20:15.487655 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:20:15.487529 2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1cf71e65-e876-45f1-b242-81a6b329dad8-trusted-ca-bundle\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:20:16.083639 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:20:16.083595 2576 generic.go:356] "Generic (PLEG): container finished" podID="1cf71e65-e876-45f1-b242-81a6b329dad8" containerID="563e252c11ef79f1d2ba1bf72289a46ed120791566d194d9da8f5ad1a51ef4bd" exitCode=0 Jul 23 09:20:16.084050 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:20:16.083645 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-595f97f64c-5zqx9" Jul 23 09:20:16.084050 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:20:16.083658 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-595f97f64c-5zqx9" event={"ID":"1cf71e65-e876-45f1-b242-81a6b329dad8","Type":"ContainerDied","Data":"563e252c11ef79f1d2ba1bf72289a46ed120791566d194d9da8f5ad1a51ef4bd"} Jul 23 09:20:16.084050 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:20:16.083702 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-595f97f64c-5zqx9" event={"ID":"1cf71e65-e876-45f1-b242-81a6b329dad8","Type":"ContainerDied","Data":"166fa64e652e99705c1ca97482451217097087f72d826a9c24ef622395e00659"} Jul 23 09:20:16.084050 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:20:16.083723 2576 scope.go:117] "RemoveContainer" containerID="563e252c11ef79f1d2ba1bf72289a46ed120791566d194d9da8f5ad1a51ef4bd" Jul 23 09:20:16.092167 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:20:16.092151 2576 scope.go:117] "RemoveContainer" containerID="563e252c11ef79f1d2ba1bf72289a46ed120791566d194d9da8f5ad1a51ef4bd" Jul 23 09:20:16.092473 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:20:16.092454 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"563e252c11ef79f1d2ba1bf72289a46ed120791566d194d9da8f5ad1a51ef4bd\": container with ID starting with 563e252c11ef79f1d2ba1bf72289a46ed120791566d194d9da8f5ad1a51ef4bd not found: ID does not exist" containerID="563e252c11ef79f1d2ba1bf72289a46ed120791566d194d9da8f5ad1a51ef4bd" Jul 23 09:20:16.092527 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:20:16.092481 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"563e252c11ef79f1d2ba1bf72289a46ed120791566d194d9da8f5ad1a51ef4bd"} err="failed to get container status \"563e252c11ef79f1d2ba1bf72289a46ed120791566d194d9da8f5ad1a51ef4bd\": rpc error: code = NotFound desc = could not find container \"563e252c11ef79f1d2ba1bf72289a46ed120791566d194d9da8f5ad1a51ef4bd\": container with ID starting with 563e252c11ef79f1d2ba1bf72289a46ed120791566d194d9da8f5ad1a51ef4bd not found: ID does not exist" Jul 23 09:20:16.105129 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:20:16.105103 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-595f97f64c-5zqx9"] Jul 23 09:20:16.109937 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:20:16.109918 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-595f97f64c-5zqx9"] Jul 23 09:20:16.984574 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:20:16.984541 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1cf71e65-e876-45f1-b242-81a6b329dad8" path="/var/lib/kubelet/pods/1cf71e65-e876-45f1-b242-81a6b329dad8/volumes" Jul 23 09:20:21.217351 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:20:21.217322 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:20:23.814373 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:20:23.814342 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:20:23.873015 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:20:23.872990 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:20:43.810866 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:20:43.810831 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:20:43.910350 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:20:43.910318 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:20:44.419468 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:20:44.419437 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z4g7z"] Jul 23 09:20:44.419780 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:20:44.419768 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cf71e65-e876-45f1-b242-81a6b329dad8" containerName="console" Jul 23 09:20:44.419823 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:20:44.419782 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cf71e65-e876-45f1-b242-81a6b329dad8" containerName="console" Jul 23 09:20:44.419861 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:20:44.419834 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cf71e65-e876-45f1-b242-81a6b329dad8" containerName="console" Jul 23 09:20:44.422105 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:20:44.422086 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z4g7z" Jul 23 09:20:44.424516 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:20:44.424487 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 23 09:20:44.424611 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:20:44.424527 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-j6qwj\"" Jul 23 09:20:44.425060 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:20:44.425046 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 23 09:20:44.431415 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:20:44.431395 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z4g7z"] Jul 23 09:20:44.527371 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:20:44.527331 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fbbee13b-9af2-42ea-98a1-bd907d9c6585-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z4g7z\" (UID: \"fbbee13b-9af2-42ea-98a1-bd907d9c6585\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z4g7z" Jul 23 09:20:44.527547 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:20:44.527402 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fbbee13b-9af2-42ea-98a1-bd907d9c6585-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z4g7z\" (UID: \"fbbee13b-9af2-42ea-98a1-bd907d9c6585\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z4g7z" Jul 23 09:20:44.527547 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:20:44.527468 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnt5l\" (UniqueName: \"kubernetes.io/projected/fbbee13b-9af2-42ea-98a1-bd907d9c6585-kube-api-access-mnt5l\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z4g7z\" (UID: \"fbbee13b-9af2-42ea-98a1-bd907d9c6585\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z4g7z" Jul 23 09:20:44.628645 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:20:44.628610 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fbbee13b-9af2-42ea-98a1-bd907d9c6585-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z4g7z\" (UID: \"fbbee13b-9af2-42ea-98a1-bd907d9c6585\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z4g7z" Jul 23 09:20:44.628797 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:20:44.628672 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnt5l\" (UniqueName: \"kubernetes.io/projected/fbbee13b-9af2-42ea-98a1-bd907d9c6585-kube-api-access-mnt5l\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z4g7z\" (UID: \"fbbee13b-9af2-42ea-98a1-bd907d9c6585\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z4g7z" Jul 23 09:20:44.628797 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:20:44.628730 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fbbee13b-9af2-42ea-98a1-bd907d9c6585-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z4g7z\" (UID: \"fbbee13b-9af2-42ea-98a1-bd907d9c6585\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z4g7z" Jul 23 09:20:44.629099 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:20:44.629078 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fbbee13b-9af2-42ea-98a1-bd907d9c6585-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z4g7z\" (UID: \"fbbee13b-9af2-42ea-98a1-bd907d9c6585\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z4g7z" Jul 23 09:20:44.629099 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:20:44.629093 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fbbee13b-9af2-42ea-98a1-bd907d9c6585-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z4g7z\" (UID: \"fbbee13b-9af2-42ea-98a1-bd907d9c6585\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z4g7z" Jul 23 09:20:44.636860 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:20:44.636838 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnt5l\" (UniqueName: \"kubernetes.io/projected/fbbee13b-9af2-42ea-98a1-bd907d9c6585-kube-api-access-mnt5l\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z4g7z\" (UID: \"fbbee13b-9af2-42ea-98a1-bd907d9c6585\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z4g7z" Jul 23 09:20:44.731208 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:20:44.731118 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z4g7z" Jul 23 09:20:44.858750 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:20:44.858699 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z4g7z"] Jul 23 09:20:44.860865 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:20:44.860843 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbbee13b_9af2_42ea_98a1_bd907d9c6585.slice/crio-89d33851cb014b4c7ceb97bb1c0da6a9d37925f3175b489a1299a0827c15594c WatchSource:0}: Error finding container 89d33851cb014b4c7ceb97bb1c0da6a9d37925f3175b489a1299a0827c15594c: Status 404 returned error can't find the container with id 89d33851cb014b4c7ceb97bb1c0da6a9d37925f3175b489a1299a0827c15594c Jul 23 09:20:45.195599 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:20:45.195562 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z4g7z" event={"ID":"fbbee13b-9af2-42ea-98a1-bd907d9c6585","Type":"ContainerStarted","Data":"89d33851cb014b4c7ceb97bb1c0da6a9d37925f3175b489a1299a0827c15594c"} Jul 23 09:20:52.233070 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:20:52.233034 2576 generic.go:356] "Generic (PLEG): container finished" podID="fbbee13b-9af2-42ea-98a1-bd907d9c6585" containerID="10faee335d907af4d5dd5cba8fce6afa55daed85f4133f256d997622caa8edf9" exitCode=0 Jul 23 09:20:52.233490 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:20:52.233118 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z4g7z" event={"ID":"fbbee13b-9af2-42ea-98a1-bd907d9c6585","Type":"ContainerDied","Data":"10faee335d907af4d5dd5cba8fce6afa55daed85f4133f256d997622caa8edf9"} Jul 23 09:20:56.249991 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:20:56.249949 2576 generic.go:356] "Generic (PLEG): container finished" podID="fbbee13b-9af2-42ea-98a1-bd907d9c6585" containerID="bc9188a6f62e4bd8e30c2726116ecd595f68b0d19432bfb34e1bf3ac1bf1eb05" exitCode=0 Jul 23 09:20:56.250409 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:20:56.250044 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z4g7z" event={"ID":"fbbee13b-9af2-42ea-98a1-bd907d9c6585","Type":"ContainerDied","Data":"bc9188a6f62e4bd8e30c2726116ecd595f68b0d19432bfb34e1bf3ac1bf1eb05"} Jul 23 09:21:04.287865 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:04.287832 2576 generic.go:356] "Generic (PLEG): container finished" podID="fbbee13b-9af2-42ea-98a1-bd907d9c6585" containerID="8a45662ebef5169f3c0f11882e72cb51abdff3597ed013852b75c0330bacb8aa" exitCode=0 Jul 23 09:21:04.288281 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:04.287907 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z4g7z" event={"ID":"fbbee13b-9af2-42ea-98a1-bd907d9c6585","Type":"ContainerDied","Data":"8a45662ebef5169f3c0f11882e72cb51abdff3597ed013852b75c0330bacb8aa"} Jul 23 09:21:05.422026 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:05.421997 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z4g7z" Jul 23 09:21:05.516555 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:05.516523 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fbbee13b-9af2-42ea-98a1-bd907d9c6585-util\") pod \"fbbee13b-9af2-42ea-98a1-bd907d9c6585\" (UID: \"fbbee13b-9af2-42ea-98a1-bd907d9c6585\") " Jul 23 09:21:05.516736 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:05.516589 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fbbee13b-9af2-42ea-98a1-bd907d9c6585-bundle\") pod \"fbbee13b-9af2-42ea-98a1-bd907d9c6585\" (UID: \"fbbee13b-9af2-42ea-98a1-bd907d9c6585\") " Jul 23 09:21:05.516736 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:05.516671 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mnt5l\" (UniqueName: \"kubernetes.io/projected/fbbee13b-9af2-42ea-98a1-bd907d9c6585-kube-api-access-mnt5l\") pod \"fbbee13b-9af2-42ea-98a1-bd907d9c6585\" (UID: \"fbbee13b-9af2-42ea-98a1-bd907d9c6585\") " Jul 23 09:21:05.517296 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:05.517235 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbbee13b-9af2-42ea-98a1-bd907d9c6585-bundle" (OuterVolumeSpecName: "bundle") pod "fbbee13b-9af2-42ea-98a1-bd907d9c6585" (UID: "fbbee13b-9af2-42ea-98a1-bd907d9c6585"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 09:21:05.518834 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:05.518809 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbbee13b-9af2-42ea-98a1-bd907d9c6585-kube-api-access-mnt5l" (OuterVolumeSpecName: "kube-api-access-mnt5l") pod "fbbee13b-9af2-42ea-98a1-bd907d9c6585" (UID: "fbbee13b-9af2-42ea-98a1-bd907d9c6585"). InnerVolumeSpecName "kube-api-access-mnt5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 09:21:05.520784 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:05.520758 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbbee13b-9af2-42ea-98a1-bd907d9c6585-util" (OuterVolumeSpecName: "util") pod "fbbee13b-9af2-42ea-98a1-bd907d9c6585" (UID: "fbbee13b-9af2-42ea-98a1-bd907d9c6585"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 09:21:05.617893 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:05.617802 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnt5l\" (UniqueName: \"kubernetes.io/projected/fbbee13b-9af2-42ea-98a1-bd907d9c6585-kube-api-access-mnt5l\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:21:05.617893 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:05.617839 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fbbee13b-9af2-42ea-98a1-bd907d9c6585-util\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:21:05.617893 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:05.617852 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fbbee13b-9af2-42ea-98a1-bd907d9c6585-bundle\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:21:06.298125 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:06.298089 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z4g7z" event={"ID":"fbbee13b-9af2-42ea-98a1-bd907d9c6585","Type":"ContainerDied","Data":"89d33851cb014b4c7ceb97bb1c0da6a9d37925f3175b489a1299a0827c15594c"} Jul 23 09:21:06.298302 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:06.298134 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z4g7z" Jul 23 09:21:06.298302 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:06.298139 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89d33851cb014b4c7ceb97bb1c0da6a9d37925f3175b489a1299a0827c15594c" Jul 23 09:21:10.893640 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:10.893603 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:21:11.391903 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:11.391870 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:21:12.711721 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:12.711687 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-mvgxp"] Jul 23 09:21:12.712082 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:12.711998 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbbee13b-9af2-42ea-98a1-bd907d9c6585" containerName="pull" Jul 23 09:21:12.712082 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:12.712009 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbbee13b-9af2-42ea-98a1-bd907d9c6585" containerName="pull" Jul 23 09:21:12.712082 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:12.712021 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbbee13b-9af2-42ea-98a1-bd907d9c6585" containerName="util" Jul 23 09:21:12.712082 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:12.712030 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbbee13b-9af2-42ea-98a1-bd907d9c6585" containerName="util" Jul 23 09:21:12.712082 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:12.712055 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbbee13b-9af2-42ea-98a1-bd907d9c6585" containerName="extract" Jul 23 09:21:12.712082 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:12.712060 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbbee13b-9af2-42ea-98a1-bd907d9c6585" containerName="extract" Jul 23 09:21:12.712300 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:12.712117 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbbee13b-9af2-42ea-98a1-bd907d9c6585" containerName="extract" Jul 23 09:21:12.714940 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:12.714924 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-mvgxp" Jul 23 09:21:12.717915 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:12.717891 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-kc89m\"" Jul 23 09:21:12.718052 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:12.717891 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 23 09:21:12.719664 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:12.719647 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 23 09:21:12.741771 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:12.741746 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-mvgxp"] Jul 23 09:21:12.883430 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:12.883394 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e6cf2f53-6fe1-4581-98c4-dc9167d051b9-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-mvgxp\" (UID: \"e6cf2f53-6fe1-4581-98c4-dc9167d051b9\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-mvgxp" Jul 23 09:21:12.883617 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:12.883459 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltf45\" (UniqueName: \"kubernetes.io/projected/e6cf2f53-6fe1-4581-98c4-dc9167d051b9-kube-api-access-ltf45\") pod \"cert-manager-operator-controller-manager-55c444bf86-mvgxp\" (UID: \"e6cf2f53-6fe1-4581-98c4-dc9167d051b9\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-mvgxp" Jul 23 09:21:12.984107 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:12.984029 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltf45\" (UniqueName: \"kubernetes.io/projected/e6cf2f53-6fe1-4581-98c4-dc9167d051b9-kube-api-access-ltf45\") pod \"cert-manager-operator-controller-manager-55c444bf86-mvgxp\" (UID: \"e6cf2f53-6fe1-4581-98c4-dc9167d051b9\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-mvgxp" Jul 23 09:21:12.984107 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:12.984092 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e6cf2f53-6fe1-4581-98c4-dc9167d051b9-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-mvgxp\" (UID: \"e6cf2f53-6fe1-4581-98c4-dc9167d051b9\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-mvgxp" Jul 23 09:21:12.984431 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:12.984415 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e6cf2f53-6fe1-4581-98c4-dc9167d051b9-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-mvgxp\" (UID: \"e6cf2f53-6fe1-4581-98c4-dc9167d051b9\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-mvgxp" Jul 23 09:21:13.004199 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:13.004171 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltf45\" (UniqueName: \"kubernetes.io/projected/e6cf2f53-6fe1-4581-98c4-dc9167d051b9-kube-api-access-ltf45\") pod \"cert-manager-operator-controller-manager-55c444bf86-mvgxp\" (UID: \"e6cf2f53-6fe1-4581-98c4-dc9167d051b9\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-mvgxp" Jul 23 09:21:13.023994 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:13.023969 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-mvgxp" Jul 23 09:21:13.163030 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:13.163004 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-mvgxp"] Jul 23 09:21:13.166143 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:21:13.166112 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6cf2f53_6fe1_4581_98c4_dc9167d051b9.slice/crio-ca4f020bd389697bdb9d87415b4e7bf914d3cd954e7268c34455e5011a8dbfd5 WatchSource:0}: Error finding container ca4f020bd389697bdb9d87415b4e7bf914d3cd954e7268c34455e5011a8dbfd5: Status 404 returned error can't find the container with id ca4f020bd389697bdb9d87415b4e7bf914d3cd954e7268c34455e5011a8dbfd5 Jul 23 09:21:13.327423 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:13.327341 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-mvgxp" event={"ID":"e6cf2f53-6fe1-4581-98c4-dc9167d051b9","Type":"ContainerStarted","Data":"ca4f020bd389697bdb9d87415b4e7bf914d3cd954e7268c34455e5011a8dbfd5"} Jul 23 09:21:14.348956 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:14.348884 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:21:14.449195 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:14.449161 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:21:15.337230 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:15.337187 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-mvgxp" event={"ID":"e6cf2f53-6fe1-4581-98c4-dc9167d051b9","Type":"ContainerStarted","Data":"3adeb7b78277b5f4d3cd2ceab8f56f9c84cfee36df9e6880a0677870d0044482"} Jul 23 09:21:15.365137 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:15.365085 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-mvgxp" podStartSLOduration=1.733279853 podStartE2EDuration="3.365070096s" totalImagesPullingTime="1.631790243s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:21:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:21:15.364103233 +0000 UTC m=+263.903381811" watchObservedRunningTime="2026-07-23 09:21:15.365070096 +0000 UTC m=+263.904348673" Jul 23 09:21:15.772145 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:15.772110 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:21:15.921801 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:15.921759 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:21:16.377957 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:16.377921 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7f44z"] Jul 23 09:21:16.381550 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:16.381529 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7f44z" Jul 23 09:21:16.383848 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:16.383795 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-j6qwj\"" Jul 23 09:21:16.383967 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:16.383853 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 23 09:21:16.384641 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:16.384628 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 23 09:21:16.391094 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:16.391075 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7f44z"] Jul 23 09:21:16.413269 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:16.413230 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/26d53e53-185a-4659-98b7-22d2510d7eed-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7f44z\" (UID: \"26d53e53-185a-4659-98b7-22d2510d7eed\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7f44z" Jul 23 09:21:16.413407 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:16.413283 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6q2v\" (UniqueName: \"kubernetes.io/projected/26d53e53-185a-4659-98b7-22d2510d7eed-kube-api-access-g6q2v\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7f44z\" (UID: \"26d53e53-185a-4659-98b7-22d2510d7eed\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7f44z" Jul 23 09:21:16.413407 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:16.413349 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/26d53e53-185a-4659-98b7-22d2510d7eed-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7f44z\" (UID: \"26d53e53-185a-4659-98b7-22d2510d7eed\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7f44z" Jul 23 09:21:16.514656 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:16.514629 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/26d53e53-185a-4659-98b7-22d2510d7eed-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7f44z\" (UID: \"26d53e53-185a-4659-98b7-22d2510d7eed\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7f44z" Jul 23 09:21:16.514784 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:16.514691 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/26d53e53-185a-4659-98b7-22d2510d7eed-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7f44z\" (UID: \"26d53e53-185a-4659-98b7-22d2510d7eed\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7f44z" Jul 23 09:21:16.514784 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:16.514713 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6q2v\" (UniqueName: \"kubernetes.io/projected/26d53e53-185a-4659-98b7-22d2510d7eed-kube-api-access-g6q2v\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7f44z\" (UID: \"26d53e53-185a-4659-98b7-22d2510d7eed\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7f44z" Jul 23 09:21:16.515062 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:16.515039 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/26d53e53-185a-4659-98b7-22d2510d7eed-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7f44z\" (UID: \"26d53e53-185a-4659-98b7-22d2510d7eed\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7f44z" Jul 23 09:21:16.515228 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:16.515063 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/26d53e53-185a-4659-98b7-22d2510d7eed-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7f44z\" (UID: \"26d53e53-185a-4659-98b7-22d2510d7eed\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7f44z" Jul 23 09:21:16.530723 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:16.530696 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6q2v\" (UniqueName: \"kubernetes.io/projected/26d53e53-185a-4659-98b7-22d2510d7eed-kube-api-access-g6q2v\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7f44z\" (UID: \"26d53e53-185a-4659-98b7-22d2510d7eed\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7f44z" Jul 23 09:21:16.672174 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:16.672107 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:21:16.690569 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:16.690542 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7f44z" Jul 23 09:21:16.765325 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:16.765232 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:21:16.777947 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:16.777918 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-mx5kh"] Jul 23 09:21:16.781453 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:16.781431 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-mx5kh" Jul 23 09:21:16.783864 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:16.783843 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 23 09:21:16.783986 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:16.783843 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 23 09:21:16.793598 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:16.793576 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-mx5kh"] Jul 23 09:21:16.817761 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:16.817733 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqnbm\" (UniqueName: \"kubernetes.io/projected/24ee5dd9-022a-4d99-bf0d-2fbdd1bbc2fe-kube-api-access-dqnbm\") pod \"cert-manager-webhook-6486ffd997-mx5kh\" (UID: \"24ee5dd9-022a-4d99-bf0d-2fbdd1bbc2fe\") " pod="cert-manager/cert-manager-webhook-6486ffd997-mx5kh" Jul 23 09:21:16.817895 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:16.817836 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/24ee5dd9-022a-4d99-bf0d-2fbdd1bbc2fe-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-mx5kh\" (UID: \"24ee5dd9-022a-4d99-bf0d-2fbdd1bbc2fe\") " pod="cert-manager/cert-manager-webhook-6486ffd997-mx5kh" Jul 23 09:21:16.825721 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:16.825699 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7f44z"] Jul 23 09:21:16.827758 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:21:16.827736 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26d53e53_185a_4659_98b7_22d2510d7eed.slice/crio-f1ac0815cab19ffe6be3d300128c243ecc629408a52db78f10ca09e3dddb9fe5 WatchSource:0}: Error finding container f1ac0815cab19ffe6be3d300128c243ecc629408a52db78f10ca09e3dddb9fe5: Status 404 returned error can't find the container with id f1ac0815cab19ffe6be3d300128c243ecc629408a52db78f10ca09e3dddb9fe5 Jul 23 09:21:16.918740 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:16.918708 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqnbm\" (UniqueName: \"kubernetes.io/projected/24ee5dd9-022a-4d99-bf0d-2fbdd1bbc2fe-kube-api-access-dqnbm\") pod \"cert-manager-webhook-6486ffd997-mx5kh\" (UID: \"24ee5dd9-022a-4d99-bf0d-2fbdd1bbc2fe\") " pod="cert-manager/cert-manager-webhook-6486ffd997-mx5kh" Jul 23 09:21:16.918943 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:16.918770 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/24ee5dd9-022a-4d99-bf0d-2fbdd1bbc2fe-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-mx5kh\" (UID: \"24ee5dd9-022a-4d99-bf0d-2fbdd1bbc2fe\") " pod="cert-manager/cert-manager-webhook-6486ffd997-mx5kh" Jul 23 09:21:16.933234 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:16.933210 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqnbm\" (UniqueName: \"kubernetes.io/projected/24ee5dd9-022a-4d99-bf0d-2fbdd1bbc2fe-kube-api-access-dqnbm\") pod \"cert-manager-webhook-6486ffd997-mx5kh\" (UID: \"24ee5dd9-022a-4d99-bf0d-2fbdd1bbc2fe\") " pod="cert-manager/cert-manager-webhook-6486ffd997-mx5kh" Jul 23 09:21:16.937555 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:16.937534 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/24ee5dd9-022a-4d99-bf0d-2fbdd1bbc2fe-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-mx5kh\" (UID: \"24ee5dd9-022a-4d99-bf0d-2fbdd1bbc2fe\") " pod="cert-manager/cert-manager-webhook-6486ffd997-mx5kh" Jul 23 09:21:17.099341 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:17.099301 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-mx5kh" Jul 23 09:21:17.250684 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:17.250643 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-mx5kh"] Jul 23 09:21:17.253369 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:21:17.253342 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod24ee5dd9_022a_4d99_bf0d_2fbdd1bbc2fe.slice/crio-09cb0a3c7bae304309a4bbc163a05228d95175b466cbc406041f37195ef7017a WatchSource:0}: Error finding container 09cb0a3c7bae304309a4bbc163a05228d95175b466cbc406041f37195ef7017a: Status 404 returned error can't find the container with id 09cb0a3c7bae304309a4bbc163a05228d95175b466cbc406041f37195ef7017a Jul 23 09:21:17.347879 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:17.347846 2576 generic.go:356] "Generic (PLEG): container finished" podID="26d53e53-185a-4659-98b7-22d2510d7eed" containerID="ff908b0b6b326c69197e40d0908cd532a9296f78fab4e31a746bc55ab9b5098d" exitCode=0 Jul 23 09:21:17.348059 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:17.347891 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7f44z" event={"ID":"26d53e53-185a-4659-98b7-22d2510d7eed","Type":"ContainerDied","Data":"ff908b0b6b326c69197e40d0908cd532a9296f78fab4e31a746bc55ab9b5098d"} Jul 23 09:21:17.348059 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:17.347937 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7f44z" event={"ID":"26d53e53-185a-4659-98b7-22d2510d7eed","Type":"ContainerStarted","Data":"f1ac0815cab19ffe6be3d300128c243ecc629408a52db78f10ca09e3dddb9fe5"} Jul 23 09:21:17.349217 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:17.349195 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-mx5kh" event={"ID":"24ee5dd9-022a-4d99-bf0d-2fbdd1bbc2fe","Type":"ContainerStarted","Data":"09cb0a3c7bae304309a4bbc163a05228d95175b466cbc406041f37195ef7017a"} Jul 23 09:21:18.667742 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:18.667708 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:21:19.064008 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:19.063927 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:21:19.567790 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:19.567756 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-6ck25"] Jul 23 09:21:19.571090 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:19.571071 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-6ck25" Jul 23 09:21:19.574407 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:19.573536 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-x2tgt\"" Jul 23 09:21:19.582826 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:19.582800 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-6ck25"] Jul 23 09:21:19.642791 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:19.642758 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0544aab6-f182-423f-b785-91c268d87478-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-6ck25\" (UID: \"0544aab6-f182-423f-b785-91c268d87478\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-6ck25" Jul 23 09:21:19.642983 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:19.642827 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcds9\" (UniqueName: \"kubernetes.io/projected/0544aab6-f182-423f-b785-91c268d87478-kube-api-access-pcds9\") pod \"cert-manager-cainjector-559cdd8c87-6ck25\" (UID: \"0544aab6-f182-423f-b785-91c268d87478\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-6ck25" Jul 23 09:21:19.744173 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:19.744132 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0544aab6-f182-423f-b785-91c268d87478-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-6ck25\" (UID: \"0544aab6-f182-423f-b785-91c268d87478\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-6ck25" Jul 23 09:21:19.744629 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:19.744223 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcds9\" (UniqueName: \"kubernetes.io/projected/0544aab6-f182-423f-b785-91c268d87478-kube-api-access-pcds9\") pod \"cert-manager-cainjector-559cdd8c87-6ck25\" (UID: \"0544aab6-f182-423f-b785-91c268d87478\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-6ck25" Jul 23 09:21:19.754904 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:19.754876 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0544aab6-f182-423f-b785-91c268d87478-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-6ck25\" (UID: \"0544aab6-f182-423f-b785-91c268d87478\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-6ck25" Jul 23 09:21:19.755733 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:19.755708 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcds9\" (UniqueName: \"kubernetes.io/projected/0544aab6-f182-423f-b785-91c268d87478-kube-api-access-pcds9\") pod \"cert-manager-cainjector-559cdd8c87-6ck25\" (UID: \"0544aab6-f182-423f-b785-91c268d87478\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-6ck25" Jul 23 09:21:19.884337 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:19.884301 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-6ck25" Jul 23 09:21:20.622462 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:20.622421 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-6ck25"] Jul 23 09:21:20.624982 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:21:20.624958 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0544aab6_f182_423f_b785_91c268d87478.slice/crio-f6c8e3a906eb19b8ff369ef66bbae70af43bfa3ab252453c194295ec44127163 WatchSource:0}: Error finding container f6c8e3a906eb19b8ff369ef66bbae70af43bfa3ab252453c194295ec44127163: Status 404 returned error can't find the container with id f6c8e3a906eb19b8ff369ef66bbae70af43bfa3ab252453c194295ec44127163 Jul 23 09:21:21.133489 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:21.133459 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:21:21.303671 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:21.303644 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:21:21.371061 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:21.371029 2576 generic.go:356] "Generic (PLEG): container finished" podID="26d53e53-185a-4659-98b7-22d2510d7eed" containerID="92d0ea5ce6a2208246a802ea256fc9cc6ca967e675c7071b308afe73985e36d3" exitCode=0 Jul 23 09:21:21.371266 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:21.371124 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7f44z" event={"ID":"26d53e53-185a-4659-98b7-22d2510d7eed","Type":"ContainerDied","Data":"92d0ea5ce6a2208246a802ea256fc9cc6ca967e675c7071b308afe73985e36d3"} Jul 23 09:21:21.372706 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:21.372678 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-6ck25" event={"ID":"0544aab6-f182-423f-b785-91c268d87478","Type":"ContainerStarted","Data":"5031ed9c8747367c7b35e1f259c20a3fe719e0d72565c0ac8f5845da563d35e5"} Jul 23 09:21:21.372811 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:21.372718 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-6ck25" event={"ID":"0544aab6-f182-423f-b785-91c268d87478","Type":"ContainerStarted","Data":"f6c8e3a906eb19b8ff369ef66bbae70af43bfa3ab252453c194295ec44127163"} Jul 23 09:21:21.374361 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:21.374308 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-mx5kh" event={"ID":"24ee5dd9-022a-4d99-bf0d-2fbdd1bbc2fe","Type":"ContainerStarted","Data":"706d440dc5f15099dbf0cba642f074685335db01e0c431109e9a24f9884b98f6"} Jul 23 09:21:21.374458 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:21.374432 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-mx5kh" Jul 23 09:21:21.405742 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:21.405701 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-mx5kh" podStartSLOduration=2.169591636 podStartE2EDuration="5.405685575s" totalImagesPullingTime="3.236093939s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:21:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:21:21.403965539 +0000 UTC m=+269.943244120" watchObservedRunningTime="2026-07-23 09:21:21.405685575 +0000 UTC m=+269.944964152" Jul 23 09:21:21.421295 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:21.421255 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-6ck25" podStartSLOduration=2.421225411 podStartE2EDuration="2.421225411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:21:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:21:21.419657568 +0000 UTC m=+269.958936145" watchObservedRunningTime="2026-07-23 09:21:21.421225411 +0000 UTC m=+269.960503987" Jul 23 09:21:22.380775 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:22.380740 2576 generic.go:356] "Generic (PLEG): container finished" podID="26d53e53-185a-4659-98b7-22d2510d7eed" containerID="aa75ca9ef8115a08e10bf9cd8cbee12054018122ce5eb62bcb0add1a734bd468" exitCode=0 Jul 23 09:21:22.381148 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:22.380807 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7f44z" event={"ID":"26d53e53-185a-4659-98b7-22d2510d7eed","Type":"ContainerDied","Data":"aa75ca9ef8115a08e10bf9cd8cbee12054018122ce5eb62bcb0add1a734bd468"} Jul 23 09:21:23.512519 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:23.512488 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7f44z" Jul 23 09:21:23.575985 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:23.575951 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g6q2v\" (UniqueName: \"kubernetes.io/projected/26d53e53-185a-4659-98b7-22d2510d7eed-kube-api-access-g6q2v\") pod \"26d53e53-185a-4659-98b7-22d2510d7eed\" (UID: \"26d53e53-185a-4659-98b7-22d2510d7eed\") " Jul 23 09:21:23.575985 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:23.575987 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/26d53e53-185a-4659-98b7-22d2510d7eed-util\") pod \"26d53e53-185a-4659-98b7-22d2510d7eed\" (UID: \"26d53e53-185a-4659-98b7-22d2510d7eed\") " Jul 23 09:21:23.576178 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:23.576045 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/26d53e53-185a-4659-98b7-22d2510d7eed-bundle\") pod \"26d53e53-185a-4659-98b7-22d2510d7eed\" (UID: \"26d53e53-185a-4659-98b7-22d2510d7eed\") " Jul 23 09:21:23.576503 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:23.576470 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26d53e53-185a-4659-98b7-22d2510d7eed-bundle" (OuterVolumeSpecName: "bundle") pod "26d53e53-185a-4659-98b7-22d2510d7eed" (UID: "26d53e53-185a-4659-98b7-22d2510d7eed"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 09:21:23.578012 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:23.577990 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26d53e53-185a-4659-98b7-22d2510d7eed-kube-api-access-g6q2v" (OuterVolumeSpecName: "kube-api-access-g6q2v") pod "26d53e53-185a-4659-98b7-22d2510d7eed" (UID: "26d53e53-185a-4659-98b7-22d2510d7eed"). InnerVolumeSpecName "kube-api-access-g6q2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 09:21:23.581022 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:23.581002 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26d53e53-185a-4659-98b7-22d2510d7eed-util" (OuterVolumeSpecName: "util") pod "26d53e53-185a-4659-98b7-22d2510d7eed" (UID: "26d53e53-185a-4659-98b7-22d2510d7eed"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 09:21:23.677109 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:23.677006 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/26d53e53-185a-4659-98b7-22d2510d7eed-bundle\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:21:23.677109 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:23.677050 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6q2v\" (UniqueName: \"kubernetes.io/projected/26d53e53-185a-4659-98b7-22d2510d7eed-kube-api-access-g6q2v\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:21:23.677109 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:23.677064 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/26d53e53-185a-4659-98b7-22d2510d7eed-util\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:21:24.390945 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:24.390906 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7f44z" event={"ID":"26d53e53-185a-4659-98b7-22d2510d7eed","Type":"ContainerDied","Data":"f1ac0815cab19ffe6be3d300128c243ecc629408a52db78f10ca09e3dddb9fe5"} Jul 23 09:21:24.390945 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:24.390940 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7f44z" Jul 23 09:21:24.391151 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:24.390965 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1ac0815cab19ffe6be3d300128c243ecc629408a52db78f10ca09e3dddb9fe5" Jul 23 09:21:27.269910 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:27.269881 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:21:27.383770 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:27.383744 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-mx5kh" Jul 23 09:21:29.654192 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:29.654159 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-v5drt"] Jul 23 09:21:29.654578 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:29.654528 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26d53e53-185a-4659-98b7-22d2510d7eed" containerName="extract" Jul 23 09:21:29.654578 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:29.654540 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="26d53e53-185a-4659-98b7-22d2510d7eed" containerName="extract" Jul 23 09:21:29.654578 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:29.654558 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26d53e53-185a-4659-98b7-22d2510d7eed" containerName="util" Jul 23 09:21:29.654578 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:29.654568 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="26d53e53-185a-4659-98b7-22d2510d7eed" containerName="util" Jul 23 09:21:29.654578 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:29.654580 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26d53e53-185a-4659-98b7-22d2510d7eed" containerName="pull" Jul 23 09:21:29.654756 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:29.654586 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="26d53e53-185a-4659-98b7-22d2510d7eed" containerName="pull" Jul 23 09:21:29.654756 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:29.654648 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="26d53e53-185a-4659-98b7-22d2510d7eed" containerName="extract" Jul 23 09:21:29.660763 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:29.660737 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-v5drt" Jul 23 09:21:29.663082 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:29.663059 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-vvd4w\"" Jul 23 09:21:29.668234 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:29.668137 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-v5drt"] Jul 23 09:21:29.731072 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:29.731043 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7be0e692-4152-4b66-9de4-e512e1b1fde7-bound-sa-token\") pod \"cert-manager-55b4d954d5-v5drt\" (UID: \"7be0e692-4152-4b66-9de4-e512e1b1fde7\") " pod="cert-manager/cert-manager-55b4d954d5-v5drt" Jul 23 09:21:29.731227 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:29.731082 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bgdp\" (UniqueName: \"kubernetes.io/projected/7be0e692-4152-4b66-9de4-e512e1b1fde7-kube-api-access-2bgdp\") pod \"cert-manager-55b4d954d5-v5drt\" (UID: \"7be0e692-4152-4b66-9de4-e512e1b1fde7\") " pod="cert-manager/cert-manager-55b4d954d5-v5drt" Jul 23 09:21:29.832360 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:29.832319 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7be0e692-4152-4b66-9de4-e512e1b1fde7-bound-sa-token\") pod \"cert-manager-55b4d954d5-v5drt\" (UID: \"7be0e692-4152-4b66-9de4-e512e1b1fde7\") " pod="cert-manager/cert-manager-55b4d954d5-v5drt" Jul 23 09:21:29.832360 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:29.832361 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bgdp\" (UniqueName: \"kubernetes.io/projected/7be0e692-4152-4b66-9de4-e512e1b1fde7-kube-api-access-2bgdp\") pod \"cert-manager-55b4d954d5-v5drt\" (UID: \"7be0e692-4152-4b66-9de4-e512e1b1fde7\") " pod="cert-manager/cert-manager-55b4d954d5-v5drt" Jul 23 09:21:29.840784 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:29.840749 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7be0e692-4152-4b66-9de4-e512e1b1fde7-bound-sa-token\") pod \"cert-manager-55b4d954d5-v5drt\" (UID: \"7be0e692-4152-4b66-9de4-e512e1b1fde7\") " pod="cert-manager/cert-manager-55b4d954d5-v5drt" Jul 23 09:21:29.840917 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:29.840787 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bgdp\" (UniqueName: \"kubernetes.io/projected/7be0e692-4152-4b66-9de4-e512e1b1fde7-kube-api-access-2bgdp\") pod \"cert-manager-55b4d954d5-v5drt\" (UID: \"7be0e692-4152-4b66-9de4-e512e1b1fde7\") " pod="cert-manager/cert-manager-55b4d954d5-v5drt" Jul 23 09:21:29.971291 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:29.971201 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-v5drt" Jul 23 09:21:30.107479 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:30.107454 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-v5drt"] Jul 23 09:21:30.109892 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:21:30.109862 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7be0e692_4152_4b66_9de4_e512e1b1fde7.slice/crio-9040a15171f313bc7e8a57c6a6ad20e3b2b02105c17a423a10d262b5a3e54f79 WatchSource:0}: Error finding container 9040a15171f313bc7e8a57c6a6ad20e3b2b02105c17a423a10d262b5a3e54f79: Status 404 returned error can't find the container with id 9040a15171f313bc7e8a57c6a6ad20e3b2b02105c17a423a10d262b5a3e54f79 Jul 23 09:21:30.419183 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:30.419145 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-v5drt" event={"ID":"7be0e692-4152-4b66-9de4-e512e1b1fde7","Type":"ContainerStarted","Data":"9eecbd6cddb971b1fcd6034b46677f89a391cbaf29a61cb1a438ebfcd8cd7829"} Jul 23 09:21:30.419183 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:30.419191 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-v5drt" event={"ID":"7be0e692-4152-4b66-9de4-e512e1b1fde7","Type":"ContainerStarted","Data":"9040a15171f313bc7e8a57c6a6ad20e3b2b02105c17a423a10d262b5a3e54f79"} Jul 23 09:21:30.438271 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:30.438212 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-v5drt" podStartSLOduration=1.4381961269999999 podStartE2EDuration="1.438196127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:21:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:21:30.436331085 +0000 UTC m=+278.975609663" watchObservedRunningTime="2026-07-23 09:21:30.438196127 +0000 UTC m=+278.977474703" Jul 23 09:21:32.566628 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:32.566599 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:21:32.662748 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:32.662715 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:21:38.934561 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:38.934530 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:21:39.033329 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:39.033296 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:21:39.312129 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:39.312046 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:21:39.419554 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:39.419524 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:21:41.219042 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:41.219002 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkvbsh"] Jul 23 09:21:41.222357 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:41.222336 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkvbsh" Jul 23 09:21:41.224870 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:41.224851 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 23 09:21:41.224973 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:41.224848 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-j6qwj\"" Jul 23 09:21:41.225768 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:41.225753 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 23 09:21:41.232958 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:41.232939 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkvbsh"] Jul 23 09:21:41.330217 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:41.330188 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/63514944-96dc-44a1-b8d4-79c552dd90e7-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkvbsh\" (UID: \"63514944-96dc-44a1-b8d4-79c552dd90e7\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkvbsh" Jul 23 09:21:41.330375 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:41.330233 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm54p\" (UniqueName: \"kubernetes.io/projected/63514944-96dc-44a1-b8d4-79c552dd90e7-kube-api-access-dm54p\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkvbsh\" (UID: \"63514944-96dc-44a1-b8d4-79c552dd90e7\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkvbsh" Jul 23 09:21:41.330375 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:41.330275 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/63514944-96dc-44a1-b8d4-79c552dd90e7-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkvbsh\" (UID: \"63514944-96dc-44a1-b8d4-79c552dd90e7\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkvbsh" Jul 23 09:21:41.430987 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:41.430952 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/63514944-96dc-44a1-b8d4-79c552dd90e7-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkvbsh\" (UID: \"63514944-96dc-44a1-b8d4-79c552dd90e7\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkvbsh" Jul 23 09:21:41.431151 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:41.431003 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm54p\" (UniqueName: \"kubernetes.io/projected/63514944-96dc-44a1-b8d4-79c552dd90e7-kube-api-access-dm54p\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkvbsh\" (UID: \"63514944-96dc-44a1-b8d4-79c552dd90e7\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkvbsh" Jul 23 09:21:41.431151 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:41.431031 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/63514944-96dc-44a1-b8d4-79c552dd90e7-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkvbsh\" (UID: \"63514944-96dc-44a1-b8d4-79c552dd90e7\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkvbsh" Jul 23 09:21:41.431407 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:41.431383 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/63514944-96dc-44a1-b8d4-79c552dd90e7-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkvbsh\" (UID: \"63514944-96dc-44a1-b8d4-79c552dd90e7\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkvbsh" Jul 23 09:21:41.431481 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:41.431434 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/63514944-96dc-44a1-b8d4-79c552dd90e7-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkvbsh\" (UID: \"63514944-96dc-44a1-b8d4-79c552dd90e7\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkvbsh" Jul 23 09:21:41.442088 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:41.442056 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm54p\" (UniqueName: \"kubernetes.io/projected/63514944-96dc-44a1-b8d4-79c552dd90e7-kube-api-access-dm54p\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkvbsh\" (UID: \"63514944-96dc-44a1-b8d4-79c552dd90e7\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkvbsh" Jul 23 09:21:41.457779 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:41.457750 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:21:41.533531 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:41.533450 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkvbsh" Jul 23 09:21:41.665831 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:41.665807 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkvbsh"] Jul 23 09:21:41.668391 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:21:41.668367 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63514944_96dc_44a1_b8d4_79c552dd90e7.slice/crio-20c6688cf7b0bdfa0e2d9f2993bf12fae68472fed488449572e0c16aee5a4f9c WatchSource:0}: Error finding container 20c6688cf7b0bdfa0e2d9f2993bf12fae68472fed488449572e0c16aee5a4f9c: Status 404 returned error can't find the container with id 20c6688cf7b0bdfa0e2d9f2993bf12fae68472fed488449572e0c16aee5a4f9c Jul 23 09:21:42.474053 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:42.474019 2576 generic.go:356] "Generic (PLEG): container finished" podID="63514944-96dc-44a1-b8d4-79c552dd90e7" containerID="76916b51e61e77fe80ac74f46d3eb84d24b79592cb4786329d3187e6c40da3b2" exitCode=0 Jul 23 09:21:42.474411 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:42.474108 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkvbsh" event={"ID":"63514944-96dc-44a1-b8d4-79c552dd90e7","Type":"ContainerDied","Data":"76916b51e61e77fe80ac74f46d3eb84d24b79592cb4786329d3187e6c40da3b2"} Jul 23 09:21:42.474411 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:42.474156 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkvbsh" event={"ID":"63514944-96dc-44a1-b8d4-79c552dd90e7","Type":"ContainerStarted","Data":"20c6688cf7b0bdfa0e2d9f2993bf12fae68472fed488449572e0c16aee5a4f9c"} Jul 23 09:21:43.480492 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:43.480412 2576 generic.go:356] "Generic (PLEG): container finished" podID="63514944-96dc-44a1-b8d4-79c552dd90e7" containerID="95cc1c2886c3965bd98b4eb755e6d88ec4dc1e503bab3d0b6821612669895275" exitCode=0 Jul 23 09:21:43.480492 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:43.480456 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkvbsh" event={"ID":"63514944-96dc-44a1-b8d4-79c552dd90e7","Type":"ContainerDied","Data":"95cc1c2886c3965bd98b4eb755e6d88ec4dc1e503bab3d0b6821612669895275"} Jul 23 09:21:44.487423 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:44.487389 2576 generic.go:356] "Generic (PLEG): container finished" podID="63514944-96dc-44a1-b8d4-79c552dd90e7" containerID="827f1a337d7d155dd8abb3822aa37e50de909f3d9fc3611bc35f37d7883bbfbe" exitCode=0 Jul 23 09:21:44.487878 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:44.487440 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkvbsh" event={"ID":"63514944-96dc-44a1-b8d4-79c552dd90e7","Type":"ContainerDied","Data":"827f1a337d7d155dd8abb3822aa37e50de909f3d9fc3611bc35f37d7883bbfbe"} Jul 23 09:21:45.617298 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:45.617234 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkvbsh" Jul 23 09:21:45.770552 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:45.770463 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dm54p\" (UniqueName: \"kubernetes.io/projected/63514944-96dc-44a1-b8d4-79c552dd90e7-kube-api-access-dm54p\") pod \"63514944-96dc-44a1-b8d4-79c552dd90e7\" (UID: \"63514944-96dc-44a1-b8d4-79c552dd90e7\") " Jul 23 09:21:45.770707 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:45.770578 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/63514944-96dc-44a1-b8d4-79c552dd90e7-util\") pod \"63514944-96dc-44a1-b8d4-79c552dd90e7\" (UID: \"63514944-96dc-44a1-b8d4-79c552dd90e7\") " Jul 23 09:21:45.770707 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:45.770601 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/63514944-96dc-44a1-b8d4-79c552dd90e7-bundle\") pod \"63514944-96dc-44a1-b8d4-79c552dd90e7\" (UID: \"63514944-96dc-44a1-b8d4-79c552dd90e7\") " Jul 23 09:21:45.771430 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:45.771400 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63514944-96dc-44a1-b8d4-79c552dd90e7-bundle" (OuterVolumeSpecName: "bundle") pod "63514944-96dc-44a1-b8d4-79c552dd90e7" (UID: "63514944-96dc-44a1-b8d4-79c552dd90e7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 09:21:45.772693 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:45.772667 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63514944-96dc-44a1-b8d4-79c552dd90e7-kube-api-access-dm54p" (OuterVolumeSpecName: "kube-api-access-dm54p") pod "63514944-96dc-44a1-b8d4-79c552dd90e7" (UID: "63514944-96dc-44a1-b8d4-79c552dd90e7"). InnerVolumeSpecName "kube-api-access-dm54p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 09:21:45.776579 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:45.776545 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63514944-96dc-44a1-b8d4-79c552dd90e7-util" (OuterVolumeSpecName: "util") pod "63514944-96dc-44a1-b8d4-79c552dd90e7" (UID: "63514944-96dc-44a1-b8d4-79c552dd90e7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 09:21:45.861030 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:45.860999 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:21:45.871531 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:45.871509 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dm54p\" (UniqueName: \"kubernetes.io/projected/63514944-96dc-44a1-b8d4-79c552dd90e7-kube-api-access-dm54p\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:21:45.871531 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:45.871530 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/63514944-96dc-44a1-b8d4-79c552dd90e7-util\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:21:45.871647 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:45.871539 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/63514944-96dc-44a1-b8d4-79c552dd90e7-bundle\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:21:45.916530 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:45.916504 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:21:46.499703 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:46.499669 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkvbsh" event={"ID":"63514944-96dc-44a1-b8d4-79c552dd90e7","Type":"ContainerDied","Data":"20c6688cf7b0bdfa0e2d9f2993bf12fae68472fed488449572e0c16aee5a4f9c"} Jul 23 09:21:46.499868 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:46.499726 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20c6688cf7b0bdfa0e2d9f2993bf12fae68472fed488449572e0c16aee5a4f9c" Jul 23 09:21:46.499868 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:46.499679 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkvbsh" Jul 23 09:21:51.813845 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:51.813819 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-5szxg_4908aff3-196e-44fa-b8e6-c0e81b476433/console-operator/2.log" Jul 23 09:21:51.825252 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:51.825195 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-5szxg_4908aff3-196e-44fa-b8e6-c0e81b476433/console-operator/2.log" Jul 23 09:21:51.832878 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:51.832855 2576 kubelet.go:1648] "Image garbage collection succeeded" Jul 23 09:21:53.221501 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:53.221472 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:21:53.430342 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:53.430311 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:21:53.764409 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:53.764376 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:21:53.958048 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:53.958018 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:21:54.083473 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:54.083371 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:21:54.161514 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:54.161475 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-64d6699bc9-xvzrs"] Jul 23 09:21:54.161865 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:54.161848 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63514944-96dc-44a1-b8d4-79c552dd90e7" containerName="extract" Jul 23 09:21:54.161947 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:54.161869 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="63514944-96dc-44a1-b8d4-79c552dd90e7" containerName="extract" Jul 23 09:21:54.161947 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:54.161910 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63514944-96dc-44a1-b8d4-79c552dd90e7" containerName="pull" Jul 23 09:21:54.161947 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:54.161920 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="63514944-96dc-44a1-b8d4-79c552dd90e7" containerName="pull" Jul 23 09:21:54.161947 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:54.161940 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63514944-96dc-44a1-b8d4-79c552dd90e7" containerName="util" Jul 23 09:21:54.161947 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:54.161948 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="63514944-96dc-44a1-b8d4-79c552dd90e7" containerName="util" Jul 23 09:21:54.162193 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:54.162027 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="63514944-96dc-44a1-b8d4-79c552dd90e7" containerName="extract" Jul 23 09:21:54.163961 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:54.163939 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-64d6699bc9-xvzrs" Jul 23 09:21:54.166838 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:54.166813 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 23 09:21:54.172532 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:54.172512 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-lfn6j\"" Jul 23 09:21:54.172649 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:54.172539 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 23 09:21:54.172649 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:54.172576 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 23 09:21:54.172649 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:54.172586 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 23 09:21:54.182509 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:54.182483 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-64d6699bc9-xvzrs"] Jul 23 09:21:54.348540 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:54.348450 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:21:54.348540 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:54.348495 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crpwr\" (UniqueName: \"kubernetes.io/projected/01d0eb90-24fa-47a5-8019-bf35aaba7e1d-kube-api-access-crpwr\") pod \"opendatahub-operator-controller-manager-64d6699bc9-xvzrs\" (UID: \"01d0eb90-24fa-47a5-8019-bf35aaba7e1d\") " pod="opendatahub/opendatahub-operator-controller-manager-64d6699bc9-xvzrs" Jul 23 09:21:54.348952 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:54.348552 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/01d0eb90-24fa-47a5-8019-bf35aaba7e1d-tmp\") pod \"opendatahub-operator-controller-manager-64d6699bc9-xvzrs\" (UID: \"01d0eb90-24fa-47a5-8019-bf35aaba7e1d\") " pod="opendatahub/opendatahub-operator-controller-manager-64d6699bc9-xvzrs" Jul 23 09:21:54.348952 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:54.348588 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/01d0eb90-24fa-47a5-8019-bf35aaba7e1d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-64d6699bc9-xvzrs\" (UID: \"01d0eb90-24fa-47a5-8019-bf35aaba7e1d\") " pod="opendatahub/opendatahub-operator-controller-manager-64d6699bc9-xvzrs" Jul 23 09:21:54.348952 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:54.348622 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/01d0eb90-24fa-47a5-8019-bf35aaba7e1d-manifests\") pod \"opendatahub-operator-controller-manager-64d6699bc9-xvzrs\" (UID: \"01d0eb90-24fa-47a5-8019-bf35aaba7e1d\") " pod="opendatahub/opendatahub-operator-controller-manager-64d6699bc9-xvzrs" Jul 23 09:21:54.348952 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:54.348640 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/01d0eb90-24fa-47a5-8019-bf35aaba7e1d-webhook-cert\") pod \"opendatahub-operator-controller-manager-64d6699bc9-xvzrs\" (UID: \"01d0eb90-24fa-47a5-8019-bf35aaba7e1d\") " pod="opendatahub/opendatahub-operator-controller-manager-64d6699bc9-xvzrs" Jul 23 09:21:54.448656 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:54.448628 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:21:54.449771 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:54.449741 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crpwr\" (UniqueName: \"kubernetes.io/projected/01d0eb90-24fa-47a5-8019-bf35aaba7e1d-kube-api-access-crpwr\") pod \"opendatahub-operator-controller-manager-64d6699bc9-xvzrs\" (UID: \"01d0eb90-24fa-47a5-8019-bf35aaba7e1d\") " pod="opendatahub/opendatahub-operator-controller-manager-64d6699bc9-xvzrs" Jul 23 09:21:54.449916 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:54.449785 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/01d0eb90-24fa-47a5-8019-bf35aaba7e1d-tmp\") pod \"opendatahub-operator-controller-manager-64d6699bc9-xvzrs\" (UID: \"01d0eb90-24fa-47a5-8019-bf35aaba7e1d\") " pod="opendatahub/opendatahub-operator-controller-manager-64d6699bc9-xvzrs" Jul 23 09:21:54.449916 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:54.449809 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/01d0eb90-24fa-47a5-8019-bf35aaba7e1d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-64d6699bc9-xvzrs\" (UID: \"01d0eb90-24fa-47a5-8019-bf35aaba7e1d\") " pod="opendatahub/opendatahub-operator-controller-manager-64d6699bc9-xvzrs" Jul 23 09:21:54.449916 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:54.449849 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/01d0eb90-24fa-47a5-8019-bf35aaba7e1d-manifests\") pod \"opendatahub-operator-controller-manager-64d6699bc9-xvzrs\" (UID: \"01d0eb90-24fa-47a5-8019-bf35aaba7e1d\") " pod="opendatahub/opendatahub-operator-controller-manager-64d6699bc9-xvzrs" Jul 23 09:21:54.449916 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:54.449867 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/01d0eb90-24fa-47a5-8019-bf35aaba7e1d-webhook-cert\") pod \"opendatahub-operator-controller-manager-64d6699bc9-xvzrs\" (UID: \"01d0eb90-24fa-47a5-8019-bf35aaba7e1d\") " pod="opendatahub/opendatahub-operator-controller-manager-64d6699bc9-xvzrs" Jul 23 09:21:54.450308 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:54.450281 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/01d0eb90-24fa-47a5-8019-bf35aaba7e1d-manifests\") pod \"opendatahub-operator-controller-manager-64d6699bc9-xvzrs\" (UID: \"01d0eb90-24fa-47a5-8019-bf35aaba7e1d\") " pod="opendatahub/opendatahub-operator-controller-manager-64d6699bc9-xvzrs" Jul 23 09:21:54.452231 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:54.452200 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/01d0eb90-24fa-47a5-8019-bf35aaba7e1d-tmp\") pod \"opendatahub-operator-controller-manager-64d6699bc9-xvzrs\" (UID: \"01d0eb90-24fa-47a5-8019-bf35aaba7e1d\") " pod="opendatahub/opendatahub-operator-controller-manager-64d6699bc9-xvzrs" Jul 23 09:21:54.452360 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:54.452312 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/01d0eb90-24fa-47a5-8019-bf35aaba7e1d-webhook-cert\") pod \"opendatahub-operator-controller-manager-64d6699bc9-xvzrs\" (UID: \"01d0eb90-24fa-47a5-8019-bf35aaba7e1d\") " pod="opendatahub/opendatahub-operator-controller-manager-64d6699bc9-xvzrs" Jul 23 09:21:54.452360 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:54.452339 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/01d0eb90-24fa-47a5-8019-bf35aaba7e1d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-64d6699bc9-xvzrs\" (UID: \"01d0eb90-24fa-47a5-8019-bf35aaba7e1d\") " pod="opendatahub/opendatahub-operator-controller-manager-64d6699bc9-xvzrs" Jul 23 09:21:54.462374 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:54.462350 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crpwr\" (UniqueName: \"kubernetes.io/projected/01d0eb90-24fa-47a5-8019-bf35aaba7e1d-kube-api-access-crpwr\") pod \"opendatahub-operator-controller-manager-64d6699bc9-xvzrs\" (UID: \"01d0eb90-24fa-47a5-8019-bf35aaba7e1d\") " pod="opendatahub/opendatahub-operator-controller-manager-64d6699bc9-xvzrs" Jul 23 09:21:54.474312 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:54.474290 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-64d6699bc9-xvzrs" Jul 23 09:21:54.548559 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:54.548533 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:21:54.614559 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:54.614535 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-64d6699bc9-xvzrs"] Jul 23 09:21:54.617215 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:21:54.617187 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01d0eb90_24fa_47a5_8019_bf35aaba7e1d.slice/crio-14fb82bd6f7feffa30731aef40db28e55312e5cd235be1e2622bb13207d1d717 WatchSource:0}: Error finding container 14fb82bd6f7feffa30731aef40db28e55312e5cd235be1e2622bb13207d1d717: Status 404 returned error can't find the container with id 14fb82bd6f7feffa30731aef40db28e55312e5cd235be1e2622bb13207d1d717 Jul 23 09:21:54.619049 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:54.619032 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 09:21:54.652592 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:54.652558 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:21:54.753057 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:54.753031 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:21:54.849923 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:54.849887 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:21:55.543290 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:55.543224 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-64d6699bc9-xvzrs" event={"ID":"01d0eb90-24fa-47a5-8019-bf35aaba7e1d","Type":"ContainerStarted","Data":"14fb82bd6f7feffa30731aef40db28e55312e5cd235be1e2622bb13207d1d717"} Jul 23 09:21:58.560329 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:58.560296 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-64d6699bc9-xvzrs" event={"ID":"01d0eb90-24fa-47a5-8019-bf35aaba7e1d","Type":"ContainerStarted","Data":"c0d9265ea2ff8574244bddd98ea26eb2a532ad7305cd0e9daa0e1a56fbd27c8c"} Jul 23 09:21:59.566963 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:59.566926 2576 generic.go:356] "Generic (PLEG): container finished" podID="01d0eb90-24fa-47a5-8019-bf35aaba7e1d" containerID="c0d9265ea2ff8574244bddd98ea26eb2a532ad7305cd0e9daa0e1a56fbd27c8c" exitCode=0 Jul 23 09:21:59.567383 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:21:59.567015 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-64d6699bc9-xvzrs" event={"ID":"01d0eb90-24fa-47a5-8019-bf35aaba7e1d","Type":"ContainerDied","Data":"c0d9265ea2ff8574244bddd98ea26eb2a532ad7305cd0e9daa0e1a56fbd27c8c"} Jul 23 09:22:00.573440 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:22:00.573404 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-64d6699bc9-xvzrs" event={"ID":"01d0eb90-24fa-47a5-8019-bf35aaba7e1d","Type":"ContainerStarted","Data":"9987be15cbc707cdb4ecbf27ee34d19f8ed7be9f26df6d6f7890b505faede2dd"} Jul 23 09:22:00.573867 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:22:00.573548 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-64d6699bc9-xvzrs" Jul 23 09:22:00.603217 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:22:00.602972 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-64d6699bc9-xvzrs" podStartSLOduration=2.209130074 podStartE2EDuration="6.602953248s" totalImagesPullingTime="3.305053125s" totalInitContainerRuntime="1.088770049s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:21:54 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:22:00.600957857 +0000 UTC m=+309.140236460" watchObservedRunningTime="2026-07-23 09:22:00.602953248 +0000 UTC m=+309.142231826" Jul 23 09:22:06.957004 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:22:06.956966 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:22:08.198478 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:22:08.198443 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:22:11.580053 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:22:11.580025 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-64d6699bc9-xvzrs" Jul 23 09:22:13.330054 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:22:13.330017 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:22:13.442582 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:22:13.442548 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:22:13.998779 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:22:13.998752 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:22:14.115974 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:22:14.115944 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:22:14.216458 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:22:14.216423 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:22:14.415903 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:22:14.415876 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:22:14.607306 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:22:14.607273 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:22:14.817653 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:22:14.817585 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:22:19.187193 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:22:19.187163 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:22:19.235040 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:22:19.235014 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5dccdf9c4-2fgq7"] Jul 23 09:22:19.237608 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:22:19.237587 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5dccdf9c4-2fgq7" Jul 23 09:22:19.240533 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:22:19.240513 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 23 09:22:19.240633 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:22:19.240535 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-nktwx\"" Jul 23 09:22:19.240633 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:22:19.240515 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 23 09:22:19.246979 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:22:19.246956 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5dccdf9c4-2fgq7"] Jul 23 09:22:19.361726 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:22:19.361692 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dc7f1999-2e2f-41be-ae43-298dbd611e14-tmp\") pod \"kube-auth-proxy-5dccdf9c4-2fgq7\" (UID: \"dc7f1999-2e2f-41be-ae43-298dbd611e14\") " pod="openshift-ingress/kube-auth-proxy-5dccdf9c4-2fgq7" Jul 23 09:22:19.361888 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:22:19.361764 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml7k8\" (UniqueName: \"kubernetes.io/projected/dc7f1999-2e2f-41be-ae43-298dbd611e14-kube-api-access-ml7k8\") pod \"kube-auth-proxy-5dccdf9c4-2fgq7\" (UID: \"dc7f1999-2e2f-41be-ae43-298dbd611e14\") " pod="openshift-ingress/kube-auth-proxy-5dccdf9c4-2fgq7" Jul 23 09:22:19.361888 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:22:19.361787 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc7f1999-2e2f-41be-ae43-298dbd611e14-tls-certs\") pod \"kube-auth-proxy-5dccdf9c4-2fgq7\" (UID: \"dc7f1999-2e2f-41be-ae43-298dbd611e14\") " pod="openshift-ingress/kube-auth-proxy-5dccdf9c4-2fgq7" Jul 23 09:22:19.463232 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:22:19.463145 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml7k8\" (UniqueName: \"kubernetes.io/projected/dc7f1999-2e2f-41be-ae43-298dbd611e14-kube-api-access-ml7k8\") pod \"kube-auth-proxy-5dccdf9c4-2fgq7\" (UID: \"dc7f1999-2e2f-41be-ae43-298dbd611e14\") " pod="openshift-ingress/kube-auth-proxy-5dccdf9c4-2fgq7" Jul 23 09:22:19.463232 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:22:19.463185 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc7f1999-2e2f-41be-ae43-298dbd611e14-tls-certs\") pod \"kube-auth-proxy-5dccdf9c4-2fgq7\" (UID: \"dc7f1999-2e2f-41be-ae43-298dbd611e14\") " pod="openshift-ingress/kube-auth-proxy-5dccdf9c4-2fgq7" Jul 23 09:22:19.463232 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:22:19.463216 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dc7f1999-2e2f-41be-ae43-298dbd611e14-tmp\") pod \"kube-auth-proxy-5dccdf9c4-2fgq7\" (UID: \"dc7f1999-2e2f-41be-ae43-298dbd611e14\") " pod="openshift-ingress/kube-auth-proxy-5dccdf9c4-2fgq7" Jul 23 09:22:19.463715 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:22:19.463347 2576 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 23 09:22:19.463715 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:22:19.463557 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc7f1999-2e2f-41be-ae43-298dbd611e14-tls-certs podName:dc7f1999-2e2f-41be-ae43-298dbd611e14 nodeName:}" failed. No retries permitted until 2026-07-23 09:22:19.963414818 +0000 UTC m=+328.502693376 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/dc7f1999-2e2f-41be-ae43-298dbd611e14-tls-certs") pod "kube-auth-proxy-5dccdf9c4-2fgq7" (UID: "dc7f1999-2e2f-41be-ae43-298dbd611e14") : secret "kube-auth-proxy-tls" not found Jul 23 09:22:19.465534 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:22:19.465510 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dc7f1999-2e2f-41be-ae43-298dbd611e14-tmp\") pod \"kube-auth-proxy-5dccdf9c4-2fgq7\" (UID: \"dc7f1999-2e2f-41be-ae43-298dbd611e14\") " pod="openshift-ingress/kube-auth-proxy-5dccdf9c4-2fgq7" Jul 23 09:22:19.471569 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:22:19.471546 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml7k8\" (UniqueName: \"kubernetes.io/projected/dc7f1999-2e2f-41be-ae43-298dbd611e14-kube-api-access-ml7k8\") pod \"kube-auth-proxy-5dccdf9c4-2fgq7\" (UID: \"dc7f1999-2e2f-41be-ae43-298dbd611e14\") " pod="openshift-ingress/kube-auth-proxy-5dccdf9c4-2fgq7" Jul 23 09:22:19.514512 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:22:19.514491 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:22:19.968503 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:22:19.968466 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc7f1999-2e2f-41be-ae43-298dbd611e14-tls-certs\") pod \"kube-auth-proxy-5dccdf9c4-2fgq7\" (UID: \"dc7f1999-2e2f-41be-ae43-298dbd611e14\") " pod="openshift-ingress/kube-auth-proxy-5dccdf9c4-2fgq7" Jul 23 09:22:19.970848 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:22:19.970823 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc7f1999-2e2f-41be-ae43-298dbd611e14-tls-certs\") pod \"kube-auth-proxy-5dccdf9c4-2fgq7\" (UID: \"dc7f1999-2e2f-41be-ae43-298dbd611e14\") " pod="openshift-ingress/kube-auth-proxy-5dccdf9c4-2fgq7" Jul 23 09:22:20.147615 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:22:20.147577 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5dccdf9c4-2fgq7" Jul 23 09:22:20.287691 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:22:20.287663 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5dccdf9c4-2fgq7"] Jul 23 09:22:20.290076 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:22:20.290050 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc7f1999_2e2f_41be_ae43_298dbd611e14.slice/crio-818b04070ad769cbc599283fcea1df54f9e723ce645143fbc46fb27cda8cca71 WatchSource:0}: Error finding container 818b04070ad769cbc599283fcea1df54f9e723ce645143fbc46fb27cda8cca71: Status 404 returned error can't find the container with id 818b04070ad769cbc599283fcea1df54f9e723ce645143fbc46fb27cda8cca71 Jul 23 09:22:20.630564 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:22:20.630535 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:22:20.667394 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:22:20.667364 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5dccdf9c4-2fgq7" event={"ID":"dc7f1999-2e2f-41be-ae43-298dbd611e14","Type":"ContainerStarted","Data":"818b04070ad769cbc599283fcea1df54f9e723ce645143fbc46fb27cda8cca71"} Jul 23 09:22:20.763253 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:22:20.763212 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:22:21.969642 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:22:21.969613 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:22:22.510383 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:22:22.510346 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6cb78dd96b-577wn"] Jul 23 09:22:22.513995 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:22:22.513974 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6cb78dd96b-577wn" Jul 23 09:22:22.516413 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:22:22.516199 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 23 09:22:22.516413 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:22:22.516318 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 23 09:22:22.517067 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:22:22.517014 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 23 09:22:22.517590 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:22:22.517567 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-2mvw2\"" Jul 23 09:22:22.517699 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:22:22.517571 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 23 09:22:22.517896 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:22:22.517875 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 23 09:22:22.526032 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:22:22.526009 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6cb78dd96b-577wn"] Jul 23 09:22:22.593294 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:22:22.593229 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvgtp\" (UniqueName: \"kubernetes.io/projected/402d09a2-ca01-4485-9401-7dcd9e584fc3-kube-api-access-jvgtp\") pod \"lws-controller-manager-6cb78dd96b-577wn\" (UID: \"402d09a2-ca01-4485-9401-7dcd9e584fc3\") " pod="openshift-lws-operator/lws-controller-manager-6cb78dd96b-577wn" Jul 23 09:22:22.593466 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:22:22.593367 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/402d09a2-ca01-4485-9401-7dcd9e584fc3-metrics-cert\") pod \"lws-controller-manager-6cb78dd96b-577wn\" (UID: \"402d09a2-ca01-4485-9401-7dcd9e584fc3\") " pod="openshift-lws-operator/lws-controller-manager-6cb78dd96b-577wn" Jul 23 09:22:22.593466 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:22:22.593396 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/402d09a2-ca01-4485-9401-7dcd9e584fc3-cert\") pod \"lws-controller-manager-6cb78dd96b-577wn\" (UID: \"402d09a2-ca01-4485-9401-7dcd9e584fc3\") " pod="openshift-lws-operator/lws-controller-manager-6cb78dd96b-577wn" Jul 23 09:22:22.593466 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:22:22.593421 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/402d09a2-ca01-4485-9401-7dcd9e584fc3-manager-config\") pod \"lws-controller-manager-6cb78dd96b-577wn\" (UID: \"402d09a2-ca01-4485-9401-7dcd9e584fc3\") " pod="openshift-lws-operator/lws-controller-manager-6cb78dd96b-577wn" Jul 23 09:22:22.694835 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:22:22.694814 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/402d09a2-ca01-4485-9401-7dcd9e584fc3-metrics-cert\") pod \"lws-controller-manager-6cb78dd96b-577wn\" (UID: \"402d09a2-ca01-4485-9401-7dcd9e584fc3\") " pod="openshift-lws-operator/lws-controller-manager-6cb78dd96b-577wn" Jul 23 09:22:22.694919 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:22:22.694847 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/402d09a2-ca01-4485-9401-7dcd9e584fc3-cert\") pod \"lws-controller-manager-6cb78dd96b-577wn\" (UID: \"402d09a2-ca01-4485-9401-7dcd9e584fc3\") " pod="openshift-lws-operator/lws-controller-manager-6cb78dd96b-577wn" Jul 23 09:22:22.694919 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:22:22.694870 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/402d09a2-ca01-4485-9401-7dcd9e584fc3-manager-config\") pod \"lws-controller-manager-6cb78dd96b-577wn\" (UID: \"402d09a2-ca01-4485-9401-7dcd9e584fc3\") " pod="openshift-lws-operator/lws-controller-manager-6cb78dd96b-577wn" Jul 23 09:22:22.694919 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:22:22.694918 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvgtp\" (UniqueName: \"kubernetes.io/projected/402d09a2-ca01-4485-9401-7dcd9e584fc3-kube-api-access-jvgtp\") pod \"lws-controller-manager-6cb78dd96b-577wn\" (UID: \"402d09a2-ca01-4485-9401-7dcd9e584fc3\") " pod="openshift-lws-operator/lws-controller-manager-6cb78dd96b-577wn" Jul 23 09:22:22.695612 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:22:22.695582 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/402d09a2-ca01-4485-9401-7dcd9e584fc3-manager-config\") pod \"lws-controller-manager-6cb78dd96b-577wn\" (UID: \"402d09a2-ca01-4485-9401-7dcd9e584fc3\") " pod="openshift-lws-operator/lws-controller-manager-6cb78dd96b-577wn" Jul 23 09:22:22.697319 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:22:22.697300 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/402d09a2-ca01-4485-9401-7dcd9e584fc3-metrics-cert\") pod \"lws-controller-manager-6cb78dd96b-577wn\" (UID: \"402d09a2-ca01-4485-9401-7dcd9e584fc3\") " pod="openshift-lws-operator/lws-controller-manager-6cb78dd96b-577wn" Jul 23 09:22:22.697409 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:22:22.697352 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/402d09a2-ca01-4485-9401-7dcd9e584fc3-cert\") pod \"lws-controller-manager-6cb78dd96b-577wn\" (UID: \"402d09a2-ca01-4485-9401-7dcd9e584fc3\") " pod="openshift-lws-operator/lws-controller-manager-6cb78dd96b-577wn" Jul 23 09:22:22.702779 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:22:22.702759 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvgtp\" (UniqueName: \"kubernetes.io/projected/402d09a2-ca01-4485-9401-7dcd9e584fc3-kube-api-access-jvgtp\") pod \"lws-controller-manager-6cb78dd96b-577wn\" (UID: \"402d09a2-ca01-4485-9401-7dcd9e584fc3\") " pod="openshift-lws-operator/lws-controller-manager-6cb78dd96b-577wn" Jul 23 09:22:22.826747 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:22:22.826650 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6cb78dd96b-577wn" Jul 23 09:22:22.962408 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:22:22.962370 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6cb78dd96b-577wn"] Jul 23 09:22:22.964587 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:22:22.964545 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod402d09a2_ca01_4485_9401_7dcd9e584fc3.slice/crio-3e723a020433f6286556283ef1e7d432b09f070c1f7db3795b1b34aade0ec794 WatchSource:0}: Error finding container 3e723a020433f6286556283ef1e7d432b09f070c1f7db3795b1b34aade0ec794: Status 404 returned error can't find the container with id 3e723a020433f6286556283ef1e7d432b09f070c1f7db3795b1b34aade0ec794 Jul 23 09:22:23.684858 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:22:23.684807 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6cb78dd96b-577wn" event={"ID":"402d09a2-ca01-4485-9401-7dcd9e584fc3","Type":"ContainerStarted","Data":"3e723a020433f6286556283ef1e7d432b09f070c1f7db3795b1b34aade0ec794"} Jul 23 09:22:23.686670 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:22:23.686640 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5dccdf9c4-2fgq7" event={"ID":"dc7f1999-2e2f-41be-ae43-298dbd611e14","Type":"ContainerStarted","Data":"3c95f1f3bcf9939407256920712a078ed1dd7f05027629e4b227e903138e2781"} Jul 23 09:22:23.714681 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:22:23.714635 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5dccdf9c4-2fgq7" podStartSLOduration=2.315440401 podStartE2EDuration="4.714620867s" totalImagesPullingTime="2.399180466s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:22:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:22:23.712788402 +0000 UTC m=+332.252066981" watchObservedRunningTime="2026-07-23 09:22:23.714620867 +0000 UTC m=+332.253899443" Jul 23 09:22:24.625598 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:22:24.625571 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:22:24.693078 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:22:24.693041 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6cb78dd96b-577wn" event={"ID":"402d09a2-ca01-4485-9401-7dcd9e584fc3","Type":"ContainerStarted","Data":"7325deef61c31b1bb3d416aaa25937ed07516fa041c5abdcd270a6af6dc3d68e"} Jul 23 09:22:24.730340 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:22:24.730298 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6cb78dd96b-577wn" podStartSLOduration=1.454736587 podStartE2EDuration="2.730283417s" totalImagesPullingTime="1.27554683s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:22:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:22:24.725856789 +0000 UTC m=+333.265135364" watchObservedRunningTime="2026-07-23 09:22:24.730283417 +0000 UTC m=+333.269561993" Jul 23 09:22:24.742187 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:22:24.742162 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:22:25.241568 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:22:25.241533 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:22:25.441933 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:22:25.441898 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:22:25.697893 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:22:25.697866 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6cb78dd96b-577wn" Jul 23 09:22:26.225836 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:22:26.225801 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:22:26.393298 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:22:26.393262 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:22:26.495858 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:22:26.495765 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:22:27.048372 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:22:27.048341 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:22:27.146891 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:22:27.146853 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:22:27.278123 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:22:27.278094 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:22:36.704676 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:22:36.704645 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6cb78dd96b-577wn" Jul 23 09:23:14.762411 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:14.762380 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:23:14.861951 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:14.861920 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:23:14.958695 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:14.958662 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:23:15.669854 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:15.669816 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:23:15.898603 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:15.898572 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:23:30.489131 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:30.489096 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qrfl7"] Jul 23 09:23:30.491627 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:30.491608 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qrfl7" Jul 23 09:23:30.494041 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:30.494019 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 23 09:23:30.494663 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:30.494644 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 23 09:23:30.494759 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:30.494645 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-g68sf\"" Jul 23 09:23:30.501259 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:30.501215 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qrfl7"] Jul 23 09:23:30.566438 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:30.566409 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ce7494cc-693b-4499-bb8a-6646ec33f004-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qrfl7\" (UID: \"ce7494cc-693b-4499-bb8a-6646ec33f004\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qrfl7" Jul 23 09:23:30.566585 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:30.566470 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ce7494cc-693b-4499-bb8a-6646ec33f004-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qrfl7\" (UID: \"ce7494cc-693b-4499-bb8a-6646ec33f004\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qrfl7" Jul 23 09:23:30.566585 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:30.566503 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwf9t\" (UniqueName: \"kubernetes.io/projected/ce7494cc-693b-4499-bb8a-6646ec33f004-kube-api-access-dwf9t\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qrfl7\" (UID: \"ce7494cc-693b-4499-bb8a-6646ec33f004\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qrfl7" Jul 23 09:23:30.667384 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:30.667350 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwf9t\" (UniqueName: \"kubernetes.io/projected/ce7494cc-693b-4499-bb8a-6646ec33f004-kube-api-access-dwf9t\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qrfl7\" (UID: \"ce7494cc-693b-4499-bb8a-6646ec33f004\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qrfl7" Jul 23 09:23:30.667576 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:30.667409 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ce7494cc-693b-4499-bb8a-6646ec33f004-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qrfl7\" (UID: \"ce7494cc-693b-4499-bb8a-6646ec33f004\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qrfl7" Jul 23 09:23:30.667576 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:30.667459 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ce7494cc-693b-4499-bb8a-6646ec33f004-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qrfl7\" (UID: \"ce7494cc-693b-4499-bb8a-6646ec33f004\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qrfl7" Jul 23 09:23:30.667814 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:30.667796 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ce7494cc-693b-4499-bb8a-6646ec33f004-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qrfl7\" (UID: \"ce7494cc-693b-4499-bb8a-6646ec33f004\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qrfl7" Jul 23 09:23:30.667873 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:30.667854 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ce7494cc-693b-4499-bb8a-6646ec33f004-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qrfl7\" (UID: \"ce7494cc-693b-4499-bb8a-6646ec33f004\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qrfl7" Jul 23 09:23:30.676236 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:30.676217 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwf9t\" (UniqueName: \"kubernetes.io/projected/ce7494cc-693b-4499-bb8a-6646ec33f004-kube-api-access-dwf9t\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qrfl7\" (UID: \"ce7494cc-693b-4499-bb8a-6646ec33f004\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qrfl7" Jul 23 09:23:30.801208 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:30.801135 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qrfl7" Jul 23 09:23:30.882972 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:30.882943 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595h986"] Jul 23 09:23:30.885470 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:30.885454 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595h986" Jul 23 09:23:30.895526 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:30.895468 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595h986"] Jul 23 09:23:30.945972 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:30.945950 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qrfl7"] Jul 23 09:23:30.948175 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:23:30.948136 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce7494cc_693b_4499_bb8a_6646ec33f004.slice/crio-e52887835e4ea943612c8e25f364244e569af758c4fe62073752b5a62f289add WatchSource:0}: Error finding container e52887835e4ea943612c8e25f364244e569af758c4fe62073752b5a62f289add: Status 404 returned error can't find the container with id e52887835e4ea943612c8e25f364244e569af758c4fe62073752b5a62f289add Jul 23 09:23:30.971011 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:30.970989 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/51e149f3-3d54-46ad-86c9-01c83eb781f7-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595h986\" (UID: \"51e149f3-3d54-46ad-86c9-01c83eb781f7\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595h986" Jul 23 09:23:30.971138 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:30.971039 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9rwg\" (UniqueName: \"kubernetes.io/projected/51e149f3-3d54-46ad-86c9-01c83eb781f7-kube-api-access-g9rwg\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595h986\" (UID: \"51e149f3-3d54-46ad-86c9-01c83eb781f7\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595h986" Jul 23 09:23:30.971208 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:30.971143 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/51e149f3-3d54-46ad-86c9-01c83eb781f7-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595h986\" (UID: \"51e149f3-3d54-46ad-86c9-01c83eb781f7\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595h986" Jul 23 09:23:31.009400 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:31.009373 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qrfl7" event={"ID":"ce7494cc-693b-4499-bb8a-6646ec33f004","Type":"ContainerStarted","Data":"e52887835e4ea943612c8e25f364244e569af758c4fe62073752b5a62f289add"} Jul 23 09:23:31.072718 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:31.072653 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9rwg\" (UniqueName: \"kubernetes.io/projected/51e149f3-3d54-46ad-86c9-01c83eb781f7-kube-api-access-g9rwg\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595h986\" (UID: \"51e149f3-3d54-46ad-86c9-01c83eb781f7\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595h986" Jul 23 09:23:31.072718 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:31.072708 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/51e149f3-3d54-46ad-86c9-01c83eb781f7-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595h986\" (UID: \"51e149f3-3d54-46ad-86c9-01c83eb781f7\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595h986" Jul 23 09:23:31.072940 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:31.072760 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/51e149f3-3d54-46ad-86c9-01c83eb781f7-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595h986\" (UID: \"51e149f3-3d54-46ad-86c9-01c83eb781f7\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595h986" Jul 23 09:23:31.073051 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:31.073030 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/51e149f3-3d54-46ad-86c9-01c83eb781f7-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595h986\" (UID: \"51e149f3-3d54-46ad-86c9-01c83eb781f7\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595h986" Jul 23 09:23:31.073110 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:31.073069 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/51e149f3-3d54-46ad-86c9-01c83eb781f7-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595h986\" (UID: \"51e149f3-3d54-46ad-86c9-01c83eb781f7\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595h986" Jul 23 09:23:31.080742 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:31.080719 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9rwg\" (UniqueName: \"kubernetes.io/projected/51e149f3-3d54-46ad-86c9-01c83eb781f7-kube-api-access-g9rwg\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595h986\" (UID: \"51e149f3-3d54-46ad-86c9-01c83eb781f7\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595h986" Jul 23 09:23:31.201338 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:31.201293 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595h986" Jul 23 09:23:31.282078 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:31.282050 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cz5gk"] Jul 23 09:23:31.285384 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:31.285359 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cz5gk" Jul 23 09:23:31.293822 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:31.293795 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cz5gk"] Jul 23 09:23:31.332326 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:31.332265 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595h986"] Jul 23 09:23:31.334575 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:23:31.334551 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51e149f3_3d54_46ad_86c9_01c83eb781f7.slice/crio-ad8c46fdb170713b8ecef5b15d582ecb62a85afbbd180593b3905c91edaef721 WatchSource:0}: Error finding container ad8c46fdb170713b8ecef5b15d582ecb62a85afbbd180593b3905c91edaef721: Status 404 returned error can't find the container with id ad8c46fdb170713b8ecef5b15d582ecb62a85afbbd180593b3905c91edaef721 Jul 23 09:23:31.375527 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:31.375324 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2rz8\" (UniqueName: \"kubernetes.io/projected/c13198e7-f244-465c-82ff-199bde275f68-kube-api-access-h2rz8\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cz5gk\" (UID: \"c13198e7-f244-465c-82ff-199bde275f68\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cz5gk" Jul 23 09:23:31.375527 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:31.375422 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c13198e7-f244-465c-82ff-199bde275f68-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cz5gk\" (UID: \"c13198e7-f244-465c-82ff-199bde275f68\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cz5gk" Jul 23 09:23:31.375527 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:31.375469 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c13198e7-f244-465c-82ff-199bde275f68-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cz5gk\" (UID: \"c13198e7-f244-465c-82ff-199bde275f68\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cz5gk" Jul 23 09:23:31.476749 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:31.476713 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c13198e7-f244-465c-82ff-199bde275f68-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cz5gk\" (UID: \"c13198e7-f244-465c-82ff-199bde275f68\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cz5gk" Jul 23 09:23:31.476922 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:31.476763 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c13198e7-f244-465c-82ff-199bde275f68-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cz5gk\" (UID: \"c13198e7-f244-465c-82ff-199bde275f68\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cz5gk" Jul 23 09:23:31.476922 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:31.476828 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2rz8\" (UniqueName: \"kubernetes.io/projected/c13198e7-f244-465c-82ff-199bde275f68-kube-api-access-h2rz8\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cz5gk\" (UID: \"c13198e7-f244-465c-82ff-199bde275f68\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cz5gk" Jul 23 09:23:31.477096 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:31.477075 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c13198e7-f244-465c-82ff-199bde275f68-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cz5gk\" (UID: \"c13198e7-f244-465c-82ff-199bde275f68\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cz5gk" Jul 23 09:23:31.477136 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:31.477105 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c13198e7-f244-465c-82ff-199bde275f68-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cz5gk\" (UID: \"c13198e7-f244-465c-82ff-199bde275f68\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cz5gk" Jul 23 09:23:31.484595 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:31.484577 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2rz8\" (UniqueName: \"kubernetes.io/projected/c13198e7-f244-465c-82ff-199bde275f68-kube-api-access-h2rz8\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cz5gk\" (UID: \"c13198e7-f244-465c-82ff-199bde275f68\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cz5gk" Jul 23 09:23:31.598965 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:31.598893 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cz5gk" Jul 23 09:23:31.689823 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:31.689673 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wrs5q"] Jul 23 09:23:31.692957 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:31.692933 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wrs5q" Jul 23 09:23:31.702010 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:31.701987 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wrs5q"] Jul 23 09:23:31.761388 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:31.761361 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cz5gk"] Jul 23 09:23:31.763159 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:23:31.763133 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc13198e7_f244_465c_82ff_199bde275f68.slice/crio-0680fb85343abb6f86641571e11099d583107e7256123e6a08377a71803e845a WatchSource:0}: Error finding container 0680fb85343abb6f86641571e11099d583107e7256123e6a08377a71803e845a: Status 404 returned error can't find the container with id 0680fb85343abb6f86641571e11099d583107e7256123e6a08377a71803e845a Jul 23 09:23:31.779879 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:31.779852 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f4dae8b7-1352-4f26-93c6-c5ffd5f8369f-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wrs5q\" (UID: \"f4dae8b7-1352-4f26-93c6-c5ffd5f8369f\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wrs5q" Jul 23 09:23:31.779955 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:31.779899 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr6bf\" (UniqueName: \"kubernetes.io/projected/f4dae8b7-1352-4f26-93c6-c5ffd5f8369f-kube-api-access-rr6bf\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wrs5q\" (UID: \"f4dae8b7-1352-4f26-93c6-c5ffd5f8369f\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wrs5q" Jul 23 09:23:31.780046 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:31.780021 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f4dae8b7-1352-4f26-93c6-c5ffd5f8369f-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wrs5q\" (UID: \"f4dae8b7-1352-4f26-93c6-c5ffd5f8369f\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wrs5q" Jul 23 09:23:31.880546 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:31.880509 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f4dae8b7-1352-4f26-93c6-c5ffd5f8369f-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wrs5q\" (UID: \"f4dae8b7-1352-4f26-93c6-c5ffd5f8369f\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wrs5q" Jul 23 09:23:31.880656 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:31.880584 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr6bf\" (UniqueName: \"kubernetes.io/projected/f4dae8b7-1352-4f26-93c6-c5ffd5f8369f-kube-api-access-rr6bf\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wrs5q\" (UID: \"f4dae8b7-1352-4f26-93c6-c5ffd5f8369f\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wrs5q" Jul 23 09:23:31.880697 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:31.880655 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f4dae8b7-1352-4f26-93c6-c5ffd5f8369f-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wrs5q\" (UID: \"f4dae8b7-1352-4f26-93c6-c5ffd5f8369f\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wrs5q" Jul 23 09:23:31.880877 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:31.880863 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f4dae8b7-1352-4f26-93c6-c5ffd5f8369f-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wrs5q\" (UID: \"f4dae8b7-1352-4f26-93c6-c5ffd5f8369f\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wrs5q" Jul 23 09:23:31.880961 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:31.880938 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f4dae8b7-1352-4f26-93c6-c5ffd5f8369f-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wrs5q\" (UID: \"f4dae8b7-1352-4f26-93c6-c5ffd5f8369f\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wrs5q" Jul 23 09:23:31.889394 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:31.889375 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr6bf\" (UniqueName: \"kubernetes.io/projected/f4dae8b7-1352-4f26-93c6-c5ffd5f8369f-kube-api-access-rr6bf\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wrs5q\" (UID: \"f4dae8b7-1352-4f26-93c6-c5ffd5f8369f\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wrs5q" Jul 23 09:23:32.002860 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:32.002828 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wrs5q" Jul 23 09:23:32.015784 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:32.015753 2576 generic.go:356] "Generic (PLEG): container finished" podID="ce7494cc-693b-4499-bb8a-6646ec33f004" containerID="6a0428545e605332656f1715ef2897c39c937ace9842f41b8444ec3fa6a9575e" exitCode=0 Jul 23 09:23:32.015907 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:32.015840 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qrfl7" event={"ID":"ce7494cc-693b-4499-bb8a-6646ec33f004","Type":"ContainerDied","Data":"6a0428545e605332656f1715ef2897c39c937ace9842f41b8444ec3fa6a9575e"} Jul 23 09:23:32.017566 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:32.017545 2576 generic.go:356] "Generic (PLEG): container finished" podID="c13198e7-f244-465c-82ff-199bde275f68" containerID="ef23d6a8c8d86fc2938012b26d34b30ce8dea76bf0d6fd58169f40fe7d7ad8c4" exitCode=0 Jul 23 09:23:32.017661 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:32.017600 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cz5gk" event={"ID":"c13198e7-f244-465c-82ff-199bde275f68","Type":"ContainerDied","Data":"ef23d6a8c8d86fc2938012b26d34b30ce8dea76bf0d6fd58169f40fe7d7ad8c4"} Jul 23 09:23:32.017661 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:32.017655 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cz5gk" event={"ID":"c13198e7-f244-465c-82ff-199bde275f68","Type":"ContainerStarted","Data":"0680fb85343abb6f86641571e11099d583107e7256123e6a08377a71803e845a"} Jul 23 09:23:32.019361 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:32.019279 2576 generic.go:356] "Generic (PLEG): container finished" podID="51e149f3-3d54-46ad-86c9-01c83eb781f7" containerID="1a8b7a36ce6669bb49983cf97af0c354fe523245f59de4d6a356281e30c8ed44" exitCode=0 Jul 23 09:23:32.019361 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:32.019321 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595h986" event={"ID":"51e149f3-3d54-46ad-86c9-01c83eb781f7","Type":"ContainerDied","Data":"1a8b7a36ce6669bb49983cf97af0c354fe523245f59de4d6a356281e30c8ed44"} Jul 23 09:23:32.019361 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:32.019353 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595h986" event={"ID":"51e149f3-3d54-46ad-86c9-01c83eb781f7","Type":"ContainerStarted","Data":"ad8c46fdb170713b8ecef5b15d582ecb62a85afbbd180593b3905c91edaef721"} Jul 23 09:23:32.138833 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:32.138810 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wrs5q"] Jul 23 09:23:32.140693 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:23:32.140660 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4dae8b7_1352_4f26_93c6_c5ffd5f8369f.slice/crio-57c4756b29ff4181df42a067fa485bd2067cf0eaf0d4e335d81cfe29b630a1be WatchSource:0}: Error finding container 57c4756b29ff4181df42a067fa485bd2067cf0eaf0d4e335d81cfe29b630a1be: Status 404 returned error can't find the container with id 57c4756b29ff4181df42a067fa485bd2067cf0eaf0d4e335d81cfe29b630a1be Jul 23 09:23:33.027466 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:33.027435 2576 generic.go:356] "Generic (PLEG): container finished" podID="f4dae8b7-1352-4f26-93c6-c5ffd5f8369f" containerID="4e6251f93413c16bd2e64ecaa967fcf2b814991648a8045df14d958f9e62e788" exitCode=0 Jul 23 09:23:33.027850 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:33.027557 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wrs5q" event={"ID":"f4dae8b7-1352-4f26-93c6-c5ffd5f8369f","Type":"ContainerDied","Data":"4e6251f93413c16bd2e64ecaa967fcf2b814991648a8045df14d958f9e62e788"} Jul 23 09:23:33.027850 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:33.027626 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wrs5q" event={"ID":"f4dae8b7-1352-4f26-93c6-c5ffd5f8369f","Type":"ContainerStarted","Data":"57c4756b29ff4181df42a067fa485bd2067cf0eaf0d4e335d81cfe29b630a1be"} Jul 23 09:23:34.034696 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:34.034664 2576 generic.go:356] "Generic (PLEG): container finished" podID="51e149f3-3d54-46ad-86c9-01c83eb781f7" containerID="33681ac4c06e6c449f38ad85ea315ac0bbec5b3ba267aaa785c3cba6be45897a" exitCode=0 Jul 23 09:23:34.035117 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:34.034754 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595h986" event={"ID":"51e149f3-3d54-46ad-86c9-01c83eb781f7","Type":"ContainerDied","Data":"33681ac4c06e6c449f38ad85ea315ac0bbec5b3ba267aaa785c3cba6be45897a"} Jul 23 09:23:34.036507 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:34.036483 2576 generic.go:356] "Generic (PLEG): container finished" podID="ce7494cc-693b-4499-bb8a-6646ec33f004" containerID="32fd81025cb2c44760aa47e78ff70b23a6f7a0d36559e108402fff7c29ed7ddf" exitCode=0 Jul 23 09:23:34.036613 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:34.036547 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qrfl7" event={"ID":"ce7494cc-693b-4499-bb8a-6646ec33f004","Type":"ContainerDied","Data":"32fd81025cb2c44760aa47e78ff70b23a6f7a0d36559e108402fff7c29ed7ddf"} Jul 23 09:23:34.038362 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:34.038342 2576 generic.go:356] "Generic (PLEG): container finished" podID="c13198e7-f244-465c-82ff-199bde275f68" containerID="e620b7cf1792f94c06630f1c099588a8b9e21d5c91da961c77026a63cca2ab0e" exitCode=0 Jul 23 09:23:34.038493 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:34.038466 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cz5gk" event={"ID":"c13198e7-f244-465c-82ff-199bde275f68","Type":"ContainerDied","Data":"e620b7cf1792f94c06630f1c099588a8b9e21d5c91da961c77026a63cca2ab0e"} Jul 23 09:23:34.713033 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:34.712998 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6f77c8c54b-vc7db"] Jul 23 09:23:34.716512 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:34.716491 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f77c8c54b-vc7db" Jul 23 09:23:34.734259 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:34.734212 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f77c8c54b-vc7db"] Jul 23 09:23:34.809280 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:34.809216 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/010072eb-4045-446e-8bbe-51f5954bc4a8-console-oauth-config\") pod \"console-6f77c8c54b-vc7db\" (UID: \"010072eb-4045-446e-8bbe-51f5954bc4a8\") " pod="openshift-console/console-6f77c8c54b-vc7db" Jul 23 09:23:34.809280 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:34.809279 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/010072eb-4045-446e-8bbe-51f5954bc4a8-trusted-ca-bundle\") pod \"console-6f77c8c54b-vc7db\" (UID: \"010072eb-4045-446e-8bbe-51f5954bc4a8\") " pod="openshift-console/console-6f77c8c54b-vc7db" Jul 23 09:23:34.809487 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:34.809358 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/010072eb-4045-446e-8bbe-51f5954bc4a8-oauth-serving-cert\") pod \"console-6f77c8c54b-vc7db\" (UID: \"010072eb-4045-446e-8bbe-51f5954bc4a8\") " pod="openshift-console/console-6f77c8c54b-vc7db" Jul 23 09:23:34.809487 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:34.809388 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/010072eb-4045-446e-8bbe-51f5954bc4a8-console-config\") pod \"console-6f77c8c54b-vc7db\" (UID: \"010072eb-4045-446e-8bbe-51f5954bc4a8\") " pod="openshift-console/console-6f77c8c54b-vc7db" Jul 23 09:23:34.809487 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:34.809425 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/010072eb-4045-446e-8bbe-51f5954bc4a8-service-ca\") pod \"console-6f77c8c54b-vc7db\" (UID: \"010072eb-4045-446e-8bbe-51f5954bc4a8\") " pod="openshift-console/console-6f77c8c54b-vc7db" Jul 23 09:23:34.809487 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:34.809447 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cnrj\" (UniqueName: \"kubernetes.io/projected/010072eb-4045-446e-8bbe-51f5954bc4a8-kube-api-access-7cnrj\") pod \"console-6f77c8c54b-vc7db\" (UID: \"010072eb-4045-446e-8bbe-51f5954bc4a8\") " pod="openshift-console/console-6f77c8c54b-vc7db" Jul 23 09:23:34.809487 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:34.809468 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/010072eb-4045-446e-8bbe-51f5954bc4a8-console-serving-cert\") pod \"console-6f77c8c54b-vc7db\" (UID: \"010072eb-4045-446e-8bbe-51f5954bc4a8\") " pod="openshift-console/console-6f77c8c54b-vc7db" Jul 23 09:23:34.882837 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:34.882806 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:23:34.910862 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:34.910834 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cnrj\" (UniqueName: \"kubernetes.io/projected/010072eb-4045-446e-8bbe-51f5954bc4a8-kube-api-access-7cnrj\") pod \"console-6f77c8c54b-vc7db\" (UID: \"010072eb-4045-446e-8bbe-51f5954bc4a8\") " pod="openshift-console/console-6f77c8c54b-vc7db" Jul 23 09:23:34.910969 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:34.910867 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/010072eb-4045-446e-8bbe-51f5954bc4a8-console-serving-cert\") pod \"console-6f77c8c54b-vc7db\" (UID: \"010072eb-4045-446e-8bbe-51f5954bc4a8\") " pod="openshift-console/console-6f77c8c54b-vc7db" Jul 23 09:23:34.910969 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:34.910921 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/010072eb-4045-446e-8bbe-51f5954bc4a8-console-oauth-config\") pod \"console-6f77c8c54b-vc7db\" (UID: \"010072eb-4045-446e-8bbe-51f5954bc4a8\") " pod="openshift-console/console-6f77c8c54b-vc7db" Jul 23 09:23:34.910969 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:34.910940 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/010072eb-4045-446e-8bbe-51f5954bc4a8-trusted-ca-bundle\") pod \"console-6f77c8c54b-vc7db\" (UID: \"010072eb-4045-446e-8bbe-51f5954bc4a8\") " pod="openshift-console/console-6f77c8c54b-vc7db" Jul 23 09:23:34.911100 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:34.910977 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/010072eb-4045-446e-8bbe-51f5954bc4a8-oauth-serving-cert\") pod \"console-6f77c8c54b-vc7db\" (UID: \"010072eb-4045-446e-8bbe-51f5954bc4a8\") " pod="openshift-console/console-6f77c8c54b-vc7db" Jul 23 09:23:34.911100 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:34.911003 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/010072eb-4045-446e-8bbe-51f5954bc4a8-console-config\") pod \"console-6f77c8c54b-vc7db\" (UID: \"010072eb-4045-446e-8bbe-51f5954bc4a8\") " pod="openshift-console/console-6f77c8c54b-vc7db" Jul 23 09:23:34.911100 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:34.911047 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/010072eb-4045-446e-8bbe-51f5954bc4a8-service-ca\") pod \"console-6f77c8c54b-vc7db\" (UID: \"010072eb-4045-446e-8bbe-51f5954bc4a8\") " pod="openshift-console/console-6f77c8c54b-vc7db" Jul 23 09:23:34.911803 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:34.911777 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/010072eb-4045-446e-8bbe-51f5954bc4a8-oauth-serving-cert\") pod \"console-6f77c8c54b-vc7db\" (UID: \"010072eb-4045-446e-8bbe-51f5954bc4a8\") " pod="openshift-console/console-6f77c8c54b-vc7db" Jul 23 09:23:34.911953 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:34.911927 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/010072eb-4045-446e-8bbe-51f5954bc4a8-console-config\") pod \"console-6f77c8c54b-vc7db\" (UID: \"010072eb-4045-446e-8bbe-51f5954bc4a8\") " pod="openshift-console/console-6f77c8c54b-vc7db" Jul 23 09:23:34.912001 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:34.911982 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/010072eb-4045-446e-8bbe-51f5954bc4a8-trusted-ca-bundle\") pod \"console-6f77c8c54b-vc7db\" (UID: \"010072eb-4045-446e-8bbe-51f5954bc4a8\") " pod="openshift-console/console-6f77c8c54b-vc7db" Jul 23 09:23:34.912038 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:34.911996 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/010072eb-4045-446e-8bbe-51f5954bc4a8-service-ca\") pod \"console-6f77c8c54b-vc7db\" (UID: \"010072eb-4045-446e-8bbe-51f5954bc4a8\") " pod="openshift-console/console-6f77c8c54b-vc7db" Jul 23 09:23:34.913587 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:34.913567 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/010072eb-4045-446e-8bbe-51f5954bc4a8-console-oauth-config\") pod \"console-6f77c8c54b-vc7db\" (UID: \"010072eb-4045-446e-8bbe-51f5954bc4a8\") " pod="openshift-console/console-6f77c8c54b-vc7db" Jul 23 09:23:34.913667 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:34.913651 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/010072eb-4045-446e-8bbe-51f5954bc4a8-console-serving-cert\") pod \"console-6f77c8c54b-vc7db\" (UID: \"010072eb-4045-446e-8bbe-51f5954bc4a8\") " pod="openshift-console/console-6f77c8c54b-vc7db" Jul 23 09:23:34.919834 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:34.919812 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cnrj\" (UniqueName: \"kubernetes.io/projected/010072eb-4045-446e-8bbe-51f5954bc4a8-kube-api-access-7cnrj\") pod \"console-6f77c8c54b-vc7db\" (UID: \"010072eb-4045-446e-8bbe-51f5954bc4a8\") " pod="openshift-console/console-6f77c8c54b-vc7db" Jul 23 09:23:34.943037 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:34.942976 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:23:35.026387 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:35.026354 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f77c8c54b-vc7db" Jul 23 09:23:35.047521 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:35.047488 2576 generic.go:356] "Generic (PLEG): container finished" podID="f4dae8b7-1352-4f26-93c6-c5ffd5f8369f" containerID="06276233f82dd2d80b472ede4e6b2a5699dccb9484457f097b3d9b65571b971f" exitCode=0 Jul 23 09:23:35.047926 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:35.047568 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wrs5q" event={"ID":"f4dae8b7-1352-4f26-93c6-c5ffd5f8369f","Type":"ContainerDied","Data":"06276233f82dd2d80b472ede4e6b2a5699dccb9484457f097b3d9b65571b971f"} Jul 23 09:23:35.049926 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:35.049905 2576 generic.go:356] "Generic (PLEG): container finished" podID="51e149f3-3d54-46ad-86c9-01c83eb781f7" containerID="1b3dfbff73973c356ddb1f7f989dcf33e7615529c2c03a23b72437e3a7aad45b" exitCode=0 Jul 23 09:23:35.050034 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:35.049987 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595h986" event={"ID":"51e149f3-3d54-46ad-86c9-01c83eb781f7","Type":"ContainerDied","Data":"1b3dfbff73973c356ddb1f7f989dcf33e7615529c2c03a23b72437e3a7aad45b"} Jul 23 09:23:35.052417 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:35.052394 2576 generic.go:356] "Generic (PLEG): container finished" podID="ce7494cc-693b-4499-bb8a-6646ec33f004" containerID="f7654fb0b420c6ef1dbe0cc18d7896a0b5dc8c168a0092252c8d4965cf338577" exitCode=0 Jul 23 09:23:35.052594 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:35.052449 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qrfl7" event={"ID":"ce7494cc-693b-4499-bb8a-6646ec33f004","Type":"ContainerDied","Data":"f7654fb0b420c6ef1dbe0cc18d7896a0b5dc8c168a0092252c8d4965cf338577"} Jul 23 09:23:35.055322 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:35.055295 2576 generic.go:356] "Generic (PLEG): container finished" podID="c13198e7-f244-465c-82ff-199bde275f68" containerID="ebcbfc058e1f5f7c26842fb2b90409695923c5f6f89c8bd79b43d768b2c2b717" exitCode=0 Jul 23 09:23:35.055415 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:35.055378 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cz5gk" event={"ID":"c13198e7-f244-465c-82ff-199bde275f68","Type":"ContainerDied","Data":"ebcbfc058e1f5f7c26842fb2b90409695923c5f6f89c8bd79b43d768b2c2b717"} Jul 23 09:23:35.182675 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:35.182639 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f77c8c54b-vc7db"] Jul 23 09:23:35.184911 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:23:35.184882 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod010072eb_4045_446e_8bbe_51f5954bc4a8.slice/crio-4a2a1ecaf4963baa3b0689722781aa1dd46621f6c3debe0fd3d9c10e10ef800a WatchSource:0}: Error finding container 4a2a1ecaf4963baa3b0689722781aa1dd46621f6c3debe0fd3d9c10e10ef800a: Status 404 returned error can't find the container with id 4a2a1ecaf4963baa3b0689722781aa1dd46621f6c3debe0fd3d9c10e10ef800a Jul 23 09:23:35.511879 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:35.511791 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:23:35.611865 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:35.611832 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:23:36.062749 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:36.062716 2576 generic.go:356] "Generic (PLEG): container finished" podID="f4dae8b7-1352-4f26-93c6-c5ffd5f8369f" containerID="936c298c670e21f28b9807c6f52b551cc9b2c09efc73a09e51970af66d9a044a" exitCode=0 Jul 23 09:23:36.063311 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:36.062804 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wrs5q" event={"ID":"f4dae8b7-1352-4f26-93c6-c5ffd5f8369f","Type":"ContainerDied","Data":"936c298c670e21f28b9807c6f52b551cc9b2c09efc73a09e51970af66d9a044a"} Jul 23 09:23:36.064424 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:36.064403 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f77c8c54b-vc7db" event={"ID":"010072eb-4045-446e-8bbe-51f5954bc4a8","Type":"ContainerStarted","Data":"50531228d509b945ca4e3e60536a3c0a6773d808834ef87e90a1241fb4480c84"} Jul 23 09:23:36.064513 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:36.064438 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f77c8c54b-vc7db" event={"ID":"010072eb-4045-446e-8bbe-51f5954bc4a8","Type":"ContainerStarted","Data":"4a2a1ecaf4963baa3b0689722781aa1dd46621f6c3debe0fd3d9c10e10ef800a"} Jul 23 09:23:36.110349 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:36.110299 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6f77c8c54b-vc7db" podStartSLOduration=2.110283743 podStartE2EDuration="2.110283743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:23:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:23:36.108698304 +0000 UTC m=+404.647976882" watchObservedRunningTime="2026-07-23 09:23:36.110283743 +0000 UTC m=+404.649562320" Jul 23 09:23:36.209819 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:36.209795 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cz5gk" Jul 23 09:23:36.251920 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:36.251897 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595h986" Jul 23 09:23:36.255283 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:36.255259 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qrfl7" Jul 23 09:23:36.325723 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:36.325644 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c13198e7-f244-465c-82ff-199bde275f68-bundle\") pod \"c13198e7-f244-465c-82ff-199bde275f68\" (UID: \"c13198e7-f244-465c-82ff-199bde275f68\") " Jul 23 09:23:36.325900 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:36.325730 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c13198e7-f244-465c-82ff-199bde275f68-util\") pod \"c13198e7-f244-465c-82ff-199bde275f68\" (UID: \"c13198e7-f244-465c-82ff-199bde275f68\") " Jul 23 09:23:36.325900 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:36.325761 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2rz8\" (UniqueName: \"kubernetes.io/projected/c13198e7-f244-465c-82ff-199bde275f68-kube-api-access-h2rz8\") pod \"c13198e7-f244-465c-82ff-199bde275f68\" (UID: \"c13198e7-f244-465c-82ff-199bde275f68\") " Jul 23 09:23:36.326268 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:36.326208 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c13198e7-f244-465c-82ff-199bde275f68-bundle" (OuterVolumeSpecName: "bundle") pod "c13198e7-f244-465c-82ff-199bde275f68" (UID: "c13198e7-f244-465c-82ff-199bde275f68"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 09:23:36.328028 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:36.328005 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c13198e7-f244-465c-82ff-199bde275f68-kube-api-access-h2rz8" (OuterVolumeSpecName: "kube-api-access-h2rz8") pod "c13198e7-f244-465c-82ff-199bde275f68" (UID: "c13198e7-f244-465c-82ff-199bde275f68"). InnerVolumeSpecName "kube-api-access-h2rz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 09:23:36.330525 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:36.330493 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c13198e7-f244-465c-82ff-199bde275f68-util" (OuterVolumeSpecName: "util") pod "c13198e7-f244-465c-82ff-199bde275f68" (UID: "c13198e7-f244-465c-82ff-199bde275f68"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 09:23:36.427136 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:36.427104 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ce7494cc-693b-4499-bb8a-6646ec33f004-util\") pod \"ce7494cc-693b-4499-bb8a-6646ec33f004\" (UID: \"ce7494cc-693b-4499-bb8a-6646ec33f004\") " Jul 23 09:23:36.427340 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:36.427148 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dwf9t\" (UniqueName: \"kubernetes.io/projected/ce7494cc-693b-4499-bb8a-6646ec33f004-kube-api-access-dwf9t\") pod \"ce7494cc-693b-4499-bb8a-6646ec33f004\" (UID: \"ce7494cc-693b-4499-bb8a-6646ec33f004\") " Jul 23 09:23:36.427340 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:36.427169 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ce7494cc-693b-4499-bb8a-6646ec33f004-bundle\") pod \"ce7494cc-693b-4499-bb8a-6646ec33f004\" (UID: \"ce7494cc-693b-4499-bb8a-6646ec33f004\") " Jul 23 09:23:36.427340 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:36.427301 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g9rwg\" (UniqueName: \"kubernetes.io/projected/51e149f3-3d54-46ad-86c9-01c83eb781f7-kube-api-access-g9rwg\") pod \"51e149f3-3d54-46ad-86c9-01c83eb781f7\" (UID: \"51e149f3-3d54-46ad-86c9-01c83eb781f7\") " Jul 23 09:23:36.427527 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:36.427353 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/51e149f3-3d54-46ad-86c9-01c83eb781f7-bundle\") pod \"51e149f3-3d54-46ad-86c9-01c83eb781f7\" (UID: \"51e149f3-3d54-46ad-86c9-01c83eb781f7\") " Jul 23 09:23:36.427527 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:36.427383 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/51e149f3-3d54-46ad-86c9-01c83eb781f7-util\") pod \"51e149f3-3d54-46ad-86c9-01c83eb781f7\" (UID: \"51e149f3-3d54-46ad-86c9-01c83eb781f7\") " Jul 23 09:23:36.427783 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:36.427760 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2rz8\" (UniqueName: \"kubernetes.io/projected/c13198e7-f244-465c-82ff-199bde275f68-kube-api-access-h2rz8\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:23:36.427867 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:36.427787 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c13198e7-f244-465c-82ff-199bde275f68-bundle\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:23:36.427867 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:36.427802 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c13198e7-f244-465c-82ff-199bde275f68-util\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:23:36.427965 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:36.427885 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51e149f3-3d54-46ad-86c9-01c83eb781f7-bundle" (OuterVolumeSpecName: "bundle") pod "51e149f3-3d54-46ad-86c9-01c83eb781f7" (UID: "51e149f3-3d54-46ad-86c9-01c83eb781f7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 09:23:36.428067 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:36.428037 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce7494cc-693b-4499-bb8a-6646ec33f004-bundle" (OuterVolumeSpecName: "bundle") pod "ce7494cc-693b-4499-bb8a-6646ec33f004" (UID: "ce7494cc-693b-4499-bb8a-6646ec33f004"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 09:23:36.429434 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:36.429401 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce7494cc-693b-4499-bb8a-6646ec33f004-kube-api-access-dwf9t" (OuterVolumeSpecName: "kube-api-access-dwf9t") pod "ce7494cc-693b-4499-bb8a-6646ec33f004" (UID: "ce7494cc-693b-4499-bb8a-6646ec33f004"). InnerVolumeSpecName "kube-api-access-dwf9t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 09:23:36.429768 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:36.429751 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51e149f3-3d54-46ad-86c9-01c83eb781f7-kube-api-access-g9rwg" (OuterVolumeSpecName: "kube-api-access-g9rwg") pod "51e149f3-3d54-46ad-86c9-01c83eb781f7" (UID: "51e149f3-3d54-46ad-86c9-01c83eb781f7"). InnerVolumeSpecName "kube-api-access-g9rwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 09:23:36.432775 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:36.432753 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce7494cc-693b-4499-bb8a-6646ec33f004-util" (OuterVolumeSpecName: "util") pod "ce7494cc-693b-4499-bb8a-6646ec33f004" (UID: "ce7494cc-693b-4499-bb8a-6646ec33f004"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 09:23:36.433164 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:36.433146 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51e149f3-3d54-46ad-86c9-01c83eb781f7-util" (OuterVolumeSpecName: "util") pod "51e149f3-3d54-46ad-86c9-01c83eb781f7" (UID: "51e149f3-3d54-46ad-86c9-01c83eb781f7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 09:23:36.528436 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:36.528397 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/51e149f3-3d54-46ad-86c9-01c83eb781f7-bundle\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:23:36.528436 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:36.528429 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/51e149f3-3d54-46ad-86c9-01c83eb781f7-util\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:23:36.528436 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:36.528438 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ce7494cc-693b-4499-bb8a-6646ec33f004-util\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:23:36.528436 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:36.528446 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwf9t\" (UniqueName: \"kubernetes.io/projected/ce7494cc-693b-4499-bb8a-6646ec33f004-kube-api-access-dwf9t\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:23:36.528436 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:36.528455 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ce7494cc-693b-4499-bb8a-6646ec33f004-bundle\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:23:36.528751 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:36.528464 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9rwg\" (UniqueName: \"kubernetes.io/projected/51e149f3-3d54-46ad-86c9-01c83eb781f7-kube-api-access-g9rwg\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:23:37.070989 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:37.070956 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595h986" Jul 23 09:23:37.071443 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:37.070962 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595h986" event={"ID":"51e149f3-3d54-46ad-86c9-01c83eb781f7","Type":"ContainerDied","Data":"ad8c46fdb170713b8ecef5b15d582ecb62a85afbbd180593b3905c91edaef721"} Jul 23 09:23:37.071443 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:37.071087 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad8c46fdb170713b8ecef5b15d582ecb62a85afbbd180593b3905c91edaef721" Jul 23 09:23:37.072972 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:37.072943 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qrfl7" event={"ID":"ce7494cc-693b-4499-bb8a-6646ec33f004","Type":"ContainerDied","Data":"e52887835e4ea943612c8e25f364244e569af758c4fe62073752b5a62f289add"} Jul 23 09:23:37.072972 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:37.072978 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qrfl7" Jul 23 09:23:37.073321 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:37.072991 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e52887835e4ea943612c8e25f364244e569af758c4fe62073752b5a62f289add" Jul 23 09:23:37.075303 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:37.075275 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cz5gk" event={"ID":"c13198e7-f244-465c-82ff-199bde275f68","Type":"ContainerDied","Data":"0680fb85343abb6f86641571e11099d583107e7256123e6a08377a71803e845a"} Jul 23 09:23:37.075414 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:37.075323 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0680fb85343abb6f86641571e11099d583107e7256123e6a08377a71803e845a" Jul 23 09:23:37.075414 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:37.075283 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cz5gk" Jul 23 09:23:37.203524 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:37.203499 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wrs5q" Jul 23 09:23:37.334503 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:37.334420 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f4dae8b7-1352-4f26-93c6-c5ffd5f8369f-util\") pod \"f4dae8b7-1352-4f26-93c6-c5ffd5f8369f\" (UID: \"f4dae8b7-1352-4f26-93c6-c5ffd5f8369f\") " Jul 23 09:23:37.334644 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:37.334544 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rr6bf\" (UniqueName: \"kubernetes.io/projected/f4dae8b7-1352-4f26-93c6-c5ffd5f8369f-kube-api-access-rr6bf\") pod \"f4dae8b7-1352-4f26-93c6-c5ffd5f8369f\" (UID: \"f4dae8b7-1352-4f26-93c6-c5ffd5f8369f\") " Jul 23 09:23:37.334644 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:37.334575 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f4dae8b7-1352-4f26-93c6-c5ffd5f8369f-bundle\") pod \"f4dae8b7-1352-4f26-93c6-c5ffd5f8369f\" (UID: \"f4dae8b7-1352-4f26-93c6-c5ffd5f8369f\") " Jul 23 09:23:37.335060 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:37.335028 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4dae8b7-1352-4f26-93c6-c5ffd5f8369f-bundle" (OuterVolumeSpecName: "bundle") pod "f4dae8b7-1352-4f26-93c6-c5ffd5f8369f" (UID: "f4dae8b7-1352-4f26-93c6-c5ffd5f8369f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 09:23:37.336734 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:37.336714 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4dae8b7-1352-4f26-93c6-c5ffd5f8369f-kube-api-access-rr6bf" (OuterVolumeSpecName: "kube-api-access-rr6bf") pod "f4dae8b7-1352-4f26-93c6-c5ffd5f8369f" (UID: "f4dae8b7-1352-4f26-93c6-c5ffd5f8369f"). InnerVolumeSpecName "kube-api-access-rr6bf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 09:23:37.414253 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:37.414203 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4dae8b7-1352-4f26-93c6-c5ffd5f8369f-util" (OuterVolumeSpecName: "util") pod "f4dae8b7-1352-4f26-93c6-c5ffd5f8369f" (UID: "f4dae8b7-1352-4f26-93c6-c5ffd5f8369f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 09:23:37.436146 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:37.436120 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rr6bf\" (UniqueName: \"kubernetes.io/projected/f4dae8b7-1352-4f26-93c6-c5ffd5f8369f-kube-api-access-rr6bf\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:23:37.436146 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:37.436146 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f4dae8b7-1352-4f26-93c6-c5ffd5f8369f-bundle\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:23:37.436309 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:37.436155 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f4dae8b7-1352-4f26-93c6-c5ffd5f8369f-util\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:23:38.082751 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:38.082721 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wrs5q" Jul 23 09:23:38.083269 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:38.082750 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wrs5q" event={"ID":"f4dae8b7-1352-4f26-93c6-c5ffd5f8369f","Type":"ContainerDied","Data":"57c4756b29ff4181df42a067fa485bd2067cf0eaf0d4e335d81cfe29b630a1be"} Jul 23 09:23:38.083269 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:38.082795 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57c4756b29ff4181df42a067fa485bd2067cf0eaf0d4e335d81cfe29b630a1be" Jul 23 09:23:43.687974 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:43.687941 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:23:45.026900 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:45.026870 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6f77c8c54b-vc7db" Jul 23 09:23:45.027346 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:45.026948 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6f77c8c54b-vc7db" Jul 23 09:23:45.031778 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:45.031758 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6f77c8c54b-vc7db" Jul 23 09:23:45.120903 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:45.120878 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6f77c8c54b-vc7db" Jul 23 09:23:45.187061 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:45.187027 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7457b69c65-wfpk6"] Jul 23 09:23:45.304705 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:45.304631 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:23:45.415410 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:45.415377 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:23:45.530843 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:45.530807 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:23:47.522476 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:47.522407 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:23:50.302718 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:50.302681 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:23:53.299904 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:53.299871 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:23:53.764919 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:53.764887 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:23:54.880051 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:54.880013 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:23:55.162316 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:55.162227 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:23:55.565975 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:55.565888 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:23:57.192385 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:57.192351 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:23:59.674429 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:59.674401 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:23:59.800080 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:23:59.800047 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:24:00.527954 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:24:00.527906 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:24:00.619577 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:24:00.619532 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:24:03.032399 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:24:03.032366 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:24:03.130981 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:24:03.130954 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:24:04.309156 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:24:04.309123 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:24:04.424796 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:24:04.424760 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:24:07.929874 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:24:07.929843 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:24:08.029128 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:24:08.029098 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:24:10.209675 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:24:10.209640 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7457b69c65-wfpk6" podUID="28242908-b96c-4c04-b469-f36acdf6e5f3" containerName="console" containerID="cri-o://af20805fdca3a6ab696e794382fedadc79e8d4288c2283c3b28034b5e897e96e" gracePeriod=15 Jul 23 09:24:10.463726 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:24:10.463668 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7457b69c65-wfpk6" Jul 23 09:24:10.630758 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:24:10.630732 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/28242908-b96c-4c04-b469-f36acdf6e5f3-console-oauth-config\") pod \"28242908-b96c-4c04-b469-f36acdf6e5f3\" (UID: \"28242908-b96c-4c04-b469-f36acdf6e5f3\") " Jul 23 09:24:10.630928 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:24:10.630803 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/28242908-b96c-4c04-b469-f36acdf6e5f3-console-serving-cert\") pod \"28242908-b96c-4c04-b469-f36acdf6e5f3\" (UID: \"28242908-b96c-4c04-b469-f36acdf6e5f3\") " Jul 23 09:24:10.630928 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:24:10.630849 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28242908-b96c-4c04-b469-f36acdf6e5f3-trusted-ca-bundle\") pod \"28242908-b96c-4c04-b469-f36acdf6e5f3\" (UID: \"28242908-b96c-4c04-b469-f36acdf6e5f3\") " Jul 23 09:24:10.630928 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:24:10.630898 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/28242908-b96c-4c04-b469-f36acdf6e5f3-service-ca\") pod \"28242908-b96c-4c04-b469-f36acdf6e5f3\" (UID: \"28242908-b96c-4c04-b469-f36acdf6e5f3\") " Jul 23 09:24:10.630928 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:24:10.630926 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/28242908-b96c-4c04-b469-f36acdf6e5f3-console-config\") pod \"28242908-b96c-4c04-b469-f36acdf6e5f3\" (UID: \"28242908-b96c-4c04-b469-f36acdf6e5f3\") " Jul 23 09:24:10.631146 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:24:10.630971 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vgj8k\" (UniqueName: \"kubernetes.io/projected/28242908-b96c-4c04-b469-f36acdf6e5f3-kube-api-access-vgj8k\") pod \"28242908-b96c-4c04-b469-f36acdf6e5f3\" (UID: \"28242908-b96c-4c04-b469-f36acdf6e5f3\") " Jul 23 09:24:10.631146 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:24:10.630994 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/28242908-b96c-4c04-b469-f36acdf6e5f3-oauth-serving-cert\") pod \"28242908-b96c-4c04-b469-f36acdf6e5f3\" (UID: \"28242908-b96c-4c04-b469-f36acdf6e5f3\") " Jul 23 09:24:10.631356 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:24:10.631315 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28242908-b96c-4c04-b469-f36acdf6e5f3-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "28242908-b96c-4c04-b469-f36acdf6e5f3" (UID: "28242908-b96c-4c04-b469-f36acdf6e5f3"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 09:24:10.631462 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:24:10.631400 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28242908-b96c-4c04-b469-f36acdf6e5f3-service-ca" (OuterVolumeSpecName: "service-ca") pod "28242908-b96c-4c04-b469-f36acdf6e5f3" (UID: "28242908-b96c-4c04-b469-f36acdf6e5f3"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 09:24:10.631635 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:24:10.631610 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28242908-b96c-4c04-b469-f36acdf6e5f3-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "28242908-b96c-4c04-b469-f36acdf6e5f3" (UID: "28242908-b96c-4c04-b469-f36acdf6e5f3"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 09:24:10.631685 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:24:10.631641 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28242908-b96c-4c04-b469-f36acdf6e5f3-console-config" (OuterVolumeSpecName: "console-config") pod "28242908-b96c-4c04-b469-f36acdf6e5f3" (UID: "28242908-b96c-4c04-b469-f36acdf6e5f3"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 09:24:10.633275 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:24:10.633228 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28242908-b96c-4c04-b469-f36acdf6e5f3-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "28242908-b96c-4c04-b469-f36acdf6e5f3" (UID: "28242908-b96c-4c04-b469-f36acdf6e5f3"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 09:24:10.633275 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:24:10.633257 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28242908-b96c-4c04-b469-f36acdf6e5f3-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "28242908-b96c-4c04-b469-f36acdf6e5f3" (UID: "28242908-b96c-4c04-b469-f36acdf6e5f3"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 09:24:10.633422 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:24:10.633311 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28242908-b96c-4c04-b469-f36acdf6e5f3-kube-api-access-vgj8k" (OuterVolumeSpecName: "kube-api-access-vgj8k") pod "28242908-b96c-4c04-b469-f36acdf6e5f3" (UID: "28242908-b96c-4c04-b469-f36acdf6e5f3"). InnerVolumeSpecName "kube-api-access-vgj8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 09:24:10.732564 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:24:10.732479 2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/28242908-b96c-4c04-b469-f36acdf6e5f3-console-oauth-config\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:24:10.732564 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:24:10.732510 2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/28242908-b96c-4c04-b469-f36acdf6e5f3-console-serving-cert\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:24:10.732564 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:24:10.732521 2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28242908-b96c-4c04-b469-f36acdf6e5f3-trusted-ca-bundle\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:24:10.732564 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:24:10.732531 2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/28242908-b96c-4c04-b469-f36acdf6e5f3-service-ca\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:24:10.732564 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:24:10.732539 2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/28242908-b96c-4c04-b469-f36acdf6e5f3-console-config\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:24:10.732564 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:24:10.732547 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgj8k\" (UniqueName: \"kubernetes.io/projected/28242908-b96c-4c04-b469-f36acdf6e5f3-kube-api-access-vgj8k\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:24:10.732564 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:24:10.732555 2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/28242908-b96c-4c04-b469-f36acdf6e5f3-oauth-serving-cert\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:24:11.249225 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:24:11.249128 2576 generic.go:356] "Generic (PLEG): container finished" podID="28242908-b96c-4c04-b469-f36acdf6e5f3" containerID="af20805fdca3a6ab696e794382fedadc79e8d4288c2283c3b28034b5e897e96e" exitCode=0 Jul 23 09:24:11.249225 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:24:11.249167 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7457b69c65-wfpk6" Jul 23 09:24:11.249704 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:24:11.249222 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7457b69c65-wfpk6" event={"ID":"28242908-b96c-4c04-b469-f36acdf6e5f3","Type":"ContainerDied","Data":"af20805fdca3a6ab696e794382fedadc79e8d4288c2283c3b28034b5e897e96e"} Jul 23 09:24:11.249704 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:24:11.249283 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7457b69c65-wfpk6" event={"ID":"28242908-b96c-4c04-b469-f36acdf6e5f3","Type":"ContainerDied","Data":"7819cb4af7d7737372d66eb30b3aa551833f8ee5063616b5699c7bf3f6ba31c6"} Jul 23 09:24:11.249704 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:24:11.249305 2576 scope.go:117] "RemoveContainer" containerID="af20805fdca3a6ab696e794382fedadc79e8d4288c2283c3b28034b5e897e96e" Jul 23 09:24:11.260464 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:24:11.260441 2576 scope.go:117] "RemoveContainer" containerID="af20805fdca3a6ab696e794382fedadc79e8d4288c2283c3b28034b5e897e96e" Jul 23 09:24:11.261011 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:24:11.260992 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af20805fdca3a6ab696e794382fedadc79e8d4288c2283c3b28034b5e897e96e\": container with ID starting with af20805fdca3a6ab696e794382fedadc79e8d4288c2283c3b28034b5e897e96e not found: ID does not exist" containerID="af20805fdca3a6ab696e794382fedadc79e8d4288c2283c3b28034b5e897e96e" Jul 23 09:24:11.261077 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:24:11.261022 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af20805fdca3a6ab696e794382fedadc79e8d4288c2283c3b28034b5e897e96e"} err="failed to get container status \"af20805fdca3a6ab696e794382fedadc79e8d4288c2283c3b28034b5e897e96e\": rpc error: code = NotFound desc = could not find container \"af20805fdca3a6ab696e794382fedadc79e8d4288c2283c3b28034b5e897e96e\": container with ID starting with af20805fdca3a6ab696e794382fedadc79e8d4288c2283c3b28034b5e897e96e not found: ID does not exist" Jul 23 09:24:11.269147 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:24:11.269122 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7457b69c65-wfpk6"] Jul 23 09:24:11.271561 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:24:11.271543 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7457b69c65-wfpk6"] Jul 23 09:24:12.985630 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:24:12.985590 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28242908-b96c-4c04-b469-f36acdf6e5f3" path="/var/lib/kubelet/pods/28242908-b96c-4c04-b469-f36acdf6e5f3/volumes" Jul 23 09:24:15.891749 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:24:15.891677 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:24:15.996358 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:24:15.996327 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:24:16.903613 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:24:16.903583 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:24:16.993025 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:24:16.992995 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:24:18.363564 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:24:18.363529 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:24:18.464830 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:24:18.464803 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:24:21.188373 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:24:21.188336 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:24:21.312787 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:24:21.312753 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:24:24.827411 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:24:24.827376 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:24:24.930399 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:24:24.930369 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:24:25.777234 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:24:25.777202 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:24:31.253262 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:24:31.253210 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:24:33.626693 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:24:33.626659 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:24:33.725728 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:24:33.725696 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:24:34.830415 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:24:34.830384 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:24:36.315112 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:24:36.315076 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:24:36.416310 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:24:36.416277 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:24:39.639647 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:24:39.639614 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:24:39.747061 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:24:39.747006 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:24:43.064426 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:24:43.064395 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:24:43.182797 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:24:43.182766 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:24:47.668391 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:24:47.668359 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:24:52.610672 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:24:52.610635 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:24:52.710447 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:24:52.710414 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:24:53.218614 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:24:53.218577 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:24:53.307010 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:24:53.306977 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:24:53.551203 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:24:53.551121 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:25:04.700573 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:04.700540 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:25:04.805968 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:04.805934 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:25:04.907433 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:04.907403 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:25:05.103968 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:05.103894 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:25:05.104226 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:05.104214 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51e149f3-3d54-46ad-86c9-01c83eb781f7" containerName="util" Jul 23 09:25:05.104291 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:05.104227 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="51e149f3-3d54-46ad-86c9-01c83eb781f7" containerName="util" Jul 23 09:25:05.104291 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:05.104235 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce7494cc-693b-4499-bb8a-6646ec33f004" containerName="extract" Jul 23 09:25:05.104291 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:05.104254 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce7494cc-693b-4499-bb8a-6646ec33f004" containerName="extract" Jul 23 09:25:05.104291 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:05.104261 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4dae8b7-1352-4f26-93c6-c5ffd5f8369f" containerName="extract" Jul 23 09:25:05.104291 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:05.104267 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4dae8b7-1352-4f26-93c6-c5ffd5f8369f" containerName="extract" Jul 23 09:25:05.104291 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:05.104278 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51e149f3-3d54-46ad-86c9-01c83eb781f7" containerName="extract" Jul 23 09:25:05.104291 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:05.104283 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="51e149f3-3d54-46ad-86c9-01c83eb781f7" containerName="extract" Jul 23 09:25:05.104291 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:05.104289 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce7494cc-693b-4499-bb8a-6646ec33f004" containerName="pull" Jul 23 09:25:05.104291 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:05.104294 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce7494cc-693b-4499-bb8a-6646ec33f004" containerName="pull" Jul 23 09:25:05.104562 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:05.104304 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c13198e7-f244-465c-82ff-199bde275f68" containerName="extract" Jul 23 09:25:05.104562 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:05.104309 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c13198e7-f244-465c-82ff-199bde275f68" containerName="extract" Jul 23 09:25:05.104562 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:05.104316 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4dae8b7-1352-4f26-93c6-c5ffd5f8369f" containerName="util" Jul 23 09:25:05.104562 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:05.104321 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4dae8b7-1352-4f26-93c6-c5ffd5f8369f" containerName="util" Jul 23 09:25:05.104562 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:05.104328 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce7494cc-693b-4499-bb8a-6646ec33f004" containerName="util" Jul 23 09:25:05.104562 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:05.104333 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce7494cc-693b-4499-bb8a-6646ec33f004" containerName="util" Jul 23 09:25:05.104562 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:05.104338 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c13198e7-f244-465c-82ff-199bde275f68" containerName="util" Jul 23 09:25:05.104562 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:05.104343 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c13198e7-f244-465c-82ff-199bde275f68" containerName="util" Jul 23 09:25:05.104562 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:05.104349 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51e149f3-3d54-46ad-86c9-01c83eb781f7" containerName="pull" Jul 23 09:25:05.104562 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:05.104353 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="51e149f3-3d54-46ad-86c9-01c83eb781f7" containerName="pull" Jul 23 09:25:05.104562 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:05.104359 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4dae8b7-1352-4f26-93c6-c5ffd5f8369f" containerName="pull" Jul 23 09:25:05.104562 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:05.104365 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4dae8b7-1352-4f26-93c6-c5ffd5f8369f" containerName="pull" Jul 23 09:25:05.104562 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:05.104374 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c13198e7-f244-465c-82ff-199bde275f68" containerName="pull" Jul 23 09:25:05.104562 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:05.104378 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c13198e7-f244-465c-82ff-199bde275f68" containerName="pull" Jul 23 09:25:05.104562 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:05.104390 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28242908-b96c-4c04-b469-f36acdf6e5f3" containerName="console" Jul 23 09:25:05.104562 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:05.104395 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="28242908-b96c-4c04-b469-f36acdf6e5f3" containerName="console" Jul 23 09:25:05.104562 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:05.104451 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="28242908-b96c-4c04-b469-f36acdf6e5f3" containerName="console" Jul 23 09:25:05.104562 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:05.104458 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c13198e7-f244-465c-82ff-199bde275f68" containerName="extract" Jul 23 09:25:05.104562 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:05.104465 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4dae8b7-1352-4f26-93c6-c5ffd5f8369f" containerName="extract" Jul 23 09:25:05.104562 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:05.104473 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="51e149f3-3d54-46ad-86c9-01c83eb781f7" containerName="extract" Jul 23 09:25:05.104562 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:05.104479 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce7494cc-693b-4499-bb8a-6646ec33f004" containerName="extract" Jul 23 09:25:05.108609 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:05.108593 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-q5mq5" Jul 23 09:25:05.110861 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:05.110835 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 23 09:25:05.111517 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:05.111500 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-g68sf\"" Jul 23 09:25:05.111615 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:05.111544 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 23 09:25:05.111615 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:05.111571 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 23 09:25:05.116424 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:05.116404 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:25:05.148981 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:05.148947 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:25:05.195535 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:05.195502 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfmsk\" (UniqueName: \"kubernetes.io/projected/07acbf04-6129-4a76-ac35-87e8e8f6c19c-kube-api-access-rfmsk\") pod \"limitador-limitador-54d6d988d5-q5mq5\" (UID: \"07acbf04-6129-4a76-ac35-87e8e8f6c19c\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-q5mq5" Jul 23 09:25:05.195535 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:05.195539 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/07acbf04-6129-4a76-ac35-87e8e8f6c19c-config-file\") pod \"limitador-limitador-54d6d988d5-q5mq5\" (UID: \"07acbf04-6129-4a76-ac35-87e8e8f6c19c\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-q5mq5" Jul 23 09:25:05.297055 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:05.297017 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfmsk\" (UniqueName: \"kubernetes.io/projected/07acbf04-6129-4a76-ac35-87e8e8f6c19c-kube-api-access-rfmsk\") pod \"limitador-limitador-54d6d988d5-q5mq5\" (UID: \"07acbf04-6129-4a76-ac35-87e8e8f6c19c\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-q5mq5" Jul 23 09:25:05.297055 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:05.297056 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/07acbf04-6129-4a76-ac35-87e8e8f6c19c-config-file\") pod \"limitador-limitador-54d6d988d5-q5mq5\" (UID: \"07acbf04-6129-4a76-ac35-87e8e8f6c19c\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-q5mq5" Jul 23 09:25:05.297731 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:05.297710 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/07acbf04-6129-4a76-ac35-87e8e8f6c19c-config-file\") pod \"limitador-limitador-54d6d988d5-q5mq5\" (UID: \"07acbf04-6129-4a76-ac35-87e8e8f6c19c\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-q5mq5" Jul 23 09:25:05.305448 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:05.305422 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfmsk\" (UniqueName: \"kubernetes.io/projected/07acbf04-6129-4a76-ac35-87e8e8f6c19c-kube-api-access-rfmsk\") pod \"limitador-limitador-54d6d988d5-q5mq5\" (UID: \"07acbf04-6129-4a76-ac35-87e8e8f6c19c\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-q5mq5" Jul 23 09:25:05.419992 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:05.419960 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-q5mq5" Jul 23 09:25:05.508960 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:05.508930 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-pkqhm"] Jul 23 09:25:05.513149 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:05.513128 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-pkqhm" Jul 23 09:25:05.516067 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:05.516043 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-rsfwg\"" Jul 23 09:25:05.522048 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:05.521878 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-pkqhm"] Jul 23 09:25:05.542410 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:05.542385 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:25:05.560005 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:05.559976 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:25:05.561789 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:25:05.561766 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07acbf04_6129_4a76_ac35_87e8e8f6c19c.slice/crio-66fed43e368a757d41ab7cef88a76f6f485f10a47d821ac2fe984713d11189b4 WatchSource:0}: Error finding container 66fed43e368a757d41ab7cef88a76f6f485f10a47d821ac2fe984713d11189b4: Status 404 returned error can't find the container with id 66fed43e368a757d41ab7cef88a76f6f485f10a47d821ac2fe984713d11189b4 Jul 23 09:25:05.599882 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:05.599854 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrhsj\" (UniqueName: \"kubernetes.io/projected/ae2f39dd-b02e-4d0b-a997-cd315b693813-kube-api-access-zrhsj\") pod \"authorino-58cd955675-pkqhm\" (UID: \"ae2f39dd-b02e-4d0b-a997-cd315b693813\") " pod="kuadrant-system/authorino-58cd955675-pkqhm" Jul 23 09:25:05.625784 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:05.625759 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:25:05.686109 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:05.686023 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-qh222"] Jul 23 09:25:05.689447 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:05.689430 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-qh222" Jul 23 09:25:05.699807 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:05.699784 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-qh222"] Jul 23 09:25:05.700990 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:05.700965 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrhsj\" (UniqueName: \"kubernetes.io/projected/ae2f39dd-b02e-4d0b-a997-cd315b693813-kube-api-access-zrhsj\") pod \"authorino-58cd955675-pkqhm\" (UID: \"ae2f39dd-b02e-4d0b-a997-cd315b693813\") " pod="kuadrant-system/authorino-58cd955675-pkqhm" Jul 23 09:25:05.713057 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:05.713035 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrhsj\" (UniqueName: \"kubernetes.io/projected/ae2f39dd-b02e-4d0b-a997-cd315b693813-kube-api-access-zrhsj\") pod \"authorino-58cd955675-pkqhm\" (UID: \"ae2f39dd-b02e-4d0b-a997-cd315b693813\") " pod="kuadrant-system/authorino-58cd955675-pkqhm" Jul 23 09:25:05.802169 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:05.802128 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqdrf\" (UniqueName: \"kubernetes.io/projected/4e989f09-ff5e-4c9e-bd38-f0b29b08563c-kube-api-access-wqdrf\") pod \"authorino-75d556bd77-qh222\" (UID: \"4e989f09-ff5e-4c9e-bd38-f0b29b08563c\") " pod="kuadrant-system/authorino-75d556bd77-qh222" Jul 23 09:25:05.826992 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:05.826962 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-pkqhm" Jul 23 09:25:05.903151 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:05.903116 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqdrf\" (UniqueName: \"kubernetes.io/projected/4e989f09-ff5e-4c9e-bd38-f0b29b08563c-kube-api-access-wqdrf\") pod \"authorino-75d556bd77-qh222\" (UID: \"4e989f09-ff5e-4c9e-bd38-f0b29b08563c\") " pod="kuadrant-system/authorino-75d556bd77-qh222" Jul 23 09:25:05.911212 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:05.911185 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqdrf\" (UniqueName: \"kubernetes.io/projected/4e989f09-ff5e-4c9e-bd38-f0b29b08563c-kube-api-access-wqdrf\") pod \"authorino-75d556bd77-qh222\" (UID: \"4e989f09-ff5e-4c9e-bd38-f0b29b08563c\") " pod="kuadrant-system/authorino-75d556bd77-qh222" Jul 23 09:25:05.953846 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:05.953769 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-pkqhm"] Jul 23 09:25:05.956604 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:25:05.956581 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podae2f39dd_b02e_4d0b_a997_cd315b693813.slice/crio-b938dc449023211974a6e0be968f7ec6afb3781c01c9297268d65401e3e5471a WatchSource:0}: Error finding container b938dc449023211974a6e0be968f7ec6afb3781c01c9297268d65401e3e5471a: Status 404 returned error can't find the container with id b938dc449023211974a6e0be968f7ec6afb3781c01c9297268d65401e3e5471a Jul 23 09:25:06.000281 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:06.000257 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-qh222" Jul 23 09:25:06.130705 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:06.130678 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-qh222"] Jul 23 09:25:06.131918 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:25:06.131884 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4e989f09_ff5e_4c9e_bd38_f0b29b08563c.slice/crio-5e8c76a25a20f3d844d85a21b671cf65ff2bd9135690bddfa8301d5e2e9cc8f1 WatchSource:0}: Error finding container 5e8c76a25a20f3d844d85a21b671cf65ff2bd9135690bddfa8301d5e2e9cc8f1: Status 404 returned error can't find the container with id 5e8c76a25a20f3d844d85a21b671cf65ff2bd9135690bddfa8301d5e2e9cc8f1 Jul 23 09:25:06.540425 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:06.540323 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-q5mq5" event={"ID":"07acbf04-6129-4a76-ac35-87e8e8f6c19c","Type":"ContainerStarted","Data":"66fed43e368a757d41ab7cef88a76f6f485f10a47d821ac2fe984713d11189b4"} Jul 23 09:25:06.542146 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:06.542115 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-qh222" event={"ID":"4e989f09-ff5e-4c9e-bd38-f0b29b08563c","Type":"ContainerStarted","Data":"5e8c76a25a20f3d844d85a21b671cf65ff2bd9135690bddfa8301d5e2e9cc8f1"} Jul 23 09:25:06.544686 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:06.544557 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-pkqhm" event={"ID":"ae2f39dd-b02e-4d0b-a997-cd315b693813","Type":"ContainerStarted","Data":"b938dc449023211974a6e0be968f7ec6afb3781c01c9297268d65401e3e5471a"} Jul 23 09:25:11.580951 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:11.580909 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-q5mq5" event={"ID":"07acbf04-6129-4a76-ac35-87e8e8f6c19c","Type":"ContainerStarted","Data":"a1a5cda2619a8c2780186508c5b6fb343c2b62b0c32e3e73a75a2e6986104ffd"} Jul 23 09:25:11.581428 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:11.580979 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-q5mq5" Jul 23 09:25:11.582394 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:11.582365 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-qh222" event={"ID":"4e989f09-ff5e-4c9e-bd38-f0b29b08563c","Type":"ContainerStarted","Data":"1f6c398514ec52e136b51cc28abc40d7f8dc4ef2ae3ebd53bd303fc0482db011"} Jul 23 09:25:11.583607 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:11.583581 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-pkqhm" event={"ID":"ae2f39dd-b02e-4d0b-a997-cd315b693813","Type":"ContainerStarted","Data":"4ffaf5c35975f3bbe4fa94e692bcefa9a7aea6714cca17f0bc6a312ab504d3c7"} Jul 23 09:25:11.603477 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:11.603433 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-q5mq5" podStartSLOduration=1.3607042360000001 podStartE2EDuration="6.60341973s" totalImagesPullingTime="5.242715494s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:25:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:25:11.600578469 +0000 UTC m=+500.139857045" watchObservedRunningTime="2026-07-23 09:25:11.60341973 +0000 UTC m=+500.142698325" Jul 23 09:25:11.617214 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:11.617164 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-qh222" podStartSLOduration=1.9569799620000001 podStartE2EDuration="6.617146266s" totalImagesPullingTime="4.660166304s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:25:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:25:11.617052617 +0000 UTC m=+500.156331195" watchObservedRunningTime="2026-07-23 09:25:11.617146266 +0000 UTC m=+500.156424843" Jul 23 09:25:11.638406 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:11.638355 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-pkqhm" podStartSLOduration=1.8031611330000001 podStartE2EDuration="6.638341082s" totalImagesPullingTime="4.835179949s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:25:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:25:11.637442737 +0000 UTC m=+500.176721315" watchObservedRunningTime="2026-07-23 09:25:11.638341082 +0000 UTC m=+500.177619679" Jul 23 09:25:11.674964 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:11.674935 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-pkqhm"] Jul 23 09:25:13.593224 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:13.593178 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-pkqhm" podUID="ae2f39dd-b02e-4d0b-a997-cd315b693813" containerName="authorino" containerID="cri-o://4ffaf5c35975f3bbe4fa94e692bcefa9a7aea6714cca17f0bc6a312ab504d3c7" gracePeriod=30 Jul 23 09:25:13.837676 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:13.837650 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-pkqhm" Jul 23 09:25:13.981531 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:13.981490 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zrhsj\" (UniqueName: \"kubernetes.io/projected/ae2f39dd-b02e-4d0b-a997-cd315b693813-kube-api-access-zrhsj\") pod \"ae2f39dd-b02e-4d0b-a997-cd315b693813\" (UID: \"ae2f39dd-b02e-4d0b-a997-cd315b693813\") " Jul 23 09:25:13.983538 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:13.983507 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae2f39dd-b02e-4d0b-a997-cd315b693813-kube-api-access-zrhsj" (OuterVolumeSpecName: "kube-api-access-zrhsj") pod "ae2f39dd-b02e-4d0b-a997-cd315b693813" (UID: "ae2f39dd-b02e-4d0b-a997-cd315b693813"). InnerVolumeSpecName "kube-api-access-zrhsj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 09:25:14.082779 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:14.082740 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrhsj\" (UniqueName: \"kubernetes.io/projected/ae2f39dd-b02e-4d0b-a997-cd315b693813-kube-api-access-zrhsj\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:25:14.598802 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:14.598771 2576 generic.go:356] "Generic (PLEG): container finished" podID="ae2f39dd-b02e-4d0b-a997-cd315b693813" containerID="4ffaf5c35975f3bbe4fa94e692bcefa9a7aea6714cca17f0bc6a312ab504d3c7" exitCode=0 Jul 23 09:25:14.599203 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:14.598816 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-pkqhm" event={"ID":"ae2f39dd-b02e-4d0b-a997-cd315b693813","Type":"ContainerDied","Data":"4ffaf5c35975f3bbe4fa94e692bcefa9a7aea6714cca17f0bc6a312ab504d3c7"} Jul 23 09:25:14.599203 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:14.598854 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-pkqhm" event={"ID":"ae2f39dd-b02e-4d0b-a997-cd315b693813","Type":"ContainerDied","Data":"b938dc449023211974a6e0be968f7ec6afb3781c01c9297268d65401e3e5471a"} Jul 23 09:25:14.599203 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:14.598872 2576 scope.go:117] "RemoveContainer" containerID="4ffaf5c35975f3bbe4fa94e692bcefa9a7aea6714cca17f0bc6a312ab504d3c7" Jul 23 09:25:14.599203 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:14.598877 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-pkqhm" Jul 23 09:25:14.608789 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:14.608769 2576 scope.go:117] "RemoveContainer" containerID="4ffaf5c35975f3bbe4fa94e692bcefa9a7aea6714cca17f0bc6a312ab504d3c7" Jul 23 09:25:14.609028 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:25:14.609010 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ffaf5c35975f3bbe4fa94e692bcefa9a7aea6714cca17f0bc6a312ab504d3c7\": container with ID starting with 4ffaf5c35975f3bbe4fa94e692bcefa9a7aea6714cca17f0bc6a312ab504d3c7 not found: ID does not exist" containerID="4ffaf5c35975f3bbe4fa94e692bcefa9a7aea6714cca17f0bc6a312ab504d3c7" Jul 23 09:25:14.609070 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:14.609037 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ffaf5c35975f3bbe4fa94e692bcefa9a7aea6714cca17f0bc6a312ab504d3c7"} err="failed to get container status \"4ffaf5c35975f3bbe4fa94e692bcefa9a7aea6714cca17f0bc6a312ab504d3c7\": rpc error: code = NotFound desc = could not find container \"4ffaf5c35975f3bbe4fa94e692bcefa9a7aea6714cca17f0bc6a312ab504d3c7\": container with ID starting with 4ffaf5c35975f3bbe4fa94e692bcefa9a7aea6714cca17f0bc6a312ab504d3c7 not found: ID does not exist" Jul 23 09:25:14.620930 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:14.620909 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-pkqhm"] Jul 23 09:25:14.624837 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:14.624815 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-pkqhm"] Jul 23 09:25:14.986915 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:14.986883 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae2f39dd-b02e-4d0b-a997-cd315b693813" path="/var/lib/kubelet/pods/ae2f39dd-b02e-4d0b-a997-cd315b693813/volumes" Jul 23 09:25:22.590321 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:22.590285 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-q5mq5" Jul 23 09:25:27.108341 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:27.108303 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:25:27.282121 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:27.282085 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:25:27.495518 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:27.495492 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:25:39.544603 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:39.544567 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:25:39.797097 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:39.797007 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-99zvh"] Jul 23 09:25:39.797381 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:39.797368 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae2f39dd-b02e-4d0b-a997-cd315b693813" containerName="authorino" Jul 23 09:25:39.797482 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:39.797383 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae2f39dd-b02e-4d0b-a997-cd315b693813" containerName="authorino" Jul 23 09:25:39.797482 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:39.797459 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae2f39dd-b02e-4d0b-a997-cd315b693813" containerName="authorino" Jul 23 09:25:39.805610 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:39.805591 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-99zvh" Jul 23 09:25:39.809811 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:39.809787 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-99zvh"] Jul 23 09:25:39.833570 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:39.833518 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:25:39.901626 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:39.901597 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfnr9\" (UniqueName: \"kubernetes.io/projected/56a971b6-10af-48b1-b1cd-25a287da5888-kube-api-access-rfnr9\") pod \"authorino-5d4b67c446-99zvh\" (UID: \"56a971b6-10af-48b1-b1cd-25a287da5888\") " pod="kuadrant-system/authorino-5d4b67c446-99zvh" Jul 23 09:25:40.002950 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:40.002918 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfnr9\" (UniqueName: \"kubernetes.io/projected/56a971b6-10af-48b1-b1cd-25a287da5888-kube-api-access-rfnr9\") pod \"authorino-5d4b67c446-99zvh\" (UID: \"56a971b6-10af-48b1-b1cd-25a287da5888\") " pod="kuadrant-system/authorino-5d4b67c446-99zvh" Jul 23 09:25:40.007087 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:40.007062 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-99zvh"] Jul 23 09:25:40.007315 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:25:40.007288 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-rfnr9], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-99zvh" podUID="56a971b6-10af-48b1-b1cd-25a287da5888" Jul 23 09:25:40.015270 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:40.015252 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfnr9\" (UniqueName: \"kubernetes.io/projected/56a971b6-10af-48b1-b1cd-25a287da5888-kube-api-access-rfnr9\") pod \"authorino-5d4b67c446-99zvh\" (UID: \"56a971b6-10af-48b1-b1cd-25a287da5888\") " pod="kuadrant-system/authorino-5d4b67c446-99zvh" Jul 23 09:25:40.035728 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:40.035704 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-76877994-4gp5j"] Jul 23 09:25:40.039740 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:40.039725 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-4gp5j" Jul 23 09:25:40.042228 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:40.042204 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 23 09:25:40.046706 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:40.046683 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-76877994-4gp5j"] Jul 23 09:25:40.073376 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:40.073322 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-4gp5j"] Jul 23 09:25:40.073582 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:25:40.073562 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-dw6sp tls-cert], unattached volumes=[], failed to process volumes=[kube-api-access-dw6sp tls-cert]: context canceled" pod="kuadrant-system/authorino-76877994-4gp5j" podUID="8b7c382c-9c2e-4f5b-8e26-358020d64381" Jul 23 09:25:40.102954 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:40.102923 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7c94f6ccc-7hxfj"] Jul 23 09:25:40.104170 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:40.104147 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw6sp\" (UniqueName: \"kubernetes.io/projected/8b7c382c-9c2e-4f5b-8e26-358020d64381-kube-api-access-dw6sp\") pod \"authorino-76877994-4gp5j\" (UID: \"8b7c382c-9c2e-4f5b-8e26-358020d64381\") " pod="kuadrant-system/authorino-76877994-4gp5j" Jul 23 09:25:40.104281 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:40.104196 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8b7c382c-9c2e-4f5b-8e26-358020d64381-tls-cert\") pod \"authorino-76877994-4gp5j\" (UID: \"8b7c382c-9c2e-4f5b-8e26-358020d64381\") " pod="kuadrant-system/authorino-76877994-4gp5j" Jul 23 09:25:40.106831 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:40.106814 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c94f6ccc-7hxfj" Jul 23 09:25:40.113372 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:40.113328 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7c94f6ccc-7hxfj"] Jul 23 09:25:40.204672 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:40.204636 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw6sp\" (UniqueName: \"kubernetes.io/projected/8b7c382c-9c2e-4f5b-8e26-358020d64381-kube-api-access-dw6sp\") pod \"authorino-76877994-4gp5j\" (UID: \"8b7c382c-9c2e-4f5b-8e26-358020d64381\") " pod="kuadrant-system/authorino-76877994-4gp5j" Jul 23 09:25:40.204856 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:40.204697 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8b7c382c-9c2e-4f5b-8e26-358020d64381-tls-cert\") pod \"authorino-76877994-4gp5j\" (UID: \"8b7c382c-9c2e-4f5b-8e26-358020d64381\") " pod="kuadrant-system/authorino-76877994-4gp5j" Jul 23 09:25:40.204856 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:40.204740 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c31805e5-014e-43e6-ac72-a7259adaaf52-tls-cert\") pod \"authorino-7c94f6ccc-7hxfj\" (UID: \"c31805e5-014e-43e6-ac72-a7259adaaf52\") " pod="kuadrant-system/authorino-7c94f6ccc-7hxfj" Jul 23 09:25:40.204856 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:40.204757 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlbvj\" (UniqueName: \"kubernetes.io/projected/c31805e5-014e-43e6-ac72-a7259adaaf52-kube-api-access-tlbvj\") pod \"authorino-7c94f6ccc-7hxfj\" (UID: \"c31805e5-014e-43e6-ac72-a7259adaaf52\") " pod="kuadrant-system/authorino-7c94f6ccc-7hxfj" Jul 23 09:25:40.207254 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:40.207221 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8b7c382c-9c2e-4f5b-8e26-358020d64381-tls-cert\") pod \"authorino-76877994-4gp5j\" (UID: \"8b7c382c-9c2e-4f5b-8e26-358020d64381\") " pod="kuadrant-system/authorino-76877994-4gp5j" Jul 23 09:25:40.213033 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:40.213005 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw6sp\" (UniqueName: \"kubernetes.io/projected/8b7c382c-9c2e-4f5b-8e26-358020d64381-kube-api-access-dw6sp\") pod \"authorino-76877994-4gp5j\" (UID: \"8b7c382c-9c2e-4f5b-8e26-358020d64381\") " pod="kuadrant-system/authorino-76877994-4gp5j" Jul 23 09:25:40.305568 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:40.305533 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c31805e5-014e-43e6-ac72-a7259adaaf52-tls-cert\") pod \"authorino-7c94f6ccc-7hxfj\" (UID: \"c31805e5-014e-43e6-ac72-a7259adaaf52\") " pod="kuadrant-system/authorino-7c94f6ccc-7hxfj" Jul 23 09:25:40.305568 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:40.305569 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlbvj\" (UniqueName: \"kubernetes.io/projected/c31805e5-014e-43e6-ac72-a7259adaaf52-kube-api-access-tlbvj\") pod \"authorino-7c94f6ccc-7hxfj\" (UID: \"c31805e5-014e-43e6-ac72-a7259adaaf52\") " pod="kuadrant-system/authorino-7c94f6ccc-7hxfj" Jul 23 09:25:40.307896 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:40.307869 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c31805e5-014e-43e6-ac72-a7259adaaf52-tls-cert\") pod \"authorino-7c94f6ccc-7hxfj\" (UID: \"c31805e5-014e-43e6-ac72-a7259adaaf52\") " pod="kuadrant-system/authorino-7c94f6ccc-7hxfj" Jul 23 09:25:40.313727 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:40.313707 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlbvj\" (UniqueName: \"kubernetes.io/projected/c31805e5-014e-43e6-ac72-a7259adaaf52-kube-api-access-tlbvj\") pod \"authorino-7c94f6ccc-7hxfj\" (UID: \"c31805e5-014e-43e6-ac72-a7259adaaf52\") " pod="kuadrant-system/authorino-7c94f6ccc-7hxfj" Jul 23 09:25:40.415885 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:40.415857 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c94f6ccc-7hxfj" Jul 23 09:25:40.545699 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:40.545673 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7c94f6ccc-7hxfj"] Jul 23 09:25:40.547524 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:25:40.547502 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc31805e5_014e_43e6_ac72_a7259adaaf52.slice/crio-4710288c0094f100993a113b0ccc717146852d346e434c08e2fad06b39e9126c WatchSource:0}: Error finding container 4710288c0094f100993a113b0ccc717146852d346e434c08e2fad06b39e9126c: Status 404 returned error can't find the container with id 4710288c0094f100993a113b0ccc717146852d346e434c08e2fad06b39e9126c Jul 23 09:25:40.733075 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:40.732990 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c94f6ccc-7hxfj" event={"ID":"c31805e5-014e-43e6-ac72-a7259adaaf52","Type":"ContainerStarted","Data":"4710288c0094f100993a113b0ccc717146852d346e434c08e2fad06b39e9126c"} Jul 23 09:25:40.733075 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:40.733042 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-99zvh" Jul 23 09:25:40.733075 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:40.733048 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-4gp5j" Jul 23 09:25:40.742341 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:40.742321 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-99zvh" Jul 23 09:25:40.744685 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:40.744670 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-4gp5j" Jul 23 09:25:40.809054 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:40.809022 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rfnr9\" (UniqueName: \"kubernetes.io/projected/56a971b6-10af-48b1-b1cd-25a287da5888-kube-api-access-rfnr9\") pod \"56a971b6-10af-48b1-b1cd-25a287da5888\" (UID: \"56a971b6-10af-48b1-b1cd-25a287da5888\") " Jul 23 09:25:40.809211 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:40.809145 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8b7c382c-9c2e-4f5b-8e26-358020d64381-tls-cert\") pod \"8b7c382c-9c2e-4f5b-8e26-358020d64381\" (UID: \"8b7c382c-9c2e-4f5b-8e26-358020d64381\") " Jul 23 09:25:40.809211 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:40.809180 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dw6sp\" (UniqueName: \"kubernetes.io/projected/8b7c382c-9c2e-4f5b-8e26-358020d64381-kube-api-access-dw6sp\") pod \"8b7c382c-9c2e-4f5b-8e26-358020d64381\" (UID: \"8b7c382c-9c2e-4f5b-8e26-358020d64381\") " Jul 23 09:25:40.811187 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:40.811157 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b7c382c-9c2e-4f5b-8e26-358020d64381-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "8b7c382c-9c2e-4f5b-8e26-358020d64381" (UID: "8b7c382c-9c2e-4f5b-8e26-358020d64381"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 09:25:40.811315 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:40.811234 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56a971b6-10af-48b1-b1cd-25a287da5888-kube-api-access-rfnr9" (OuterVolumeSpecName: "kube-api-access-rfnr9") pod "56a971b6-10af-48b1-b1cd-25a287da5888" (UID: "56a971b6-10af-48b1-b1cd-25a287da5888"). InnerVolumeSpecName "kube-api-access-rfnr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 09:25:40.811315 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:40.811265 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b7c382c-9c2e-4f5b-8e26-358020d64381-kube-api-access-dw6sp" (OuterVolumeSpecName: "kube-api-access-dw6sp") pod "8b7c382c-9c2e-4f5b-8e26-358020d64381" (UID: "8b7c382c-9c2e-4f5b-8e26-358020d64381"). InnerVolumeSpecName "kube-api-access-dw6sp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 09:25:40.910835 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:40.910810 2576 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8b7c382c-9c2e-4f5b-8e26-358020d64381-tls-cert\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:25:40.910835 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:40.910836 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dw6sp\" (UniqueName: \"kubernetes.io/projected/8b7c382c-9c2e-4f5b-8e26-358020d64381-kube-api-access-dw6sp\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:25:40.910997 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:40.910845 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfnr9\" (UniqueName: \"kubernetes.io/projected/56a971b6-10af-48b1-b1cd-25a287da5888-kube-api-access-rfnr9\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:25:41.739854 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:41.739819 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-99zvh" Jul 23 09:25:41.739854 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:41.739848 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c94f6ccc-7hxfj" event={"ID":"c31805e5-014e-43e6-ac72-a7259adaaf52","Type":"ContainerStarted","Data":"4bac0561ee417f05262602dbb5c07657871632a4a15770f6edef00378635ceb0"} Jul 23 09:25:41.740410 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:41.740124 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-4gp5j" Jul 23 09:25:41.757516 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:41.757476 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7c94f6ccc-7hxfj" podStartSLOduration=1.3368847910000001 podStartE2EDuration="1.757464216s" totalImagesPullingTime="420.579425ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:25:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:25:41.755481052 +0000 UTC m=+530.294759628" watchObservedRunningTime="2026-07-23 09:25:41.757464216 +0000 UTC m=+530.296742793" Jul 23 09:25:41.785651 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:41.785620 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-qh222"] Jul 23 09:25:41.785876 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:41.785778 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-qh222" podUID="4e989f09-ff5e-4c9e-bd38-f0b29b08563c" containerName="authorino" containerID="cri-o://1f6c398514ec52e136b51cc28abc40d7f8dc4ef2ae3ebd53bd303fc0482db011" gracePeriod=30 Jul 23 09:25:41.792153 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:41.792129 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-4gp5j"] Jul 23 09:25:41.794976 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:41.794945 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-76877994-4gp5j"] Jul 23 09:25:41.820182 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:41.820160 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-99zvh"] Jul 23 09:25:41.822991 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:41.822966 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-99zvh"] Jul 23 09:25:42.043178 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:42.043151 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-qh222" Jul 23 09:25:42.122409 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:42.122377 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wqdrf\" (UniqueName: \"kubernetes.io/projected/4e989f09-ff5e-4c9e-bd38-f0b29b08563c-kube-api-access-wqdrf\") pod \"4e989f09-ff5e-4c9e-bd38-f0b29b08563c\" (UID: \"4e989f09-ff5e-4c9e-bd38-f0b29b08563c\") " Jul 23 09:25:42.124534 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:42.124510 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e989f09-ff5e-4c9e-bd38-f0b29b08563c-kube-api-access-wqdrf" (OuterVolumeSpecName: "kube-api-access-wqdrf") pod "4e989f09-ff5e-4c9e-bd38-f0b29b08563c" (UID: "4e989f09-ff5e-4c9e-bd38-f0b29b08563c"). InnerVolumeSpecName "kube-api-access-wqdrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 09:25:42.223850 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:42.223813 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqdrf\" (UniqueName: \"kubernetes.io/projected/4e989f09-ff5e-4c9e-bd38-f0b29b08563c-kube-api-access-wqdrf\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:25:42.746784 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:42.746750 2576 generic.go:356] "Generic (PLEG): container finished" podID="4e989f09-ff5e-4c9e-bd38-f0b29b08563c" containerID="1f6c398514ec52e136b51cc28abc40d7f8dc4ef2ae3ebd53bd303fc0482db011" exitCode=0 Jul 23 09:25:42.747257 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:42.746797 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-qh222" Jul 23 09:25:42.747257 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:42.746840 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-qh222" event={"ID":"4e989f09-ff5e-4c9e-bd38-f0b29b08563c","Type":"ContainerDied","Data":"1f6c398514ec52e136b51cc28abc40d7f8dc4ef2ae3ebd53bd303fc0482db011"} Jul 23 09:25:42.747257 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:42.746883 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-qh222" event={"ID":"4e989f09-ff5e-4c9e-bd38-f0b29b08563c","Type":"ContainerDied","Data":"5e8c76a25a20f3d844d85a21b671cf65ff2bd9135690bddfa8301d5e2e9cc8f1"} Jul 23 09:25:42.747257 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:42.746901 2576 scope.go:117] "RemoveContainer" containerID="1f6c398514ec52e136b51cc28abc40d7f8dc4ef2ae3ebd53bd303fc0482db011" Jul 23 09:25:42.756190 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:42.756172 2576 scope.go:117] "RemoveContainer" containerID="1f6c398514ec52e136b51cc28abc40d7f8dc4ef2ae3ebd53bd303fc0482db011" Jul 23 09:25:42.756430 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:25:42.756412 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f6c398514ec52e136b51cc28abc40d7f8dc4ef2ae3ebd53bd303fc0482db011\": container with ID starting with 1f6c398514ec52e136b51cc28abc40d7f8dc4ef2ae3ebd53bd303fc0482db011 not found: ID does not exist" containerID="1f6c398514ec52e136b51cc28abc40d7f8dc4ef2ae3ebd53bd303fc0482db011" Jul 23 09:25:42.756486 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:42.756437 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f6c398514ec52e136b51cc28abc40d7f8dc4ef2ae3ebd53bd303fc0482db011"} err="failed to get container status \"1f6c398514ec52e136b51cc28abc40d7f8dc4ef2ae3ebd53bd303fc0482db011\": rpc error: code = NotFound desc = could not find container \"1f6c398514ec52e136b51cc28abc40d7f8dc4ef2ae3ebd53bd303fc0482db011\": container with ID starting with 1f6c398514ec52e136b51cc28abc40d7f8dc4ef2ae3ebd53bd303fc0482db011 not found: ID does not exist" Jul 23 09:25:42.767904 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:42.767882 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-qh222"] Jul 23 09:25:42.771729 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:42.771707 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-qh222"] Jul 23 09:25:42.986300 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:42.986269 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e989f09-ff5e-4c9e-bd38-f0b29b08563c" path="/var/lib/kubelet/pods/4e989f09-ff5e-4c9e-bd38-f0b29b08563c/volumes" Jul 23 09:25:42.986601 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:42.986590 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56a971b6-10af-48b1-b1cd-25a287da5888" path="/var/lib/kubelet/pods/56a971b6-10af-48b1-b1cd-25a287da5888/volumes" Jul 23 09:25:42.986810 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:42.986799 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b7c382c-9c2e-4f5b-8e26-358020d64381" path="/var/lib/kubelet/pods/8b7c382c-9c2e-4f5b-8e26-358020d64381/volumes" Jul 23 09:25:46.564859 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:46.564751 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:25:46.668377 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:46.668337 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:25:46.766186 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:46.766148 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:25:47.363633 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:47.363567 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:25:50.186488 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:50.186441 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:25:50.285606 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:25:50.285556 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:26:21.863496 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:26:21.863462 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:26:21.962861 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:26:21.962812 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:26:22.518256 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:26:22.518211 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:26:22.901308 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:26:22.901278 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:26:23.001232 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:26:23.001202 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:26:23.118024 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:26:23.117996 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:26:23.226793 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:26:23.226715 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:26:23.262900 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:26:23.262867 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:26:33.415291 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:26:33.415227 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:26:33.553290 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:26:33.553236 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:26:50.837825 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:26:50.837779 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:26:51.297329 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:26:51.297291 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:26:51.859954 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:26:51.859922 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-5szxg_4908aff3-196e-44fa-b8e6-c0e81b476433/console-operator/2.log" Jul 23 09:26:51.861738 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:26:51.861713 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-5szxg_4908aff3-196e-44fa-b8e6-c0e81b476433/console-operator/2.log" Jul 23 09:26:59.198133 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:26:59.198099 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:26:59.887516 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:26:59.887481 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:27:04.607346 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:27:04.607310 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:27:05.593982 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:27:05.593941 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:27:07.908450 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:27:07.908413 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:27:09.611856 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:27:09.611827 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:27:19.172729 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:27:19.172648 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:27:21.688953 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:27:21.688908 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:27:25.299421 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:27:25.299388 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-6ccj9"] Jul 23 09:27:25.299909 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:27:25.299751 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e989f09-ff5e-4c9e-bd38-f0b29b08563c" containerName="authorino" Jul 23 09:27:25.299909 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:27:25.299762 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e989f09-ff5e-4c9e-bd38-f0b29b08563c" containerName="authorino" Jul 23 09:27:25.299909 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:27:25.299817 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e989f09-ff5e-4c9e-bd38-f0b29b08563c" containerName="authorino" Jul 23 09:27:25.302818 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:27:25.302801 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6ccj9" Jul 23 09:27:25.305402 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:27:25.305384 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-hw9h8\"" Jul 23 09:27:25.305975 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:27:25.305961 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 23 09:27:25.306029 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:27:25.305998 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 23 09:27:25.307465 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:27:25.307452 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 23 09:27:25.330817 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:27:25.330782 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-6ccj9"] Jul 23 09:27:25.363670 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:27:25.363640 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:27:25.429900 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:27:25.429871 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03bcb88b-3597-43a5-a063-869d306aebca-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6ccj9\" (UID: \"03bcb88b-3597-43a5-a063-869d306aebca\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6ccj9" Jul 23 09:27:25.430075 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:27:25.429926 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsdvm\" (UniqueName: \"kubernetes.io/projected/03bcb88b-3597-43a5-a063-869d306aebca-kube-api-access-nsdvm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6ccj9\" (UID: \"03bcb88b-3597-43a5-a063-869d306aebca\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6ccj9" Jul 23 09:27:25.430075 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:27:25.429953 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03bcb88b-3597-43a5-a063-869d306aebca-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6ccj9\" (UID: \"03bcb88b-3597-43a5-a063-869d306aebca\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6ccj9" Jul 23 09:27:25.430075 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:27:25.429996 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03bcb88b-3597-43a5-a063-869d306aebca-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6ccj9\" (UID: \"03bcb88b-3597-43a5-a063-869d306aebca\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6ccj9" Jul 23 09:27:25.430075 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:27:25.430018 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03bcb88b-3597-43a5-a063-869d306aebca-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6ccj9\" (UID: \"03bcb88b-3597-43a5-a063-869d306aebca\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6ccj9" Jul 23 09:27:25.430075 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:27:25.430057 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03bcb88b-3597-43a5-a063-869d306aebca-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6ccj9\" (UID: \"03bcb88b-3597-43a5-a063-869d306aebca\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6ccj9" Jul 23 09:27:25.531128 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:27:25.531094 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03bcb88b-3597-43a5-a063-869d306aebca-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6ccj9\" (UID: \"03bcb88b-3597-43a5-a063-869d306aebca\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6ccj9" Jul 23 09:27:25.531324 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:27:25.531141 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03bcb88b-3597-43a5-a063-869d306aebca-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6ccj9\" (UID: \"03bcb88b-3597-43a5-a063-869d306aebca\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6ccj9" Jul 23 09:27:25.531324 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:27:25.531188 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03bcb88b-3597-43a5-a063-869d306aebca-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6ccj9\" (UID: \"03bcb88b-3597-43a5-a063-869d306aebca\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6ccj9" Jul 23 09:27:25.531324 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:27:25.531221 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03bcb88b-3597-43a5-a063-869d306aebca-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6ccj9\" (UID: \"03bcb88b-3597-43a5-a063-869d306aebca\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6ccj9" Jul 23 09:27:25.531324 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:27:25.531308 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsdvm\" (UniqueName: \"kubernetes.io/projected/03bcb88b-3597-43a5-a063-869d306aebca-kube-api-access-nsdvm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6ccj9\" (UID: \"03bcb88b-3597-43a5-a063-869d306aebca\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6ccj9" Jul 23 09:27:25.531583 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:27:25.531350 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03bcb88b-3597-43a5-a063-869d306aebca-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6ccj9\" (UID: \"03bcb88b-3597-43a5-a063-869d306aebca\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6ccj9" Jul 23 09:27:25.531642 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:27:25.531599 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03bcb88b-3597-43a5-a063-869d306aebca-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6ccj9\" (UID: \"03bcb88b-3597-43a5-a063-869d306aebca\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6ccj9" Jul 23 09:27:25.531642 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:27:25.531618 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03bcb88b-3597-43a5-a063-869d306aebca-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6ccj9\" (UID: \"03bcb88b-3597-43a5-a063-869d306aebca\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6ccj9" Jul 23 09:27:25.531727 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:27:25.531666 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03bcb88b-3597-43a5-a063-869d306aebca-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6ccj9\" (UID: \"03bcb88b-3597-43a5-a063-869d306aebca\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6ccj9" Jul 23 09:27:25.533466 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:27:25.533444 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03bcb88b-3597-43a5-a063-869d306aebca-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6ccj9\" (UID: \"03bcb88b-3597-43a5-a063-869d306aebca\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6ccj9" Jul 23 09:27:25.533798 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:27:25.533782 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03bcb88b-3597-43a5-a063-869d306aebca-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6ccj9\" (UID: \"03bcb88b-3597-43a5-a063-869d306aebca\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6ccj9" Jul 23 09:27:25.539090 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:27:25.539071 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsdvm\" (UniqueName: \"kubernetes.io/projected/03bcb88b-3597-43a5-a063-869d306aebca-kube-api-access-nsdvm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6ccj9\" (UID: \"03bcb88b-3597-43a5-a063-869d306aebca\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6ccj9" Jul 23 09:27:25.611810 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:27:25.611746 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6ccj9" Jul 23 09:27:25.748478 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:27:25.748453 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-6ccj9"] Jul 23 09:27:25.749993 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:27:25.749967 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod03bcb88b_3597_43a5_a063_869d306aebca.slice/crio-4d17315a81a89abab8a97d57e5c994fb5cc96d7c0e1fd2595ab61757998b6973 WatchSource:0}: Error finding container 4d17315a81a89abab8a97d57e5c994fb5cc96d7c0e1fd2595ab61757998b6973: Status 404 returned error can't find the container with id 4d17315a81a89abab8a97d57e5c994fb5cc96d7c0e1fd2595ab61757998b6973 Jul 23 09:27:25.751197 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:27:25.751182 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 09:27:26.288788 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:27:26.288745 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6ccj9" event={"ID":"03bcb88b-3597-43a5-a063-869d306aebca","Type":"ContainerStarted","Data":"4d17315a81a89abab8a97d57e5c994fb5cc96d7c0e1fd2595ab61757998b6973"} Jul 23 09:27:27.295627 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:27:27.295596 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6ccj9" event={"ID":"03bcb88b-3597-43a5-a063-869d306aebca","Type":"ContainerStarted","Data":"3bccc4dcb71638f06208bcf096038b230ffea7406d5b3f24cd1c6adc61104918"} Jul 23 09:27:27.316326 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:27:27.316275 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6ccj9" podStartSLOduration=1.19912938 podStartE2EDuration="2.316258704s" totalImagesPullingTime="1.117129324s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:27:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:27:27.313622836 +0000 UTC m=+635.852901407" watchObservedRunningTime="2026-07-23 09:27:27.316258704 +0000 UTC m=+635.855537278" Jul 23 09:27:34.989762 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:27:34.989728 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:27:35.612733 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:27:35.612695 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6ccj9" Jul 23 09:27:35.612923 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:27:35.612747 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6ccj9" Jul 23 09:27:35.625329 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:27:35.625304 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6ccj9" Jul 23 09:27:36.355666 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:27:36.355630 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6ccj9" Jul 23 09:28:36.364974 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:28:36.364937 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5876bc94d4-gs5hp"] Jul 23 09:28:36.368395 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:28:36.368379 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5876bc94d4-gs5hp" Jul 23 09:28:36.378687 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:28:36.378663 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5876bc94d4-gs5hp"] Jul 23 09:28:36.461362 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:28:36.461327 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c662fa1f-8cb7-434e-990a-31cdea707782-tls-cert\") pod \"authorino-5876bc94d4-gs5hp\" (UID: \"c662fa1f-8cb7-434e-990a-31cdea707782\") " pod="kuadrant-system/authorino-5876bc94d4-gs5hp" Jul 23 09:28:36.461522 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:28:36.461374 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p25mb\" (UniqueName: \"kubernetes.io/projected/c662fa1f-8cb7-434e-990a-31cdea707782-kube-api-access-p25mb\") pod \"authorino-5876bc94d4-gs5hp\" (UID: \"c662fa1f-8cb7-434e-990a-31cdea707782\") " pod="kuadrant-system/authorino-5876bc94d4-gs5hp" Jul 23 09:28:36.561981 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:28:36.561949 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c662fa1f-8cb7-434e-990a-31cdea707782-tls-cert\") pod \"authorino-5876bc94d4-gs5hp\" (UID: \"c662fa1f-8cb7-434e-990a-31cdea707782\") " pod="kuadrant-system/authorino-5876bc94d4-gs5hp" Jul 23 09:28:36.562136 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:28:36.561987 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p25mb\" (UniqueName: \"kubernetes.io/projected/c662fa1f-8cb7-434e-990a-31cdea707782-kube-api-access-p25mb\") pod \"authorino-5876bc94d4-gs5hp\" (UID: \"c662fa1f-8cb7-434e-990a-31cdea707782\") " pod="kuadrant-system/authorino-5876bc94d4-gs5hp" Jul 23 09:28:36.564348 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:28:36.564320 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c662fa1f-8cb7-434e-990a-31cdea707782-tls-cert\") pod \"authorino-5876bc94d4-gs5hp\" (UID: \"c662fa1f-8cb7-434e-990a-31cdea707782\") " pod="kuadrant-system/authorino-5876bc94d4-gs5hp" Jul 23 09:28:36.569426 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:28:36.569402 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p25mb\" (UniqueName: \"kubernetes.io/projected/c662fa1f-8cb7-434e-990a-31cdea707782-kube-api-access-p25mb\") pod \"authorino-5876bc94d4-gs5hp\" (UID: \"c662fa1f-8cb7-434e-990a-31cdea707782\") " pod="kuadrant-system/authorino-5876bc94d4-gs5hp" Jul 23 09:28:36.677576 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:28:36.677551 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5876bc94d4-gs5hp" Jul 23 09:28:36.750474 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:28:36.750445 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:28:36.821337 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:28:36.821310 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5876bc94d4-gs5hp"] Jul 23 09:28:36.823114 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:28:36.823085 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:28:36.823407 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:28:36.823387 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc662fa1f_8cb7_434e_990a_31cdea707782.slice/crio-c5129d5ebf71c96a966e684dde64f9188d02dc945970bc857addce2a2e365e45 WatchSource:0}: Error finding container c5129d5ebf71c96a966e684dde64f9188d02dc945970bc857addce2a2e365e45: Status 404 returned error can't find the container with id c5129d5ebf71c96a966e684dde64f9188d02dc945970bc857addce2a2e365e45 Jul 23 09:28:37.664381 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:28:37.664335 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5876bc94d4-gs5hp" event={"ID":"c662fa1f-8cb7-434e-990a-31cdea707782","Type":"ContainerStarted","Data":"44f51c191056af141b7733f943ad1c357c21c522b041e87ad4160251da10fbf1"} Jul 23 09:28:37.664861 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:28:37.664836 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5876bc94d4-gs5hp" event={"ID":"c662fa1f-8cb7-434e-990a-31cdea707782","Type":"ContainerStarted","Data":"c5129d5ebf71c96a966e684dde64f9188d02dc945970bc857addce2a2e365e45"} Jul 23 09:28:37.684864 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:28:37.684808 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5876bc94d4-gs5hp" podStartSLOduration=1.291318126 podStartE2EDuration="1.684776886s" totalImagesPullingTime="393.45876ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:28:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:28:37.683139767 +0000 UTC m=+706.222418345" watchObservedRunningTime="2026-07-23 09:28:37.684776886 +0000 UTC m=+706.224055465" Jul 23 09:28:37.720498 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:28:37.720458 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7c94f6ccc-7hxfj"] Jul 23 09:28:37.720769 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:28:37.720735 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7c94f6ccc-7hxfj" podUID="c31805e5-014e-43e6-ac72-a7259adaaf52" containerName="authorino" containerID="cri-o://4bac0561ee417f05262602dbb5c07657871632a4a15770f6edef00378635ceb0" gracePeriod=30 Jul 23 09:28:37.861965 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:28:37.861929 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:28:37.959211 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:28:37.959111 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:28:37.994613 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:28:37.994588 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c94f6ccc-7hxfj" Jul 23 09:28:38.076194 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:28:38.076160 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tlbvj\" (UniqueName: \"kubernetes.io/projected/c31805e5-014e-43e6-ac72-a7259adaaf52-kube-api-access-tlbvj\") pod \"c31805e5-014e-43e6-ac72-a7259adaaf52\" (UID: \"c31805e5-014e-43e6-ac72-a7259adaaf52\") " Jul 23 09:28:38.076403 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:28:38.076271 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c31805e5-014e-43e6-ac72-a7259adaaf52-tls-cert\") pod \"c31805e5-014e-43e6-ac72-a7259adaaf52\" (UID: \"c31805e5-014e-43e6-ac72-a7259adaaf52\") " Jul 23 09:28:38.078394 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:28:38.078361 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c31805e5-014e-43e6-ac72-a7259adaaf52-kube-api-access-tlbvj" (OuterVolumeSpecName: "kube-api-access-tlbvj") pod "c31805e5-014e-43e6-ac72-a7259adaaf52" (UID: "c31805e5-014e-43e6-ac72-a7259adaaf52"). InnerVolumeSpecName "kube-api-access-tlbvj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 09:28:38.086910 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:28:38.086885 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c31805e5-014e-43e6-ac72-a7259adaaf52-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "c31805e5-014e-43e6-ac72-a7259adaaf52" (UID: "c31805e5-014e-43e6-ac72-a7259adaaf52"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 09:28:38.177611 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:28:38.177575 2576 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c31805e5-014e-43e6-ac72-a7259adaaf52-tls-cert\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:28:38.177611 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:28:38.177608 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlbvj\" (UniqueName: \"kubernetes.io/projected/c31805e5-014e-43e6-ac72-a7259adaaf52-kube-api-access-tlbvj\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:28:38.672436 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:28:38.672400 2576 generic.go:356] "Generic (PLEG): container finished" podID="c31805e5-014e-43e6-ac72-a7259adaaf52" containerID="4bac0561ee417f05262602dbb5c07657871632a4a15770f6edef00378635ceb0" exitCode=0 Jul 23 09:28:38.672825 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:28:38.672463 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c94f6ccc-7hxfj" Jul 23 09:28:38.672825 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:28:38.672489 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c94f6ccc-7hxfj" event={"ID":"c31805e5-014e-43e6-ac72-a7259adaaf52","Type":"ContainerDied","Data":"4bac0561ee417f05262602dbb5c07657871632a4a15770f6edef00378635ceb0"} Jul 23 09:28:38.672825 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:28:38.672536 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c94f6ccc-7hxfj" event={"ID":"c31805e5-014e-43e6-ac72-a7259adaaf52","Type":"ContainerDied","Data":"4710288c0094f100993a113b0ccc717146852d346e434c08e2fad06b39e9126c"} Jul 23 09:28:38.672825 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:28:38.672552 2576 scope.go:117] "RemoveContainer" containerID="4bac0561ee417f05262602dbb5c07657871632a4a15770f6edef00378635ceb0" Jul 23 09:28:38.681608 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:28:38.681542 2576 scope.go:117] "RemoveContainer" containerID="4bac0561ee417f05262602dbb5c07657871632a4a15770f6edef00378635ceb0" Jul 23 09:28:38.681942 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:28:38.681917 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bac0561ee417f05262602dbb5c07657871632a4a15770f6edef00378635ceb0\": container with ID starting with 4bac0561ee417f05262602dbb5c07657871632a4a15770f6edef00378635ceb0 not found: ID does not exist" containerID="4bac0561ee417f05262602dbb5c07657871632a4a15770f6edef00378635ceb0" Jul 23 09:28:38.682026 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:28:38.681947 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bac0561ee417f05262602dbb5c07657871632a4a15770f6edef00378635ceb0"} err="failed to get container status \"4bac0561ee417f05262602dbb5c07657871632a4a15770f6edef00378635ceb0\": rpc error: code = NotFound desc = could not find container \"4bac0561ee417f05262602dbb5c07657871632a4a15770f6edef00378635ceb0\": container with ID starting with 4bac0561ee417f05262602dbb5c07657871632a4a15770f6edef00378635ceb0 not found: ID does not exist" Jul 23 09:28:38.698212 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:28:38.698168 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7c94f6ccc-7hxfj"] Jul 23 09:28:38.702993 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:28:38.702965 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7c94f6ccc-7hxfj"] Jul 23 09:28:38.986542 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:28:38.986502 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c31805e5-014e-43e6-ac72-a7259adaaf52" path="/var/lib/kubelet/pods/c31805e5-014e-43e6-ac72-a7259adaaf52/volumes" Jul 23 09:29:06.103891 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:29:06.103850 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:29:15.188831 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:29:15.188793 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:29:26.788950 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:29:26.788914 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:29:33.187676 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:29:33.187646 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:30:17.731232 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:30:17.731149 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:30:23.934834 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:30:23.934796 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:30:42.200058 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:30:42.200024 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:30:43.419394 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:30:43.419361 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:31:11.731671 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:31:11.731624 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:31:19.488359 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:31:19.488321 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:31:19.581526 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:31:19.581493 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:31:19.720325 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:31:19.720290 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:31:20.856025 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:31:20.855982 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:31:25.823715 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:31:25.823667 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:31:25.934854 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:31:25.934825 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:31:26.046303 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:31:26.046264 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:31:26.236262 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:31:26.236195 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:31:32.449682 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:31:32.449645 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:31:32.537796 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:31:32.537758 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:31:49.513988 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:31:49.513908 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:31:51.897792 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:31:51.897757 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-5szxg_4908aff3-196e-44fa-b8e6-c0e81b476433/console-operator/2.log" Jul 23 09:31:51.898633 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:31:51.898613 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-5szxg_4908aff3-196e-44fa-b8e6-c0e81b476433/console-operator/2.log" Jul 23 09:31:58.917467 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:31:58.917426 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:32:09.485833 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:32:09.485785 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:32:17.788009 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:32:17.787971 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:32:27.799847 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:32:27.799807 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:32:39.311125 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:32:39.311089 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:32:41.739755 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:32:41.739720 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:32:50.190641 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:32:50.190607 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:33:00.799724 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:33:00.799687 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:33:13.579777 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:33:13.579748 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:33:23.306325 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:33:23.306280 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:33:35.431056 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:33:35.431023 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:34:01.816409 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:34:01.816370 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:34:15.709947 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:34:15.709903 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:34:27.505520 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:34:27.505469 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:34:39.326326 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:34:39.326289 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:34:49.315017 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:34:49.314923 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:34:58.116401 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:34:58.116369 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:35:07.499858 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:35:07.499815 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:35:36.267274 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:35:36.267214 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:36:30.444939 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:36:30.444836 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:36:51.933014 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:36:51.932987 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-5szxg_4908aff3-196e-44fa-b8e6-c0e81b476433/console-operator/2.log" Jul 23 09:36:51.933014 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:36:51.933013 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-5szxg_4908aff3-196e-44fa-b8e6-c0e81b476433/console-operator/2.log" Jul 23 09:36:54.040038 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:36:54.039999 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:37:05.903414 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:37:05.903373 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:37:14.907207 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:37:14.907173 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:37:26.516170 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:37:26.516136 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:37:36.105889 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:37:36.105851 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:37:49.814762 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:37:49.814684 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:37:58.710633 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:37:58.710598 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:38:08.728075 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:38:08.728040 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:38:18.708514 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:38:18.708478 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:38:27.911174 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:38:27.911130 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:38:39.097937 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:38:39.097902 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:39:06.125324 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:39:06.125278 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:39:13.990722 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:39:13.990644 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:39:29.058178 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:39:29.058130 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:39:32.204704 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:39:32.204669 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:39:45.491724 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:39:45.491686 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:39:49.209347 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:39:49.209303 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:40:36.322115 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:40:36.322083 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:40:37.386928 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:40:37.386892 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:40:40.298820 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:40:40.298780 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:40:54.195099 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:40:54.195054 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:41:00.489714 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:41:00.489674 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:41:17.901025 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:41:17.900988 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:41:27.302555 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:41:27.302517 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:41:36.596915 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:41:36.596879 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:41:44.395167 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:41:44.395129 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:41:51.971823 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:41:51.971795 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-5szxg_4908aff3-196e-44fa-b8e6-c0e81b476433/console-operator/2.log" Jul 23 09:41:51.973837 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:41:51.973814 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-5szxg_4908aff3-196e-44fa-b8e6-c0e81b476433/console-operator/2.log" Jul 23 09:41:55.392835 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:41:55.392797 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:42:15.299232 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:42:15.299193 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:42:22.840568 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:42:22.840536 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:42:32.408698 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:42:32.408655 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:42:42.196031 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:42:42.195997 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:42:43.112545 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:42:43.112512 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:42:52.198573 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:42:52.198541 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:43:03.902492 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:43:03.902456 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:43:07.886672 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:43:07.886638 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:43:08.025615 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:43:08.025582 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:43:13.406390 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:43:13.406352 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:43:26.800155 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:43:26.800118 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:43:38.493834 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:43:38.493798 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:43:49.292965 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:43:49.292884 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:43:55.818977 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:43:55.818942 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:43:55.919645 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:43:55.919604 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:43:56.397848 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:43:56.397815 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:43:58.948226 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:43:58.948190 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:44:09.391949 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:44:09.391915 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:44:27.063756 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:44:27.063727 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:44:30.408551 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:44:30.408516 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:44:30.470433 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:44:30.470398 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:44:30.575522 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:44:30.575487 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:44:30.977188 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:44:30.977156 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:44:36.729622 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:44:36.729592 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:44:37.947479 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:44:37.947446 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:44:38.062270 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:44:38.062209 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:44:40.022076 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:44:40.022041 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:44:40.143614 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:44:40.143577 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:44:40.240733 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:44:40.240705 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:44:40.457858 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:44:40.457832 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:44:40.806807 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:44:40.806720 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:44:40.907642 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:44:40.907603 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:44:47.101673 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:44:47.101641 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:44:57.422652 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:44:57.422620 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:45:16.284288 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:45:16.284202 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:45:19.291757 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:45:19.291721 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:45:20.905697 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:45:20.905659 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:45:21.078352 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:45:21.078323 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:45:21.340766 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:45:21.340697 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:45:21.640641 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:45:21.640611 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:45:21.735711 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:45:21.735685 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:45:21.866956 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:45:21.866926 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:45:22.225226 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:45:22.225193 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:45:22.468964 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:45:22.468929 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:45:28.624494 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:45:28.624465 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:45:28.850962 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:45:28.850930 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:45:29.169798 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:45:29.169771 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:45:29.421753 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:45:29.421687 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:45:29.514823 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:45:29.514795 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:45:34.735500 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:45:34.735464 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:45:53.799849 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:45:53.799808 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bd6e3a7b-nj86r"] Jul 23 09:45:53.800284 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:45:53.800175 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c31805e5-014e-43e6-ac72-a7259adaaf52" containerName="authorino" Jul 23 09:45:53.800284 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:45:53.800186 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c31805e5-014e-43e6-ac72-a7259adaaf52" containerName="authorino" Jul 23 09:45:53.800284 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:45:53.800268 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c31805e5-014e-43e6-ac72-a7259adaaf52" containerName="authorino" Jul 23 09:45:53.803173 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:45:53.803157 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bd6e3a7b-nj86r" Jul 23 09:45:53.805799 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:45:53.805777 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n4hmw\"" Jul 23 09:45:53.806006 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:45:53.805992 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 09:45:53.813369 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:45:53.813345 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bd6e3a7b-nj86r"] Jul 23 09:45:53.906163 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:45:53.906132 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80f63f2b-5316-4a56-9ee3-7558aa7d1563-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bd6e3a7b-nj86r\" (UID: \"80f63f2b-5316-4a56-9ee3-7558aa7d1563\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bd6e3a7b-nj86r" Jul 23 09:45:54.006583 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:45:54.006557 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80f63f2b-5316-4a56-9ee3-7558aa7d1563-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bd6e3a7b-nj86r\" (UID: \"80f63f2b-5316-4a56-9ee3-7558aa7d1563\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bd6e3a7b-nj86r" Jul 23 09:45:54.007169 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:45:54.007149 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80f63f2b-5316-4a56-9ee3-7558aa7d1563-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bd6e3a7b-nj86r\" (UID: \"80f63f2b-5316-4a56-9ee3-7558aa7d1563\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bd6e3a7b-nj86r" Jul 23 09:45:54.113919 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:45:54.113851 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bd6e3a7b-nj86r" Jul 23 09:45:54.243700 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:45:54.243628 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bd6e3a7b-nj86r"] Jul 23 09:45:54.245251 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:45:54.245208 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80f63f2b_5316_4a56_9ee3_7558aa7d1563.slice/crio-771e755a550bc58a2a4cf3b520492332b5e49ffe7cf3fbaa0701913bfe29ca49 WatchSource:0}: Error finding container 771e755a550bc58a2a4cf3b520492332b5e49ffe7cf3fbaa0701913bfe29ca49: Status 404 returned error can't find the container with id 771e755a550bc58a2a4cf3b520492332b5e49ffe7cf3fbaa0701913bfe29ca49 Jul 23 09:45:54.247125 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:45:54.247106 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 09:45:55.054843 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:45:55.054808 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bd6e3a7b-nj86r" event={"ID":"80f63f2b-5316-4a56-9ee3-7558aa7d1563","Type":"ContainerStarted","Data":"771e755a550bc58a2a4cf3b520492332b5e49ffe7cf3fbaa0701913bfe29ca49"} Jul 23 09:45:57.067865 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:45:57.067823 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bd6e3a7b-nj86r" event={"ID":"80f63f2b-5316-4a56-9ee3-7558aa7d1563","Type":"ContainerStarted","Data":"490bf085af706004fb6f578302be16d690d1d2709261b48823de04125e08c098"} Jul 23 09:45:57.086555 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:45:57.086500 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bd6e3a7b-nj86r" podStartSLOduration=1.880903958 podStartE2EDuration="4.086485277s" totalImagesPullingTime="2.205581319s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:45:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:45:57.084785581 +0000 UTC m=+1745.624064148" watchObservedRunningTime="2026-07-23 09:45:57.086485277 +0000 UTC m=+1745.625763831" Jul 23 09:45:58.074297 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:45:58.074265 2576 generic.go:356] "Generic (PLEG): container finished" podID="80f63f2b-5316-4a56-9ee3-7558aa7d1563" containerID="490bf085af706004fb6f578302be16d690d1d2709261b48823de04125e08c098" exitCode=0 Jul 23 09:45:58.074693 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:45:58.074389 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bd6e3a7b-nj86r" event={"ID":"80f63f2b-5316-4a56-9ee3-7558aa7d1563","Type":"ContainerDied","Data":"490bf085af706004fb6f578302be16d690d1d2709261b48823de04125e08c098"} Jul 23 09:45:59.207624 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:45:59.207587 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bd6e3a7b-nj86r" Jul 23 09:45:59.354591 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:45:59.354521 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80f63f2b-5316-4a56-9ee3-7558aa7d1563-maas-api-service-ca\") pod \"80f63f2b-5316-4a56-9ee3-7558aa7d1563\" (UID: \"80f63f2b-5316-4a56-9ee3-7558aa7d1563\") " Jul 23 09:45:59.354822 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:45:59.354800 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80f63f2b-5316-4a56-9ee3-7558aa7d1563-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "80f63f2b-5316-4a56-9ee3-7558aa7d1563" (UID: "80f63f2b-5316-4a56-9ee3-7558aa7d1563"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 09:45:59.455986 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:45:59.455961 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80f63f2b-5316-4a56-9ee3-7558aa7d1563-maas-api-service-ca\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:46:00.085657 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:46:00.085632 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bd6e3a7b-nj86r" Jul 23 09:46:00.085838 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:46:00.085626 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bd6e3a7b-nj86r" event={"ID":"80f63f2b-5316-4a56-9ee3-7558aa7d1563","Type":"ContainerDied","Data":"771e755a550bc58a2a4cf3b520492332b5e49ffe7cf3fbaa0701913bfe29ca49"} Jul 23 09:46:00.085838 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:46:00.085765 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="771e755a550bc58a2a4cf3b520492332b5e49ffe7cf3fbaa0701913bfe29ca49" Jul 23 09:46:04.327611 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:46:04.327569 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:46:07.302105 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:46:07.302071 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:46:07.647951 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:46:07.647918 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:46:07.752830 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:46:07.752801 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:46:08.852839 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:46:08.852809 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bd6e3a7b-nj86r"] Jul 23 09:46:08.855610 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:46:08.855572 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bd6e3a7b-nj86r"] Jul 23 09:46:08.856367 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:46:08.856347 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:46:08.986877 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:46:08.986847 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80f63f2b-5316-4a56-9ee3-7558aa7d1563" path="/var/lib/kubelet/pods/80f63f2b-5316-4a56-9ee3-7558aa7d1563/volumes" Jul 23 09:46:09.022362 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:46:09.022331 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:46:09.277086 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:46:09.277044 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:46:09.522740 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:46:09.522697 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:46:09.865836 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:46:09.865805 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:46:10.372340 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:46:10.372313 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:46:12.648458 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:46:12.648409 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:46:12.738701 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:46:12.738650 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:46:12.829830 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:46:12.829799 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:46:13.255131 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:46:13.255097 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:46:13.623547 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:46:13.623509 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:46:13.724376 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:46:13.724335 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:46:19.494658 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:46:19.494618 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:46:38.617992 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:46:38.617955 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ee6db4d-nbvsj"] Jul 23 09:46:38.618450 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:46:38.618320 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80f63f2b-5316-4a56-9ee3-7558aa7d1563" containerName="revoke-keys" Jul 23 09:46:38.618450 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:46:38.618331 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="80f63f2b-5316-4a56-9ee3-7558aa7d1563" containerName="revoke-keys" Jul 23 09:46:38.618450 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:46:38.618400 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="80f63f2b-5316-4a56-9ee3-7558aa7d1563" containerName="revoke-keys" Jul 23 09:46:38.623259 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:46:38.623227 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ee6db4d-nbvsj" Jul 23 09:46:38.625506 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:46:38.625485 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 09:46:38.625614 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:46:38.625596 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n4hmw\"" Jul 23 09:46:38.630169 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:46:38.630145 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ee6db4d-nbvsj"] Jul 23 09:46:38.771605 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:46:38.771576 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfbd6776-1916-4a97-a673-d47423a6ed48-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ee6db4d-nbvsj\" (UID: \"bfbd6776-1916-4a97-a673-d47423a6ed48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ee6db4d-nbvsj" Jul 23 09:46:38.872267 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:46:38.872171 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfbd6776-1916-4a97-a673-d47423a6ed48-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ee6db4d-nbvsj\" (UID: \"bfbd6776-1916-4a97-a673-d47423a6ed48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ee6db4d-nbvsj" Jul 23 09:46:38.872766 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:46:38.872748 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfbd6776-1916-4a97-a673-d47423a6ed48-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ee6db4d-nbvsj\" (UID: \"bfbd6776-1916-4a97-a673-d47423a6ed48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ee6db4d-nbvsj" Jul 23 09:46:38.934396 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:46:38.934374 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ee6db4d-nbvsj" Jul 23 09:46:39.059598 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:46:39.059567 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ee6db4d-nbvsj"] Jul 23 09:46:39.061807 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:46:39.061766 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfbd6776_1916_4a97_a673_d47423a6ed48.slice/crio-65d641feaa90beeb486da6f8a281822cae9e1b8d6f272e9bda0709f45b21ef9f WatchSource:0}: Error finding container 65d641feaa90beeb486da6f8a281822cae9e1b8d6f272e9bda0709f45b21ef9f: Status 404 returned error can't find the container with id 65d641feaa90beeb486da6f8a281822cae9e1b8d6f272e9bda0709f45b21ef9f Jul 23 09:46:39.280663 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:46:39.280633 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ee6db4d-nbvsj" event={"ID":"bfbd6776-1916-4a97-a673-d47423a6ed48","Type":"ContainerStarted","Data":"65d641feaa90beeb486da6f8a281822cae9e1b8d6f272e9bda0709f45b21ef9f"} Jul 23 09:46:40.287467 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:46:40.287433 2576 generic.go:356] "Generic (PLEG): container finished" podID="bfbd6776-1916-4a97-a673-d47423a6ed48" containerID="cf317fdc14f97d931660d1d31e1cb1225f601e9fbaced12ddda21c7ed7af2c13" exitCode=0 Jul 23 09:46:40.287801 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:46:40.287515 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ee6db4d-nbvsj" event={"ID":"bfbd6776-1916-4a97-a673-d47423a6ed48","Type":"ContainerDied","Data":"cf317fdc14f97d931660d1d31e1cb1225f601e9fbaced12ddda21c7ed7af2c13"} Jul 23 09:46:41.417999 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:46:41.417975 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ee6db4d-nbvsj" Jul 23 09:46:41.490886 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:46:41.490864 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfbd6776-1916-4a97-a673-d47423a6ed48-maas-api-service-ca\") pod \"bfbd6776-1916-4a97-a673-d47423a6ed48\" (UID: \"bfbd6776-1916-4a97-a673-d47423a6ed48\") " Jul 23 09:46:41.491263 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:46:41.491219 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfbd6776-1916-4a97-a673-d47423a6ed48-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bfbd6776-1916-4a97-a673-d47423a6ed48" (UID: "bfbd6776-1916-4a97-a673-d47423a6ed48"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 09:46:41.592399 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:46:41.592341 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfbd6776-1916-4a97-a673-d47423a6ed48-maas-api-service-ca\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:46:42.298379 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:46:42.298352 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ee6db4d-nbvsj" Jul 23 09:46:42.298545 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:46:42.298385 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ee6db4d-nbvsj" event={"ID":"bfbd6776-1916-4a97-a673-d47423a6ed48","Type":"ContainerDied","Data":"65d641feaa90beeb486da6f8a281822cae9e1b8d6f272e9bda0709f45b21ef9f"} Jul 23 09:46:42.298545 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:46:42.298438 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65d641feaa90beeb486da6f8a281822cae9e1b8d6f272e9bda0709f45b21ef9f" Jul 23 09:46:49.174451 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:46:49.174418 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:46:51.904283 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:46:51.904250 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:46:52.009664 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:46:52.009638 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-5szxg_4908aff3-196e-44fa-b8e6-c0e81b476433/console-operator/2.log" Jul 23 09:46:52.012346 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:46:52.012323 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-5szxg_4908aff3-196e-44fa-b8e6-c0e81b476433/console-operator/2.log" Jul 23 09:46:53.694360 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:46:53.694305 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:46:53.696161 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:46:53.696134 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ee6db4d-nbvsj"] Jul 23 09:46:53.698084 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:46:53.698063 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ee6db4d-nbvsj"] Jul 23 09:46:54.985916 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:46:54.985885 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfbd6776-1916-4a97-a673-d47423a6ed48" path="/var/lib/kubelet/pods/bfbd6776-1916-4a97-a673-d47423a6ed48/volumes" Jul 23 09:46:54.991320 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:46:54.991296 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:46:55.244988 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:46:55.244914 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:46:55.492635 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:46:55.492593 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:46:55.817157 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:46:55.817127 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:46:56.289057 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:46:56.289020 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:46:57.870788 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:46:57.870734 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:46:57.977345 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:46:57.977313 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:46:58.073395 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:46:58.073368 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:46:58.210172 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:46:58.210147 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:46:58.804176 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:46:58.804145 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:46:58.902884 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:46:58.902850 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:47:04.503678 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:47:04.503643 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:47:22.812660 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:47:22.812623 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0b80ea18-thbt6"] Jul 23 09:47:22.815008 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:47:22.812969 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfbd6776-1916-4a97-a673-d47423a6ed48" containerName="revoke-keys" Jul 23 09:47:22.815008 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:47:22.812980 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfbd6776-1916-4a97-a673-d47423a6ed48" containerName="revoke-keys" Jul 23 09:47:22.815008 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:47:22.813059 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfbd6776-1916-4a97-a673-d47423a6ed48" containerName="revoke-keys" Jul 23 09:47:22.815938 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:47:22.815922 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0b80ea18-thbt6" Jul 23 09:47:22.818518 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:47:22.818493 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n4hmw\"" Jul 23 09:47:22.818800 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:47:22.818784 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 09:47:22.825319 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:47:22.825291 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0b80ea18-thbt6"] Jul 23 09:47:22.937839 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:47:22.937799 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb1318e6-d78b-4b60-9d93-46c9da411e49-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0b80ea18-thbt6\" (UID: \"fb1318e6-d78b-4b60-9d93-46c9da411e49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0b80ea18-thbt6" Jul 23 09:47:23.039445 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:47:23.039407 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb1318e6-d78b-4b60-9d93-46c9da411e49-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0b80ea18-thbt6\" (UID: \"fb1318e6-d78b-4b60-9d93-46c9da411e49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0b80ea18-thbt6" Jul 23 09:47:23.040130 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:47:23.040109 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb1318e6-d78b-4b60-9d93-46c9da411e49-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0b80ea18-thbt6\" (UID: \"fb1318e6-d78b-4b60-9d93-46c9da411e49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0b80ea18-thbt6" Jul 23 09:47:23.128286 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:47:23.128006 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0b80ea18-thbt6" Jul 23 09:47:23.293137 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:47:23.293112 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0b80ea18-thbt6"] Jul 23 09:47:23.295340 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:47:23.295292 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb1318e6_d78b_4b60_9d93_46c9da411e49.slice/crio-f332a973e21673e0ce459b8edd2f628b53d5aa47f7fb43b825c02f28d953cf08 WatchSource:0}: Error finding container f332a973e21673e0ce459b8edd2f628b53d5aa47f7fb43b825c02f28d953cf08: Status 404 returned error can't find the container with id f332a973e21673e0ce459b8edd2f628b53d5aa47f7fb43b825c02f28d953cf08 Jul 23 09:47:23.514078 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:47:23.513985 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0b80ea18-thbt6" event={"ID":"fb1318e6-d78b-4b60-9d93-46c9da411e49","Type":"ContainerStarted","Data":"f332a973e21673e0ce459b8edd2f628b53d5aa47f7fb43b825c02f28d953cf08"} Jul 23 09:47:24.521979 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:47:24.521939 2576 generic.go:356] "Generic (PLEG): container finished" podID="fb1318e6-d78b-4b60-9d93-46c9da411e49" containerID="398a26f7f4d6852a52570dec201e00cd063811e0b15c33dd92618907f5b28e54" exitCode=0 Jul 23 09:47:24.522447 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:47:24.522002 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0b80ea18-thbt6" event={"ID":"fb1318e6-d78b-4b60-9d93-46c9da411e49","Type":"ContainerDied","Data":"398a26f7f4d6852a52570dec201e00cd063811e0b15c33dd92618907f5b28e54"} Jul 23 09:47:25.653710 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:47:25.653683 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0b80ea18-thbt6" Jul 23 09:47:25.764149 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:47:25.764108 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb1318e6-d78b-4b60-9d93-46c9da411e49-maas-api-service-ca\") pod \"fb1318e6-d78b-4b60-9d93-46c9da411e49\" (UID: \"fb1318e6-d78b-4b60-9d93-46c9da411e49\") " Jul 23 09:47:25.764517 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:47:25.764493 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb1318e6-d78b-4b60-9d93-46c9da411e49-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fb1318e6-d78b-4b60-9d93-46c9da411e49" (UID: "fb1318e6-d78b-4b60-9d93-46c9da411e49"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 09:47:25.865025 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:47:25.864934 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb1318e6-d78b-4b60-9d93-46c9da411e49-maas-api-service-ca\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:47:26.533500 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:47:26.533469 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0b80ea18-thbt6" event={"ID":"fb1318e6-d78b-4b60-9d93-46c9da411e49","Type":"ContainerDied","Data":"f332a973e21673e0ce459b8edd2f628b53d5aa47f7fb43b825c02f28d953cf08"} Jul 23 09:47:26.533500 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:47:26.533494 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0b80ea18-thbt6" Jul 23 09:47:26.533714 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:47:26.533517 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f332a973e21673e0ce459b8edd2f628b53d5aa47f7fb43b825c02f28d953cf08" Jul 23 09:47:33.324090 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:47:33.324061 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:47:35.087233 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:47:35.087198 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:47:37.889439 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:47:37.889389 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0b80ea18-thbt6"] Jul 23 09:47:37.891479 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:47:37.891459 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:47:37.891633 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:47:37.891611 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0b80ea18-thbt6"] Jul 23 09:47:38.060219 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:47:38.060185 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:47:38.298528 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:47:38.298445 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:47:38.820685 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:47:38.820633 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:47:38.913253 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:47:38.913215 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:47:38.984761 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:47:38.984735 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb1318e6-d78b-4b60-9d93-46c9da411e49" path="/var/lib/kubelet/pods/fb1318e6-d78b-4b60-9d93-46c9da411e49/volumes" Jul 23 09:47:39.012426 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:47:39.012389 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:47:40.445531 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:47:40.445499 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:47:46.376044 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:47:46.376013 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:47:46.485972 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:47:46.485939 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:47:46.576328 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:47:46.576284 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:47:46.678771 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:47:46.678666 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:47:47.190057 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:47:47.189987 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:48:13.377668 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:48:13.377611 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:48:19.603171 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:48:19.603135 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:48:19.697649 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:48:19.697619 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:48:19.796710 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:48:19.796675 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:48:35.748962 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:48:35.748930 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:48:41.477997 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:48:41.477956 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:48:41.541702 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:48:41.541669 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:48:41.631955 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:48:41.631919 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:48:41.729899 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:48:41.729815 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:48:52.223320 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:48:52.223286 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:48:53.037943 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:48:53.037904 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:48:59.389768 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:48:59.389727 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:48:59.504887 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:48:59.504856 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:48:59.611390 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:48:59.611359 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:48:59.713608 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:48:59.713515 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:49:00.470357 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:49:00.470321 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:49:00.570377 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:49:00.570346 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:49:00.670417 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:49:00.670386 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:49:02.389589 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:49:02.389561 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:49:02.879809 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:49:02.879770 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:49:08.239802 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:49:08.239766 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:49:08.302873 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:49:08.302846 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:49:08.904986 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:49:08.904949 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:49:14.865863 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:49:14.865824 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:49:14.930371 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:49:14.930340 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:49:14.994814 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:49:14.994783 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:49:15.112947 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:49:15.112882 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:49:15.442439 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:49:15.442408 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:49:15.537540 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:49:15.537504 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:49:18.524956 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:49:18.524924 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:49:18.602834 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:49:18.602803 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:49:18.672575 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:49:18.672548 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:49:18.942477 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:49:18.942445 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:49:19.193205 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:49:19.193112 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:49:19.300132 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:49:19.300094 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:49:19.833888 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:49:19.833857 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:49:19.934211 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:49:19.934182 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:49:22.959015 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:49:22.958976 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:49:23.641613 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:49:23.641579 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:49:23.746618 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:49:23.746584 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:49:24.128562 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:49:24.128535 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:49:29.288735 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:49:29.288693 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:49:29.788993 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:49:29.788957 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:49:29.842984 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:49:29.842947 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:49:40.585822 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:49:40.585778 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:49:43.393719 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:49:43.393681 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:49:45.221031 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:49:45.220996 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:49:45.622995 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:49:45.622967 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:49:51.883981 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:49:51.883946 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:49:51.944942 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:49:51.944907 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:49:52.148335 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:49:52.148256 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:49:52.392465 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:49:52.392432 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:49:52.509684 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:49:52.509596 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:49:52.635384 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:49:52.635346 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:49:52.823866 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:49:52.823784 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:49:58.746421 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:49:58.746389 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:49:58.854468 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:49:58.854439 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:49:58.949475 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:49:58.949442 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:49:59.072380 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:49:59.072302 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:50:05.104901 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:50:05.104849 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:50:05.204948 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:50:05.204915 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:50:11.979384 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:50:11.979319 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:50:12.293647 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:50:12.293560 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:50:12.793997 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:50:12.793962 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:50:15.541459 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:50:15.541423 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:50:15.685148 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:50:15.685104 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:50:15.774366 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:50:15.774333 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:50:16.103664 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:50:16.103631 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:50:16.204993 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:50:16.204964 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:50:16.305430 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:50:16.305395 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:50:19.999125 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:50:19.999096 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:50:22.295459 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:50:22.295424 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:50:30.334103 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:50:30.334071 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:50:30.448182 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:50:30.448152 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:50:31.059035 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:50:31.059004 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-18c2a482/e2e-lifecycle-model-18c2a482-kserve-7586f5c8bc-6257s"] Jul 23 09:50:31.059427 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:50:31.059409 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb1318e6-d78b-4b60-9d93-46c9da411e49" containerName="revoke-keys" Jul 23 09:50:31.059518 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:50:31.059429 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb1318e6-d78b-4b60-9d93-46c9da411e49" containerName="revoke-keys" Jul 23 09:50:31.059518 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:50:31.059488 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb1318e6-d78b-4b60-9d93-46c9da411e49" containerName="revoke-keys" Jul 23 09:50:31.062506 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:50:31.062488 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-18c2a482/e2e-lifecycle-model-18c2a482-kserve-7586f5c8bc-6257s" Jul 23 09:50:31.065602 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:50:31.065552 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-18c2a482\"/\"default-dockercfg-wzgbt\"" Jul 23 09:50:31.065731 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:50:31.065610 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-18c2a482\"/\"e2e-lifecycle-model-18c2a482-kserve-self-signed-certs\"" Jul 23 09:50:31.065731 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:50:31.065677 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-18c2a482\"/\"kube-root-ca.crt\"" Jul 23 09:50:31.065731 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:50:31.065691 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-18c2a482\"/\"openshift-service-ca.crt\"" Jul 23 09:50:31.073060 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:50:31.073038 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-18c2a482/e2e-lifecycle-model-18c2a482-kserve-7586f5c8bc-6257s"] Jul 23 09:50:31.091934 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:50:31.091912 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:50:31.123864 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:50:31.123842 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acdc9e8e-27e5-42dd-ac49-44ab50a33044-home\") pod \"e2e-lifecycle-model-18c2a482-kserve-7586f5c8bc-6257s\" (UID: \"acdc9e8e-27e5-42dd-ac49-44ab50a33044\") " pod="ai-tenant-e2e-mt-18c2a482/e2e-lifecycle-model-18c2a482-kserve-7586f5c8bc-6257s" Jul 23 09:50:31.123962 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:50:31.123867 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acdc9e8e-27e5-42dd-ac49-44ab50a33044-tls-certs\") pod \"e2e-lifecycle-model-18c2a482-kserve-7586f5c8bc-6257s\" (UID: \"acdc9e8e-27e5-42dd-ac49-44ab50a33044\") " pod="ai-tenant-e2e-mt-18c2a482/e2e-lifecycle-model-18c2a482-kserve-7586f5c8bc-6257s" Jul 23 09:50:31.123962 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:50:31.123888 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plc7g\" (UniqueName: \"kubernetes.io/projected/acdc9e8e-27e5-42dd-ac49-44ab50a33044-kube-api-access-plc7g\") pod \"e2e-lifecycle-model-18c2a482-kserve-7586f5c8bc-6257s\" (UID: \"acdc9e8e-27e5-42dd-ac49-44ab50a33044\") " pod="ai-tenant-e2e-mt-18c2a482/e2e-lifecycle-model-18c2a482-kserve-7586f5c8bc-6257s" Jul 23 09:50:31.124129 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:50:31.123988 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acdc9e8e-27e5-42dd-ac49-44ab50a33044-dshm\") pod \"e2e-lifecycle-model-18c2a482-kserve-7586f5c8bc-6257s\" (UID: \"acdc9e8e-27e5-42dd-ac49-44ab50a33044\") " pod="ai-tenant-e2e-mt-18c2a482/e2e-lifecycle-model-18c2a482-kserve-7586f5c8bc-6257s" Jul 23 09:50:31.124129 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:50:31.124039 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acdc9e8e-27e5-42dd-ac49-44ab50a33044-model-cache\") pod \"e2e-lifecycle-model-18c2a482-kserve-7586f5c8bc-6257s\" (UID: \"acdc9e8e-27e5-42dd-ac49-44ab50a33044\") " pod="ai-tenant-e2e-mt-18c2a482/e2e-lifecycle-model-18c2a482-kserve-7586f5c8bc-6257s" Jul 23 09:50:31.124129 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:50:31.124096 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acdc9e8e-27e5-42dd-ac49-44ab50a33044-tmp-dir\") pod \"e2e-lifecycle-model-18c2a482-kserve-7586f5c8bc-6257s\" (UID: \"acdc9e8e-27e5-42dd-ac49-44ab50a33044\") " pod="ai-tenant-e2e-mt-18c2a482/e2e-lifecycle-model-18c2a482-kserve-7586f5c8bc-6257s" Jul 23 09:50:31.225181 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:50:31.225155 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acdc9e8e-27e5-42dd-ac49-44ab50a33044-dshm\") pod \"e2e-lifecycle-model-18c2a482-kserve-7586f5c8bc-6257s\" (UID: \"acdc9e8e-27e5-42dd-ac49-44ab50a33044\") " pod="ai-tenant-e2e-mt-18c2a482/e2e-lifecycle-model-18c2a482-kserve-7586f5c8bc-6257s" Jul 23 09:50:31.225319 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:50:31.225193 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acdc9e8e-27e5-42dd-ac49-44ab50a33044-model-cache\") pod \"e2e-lifecycle-model-18c2a482-kserve-7586f5c8bc-6257s\" (UID: \"acdc9e8e-27e5-42dd-ac49-44ab50a33044\") " pod="ai-tenant-e2e-mt-18c2a482/e2e-lifecycle-model-18c2a482-kserve-7586f5c8bc-6257s" Jul 23 09:50:31.225388 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:50:31.225328 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acdc9e8e-27e5-42dd-ac49-44ab50a33044-tmp-dir\") pod \"e2e-lifecycle-model-18c2a482-kserve-7586f5c8bc-6257s\" (UID: \"acdc9e8e-27e5-42dd-ac49-44ab50a33044\") " pod="ai-tenant-e2e-mt-18c2a482/e2e-lifecycle-model-18c2a482-kserve-7586f5c8bc-6257s" Jul 23 09:50:31.225452 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:50:31.225384 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acdc9e8e-27e5-42dd-ac49-44ab50a33044-home\") pod \"e2e-lifecycle-model-18c2a482-kserve-7586f5c8bc-6257s\" (UID: \"acdc9e8e-27e5-42dd-ac49-44ab50a33044\") " pod="ai-tenant-e2e-mt-18c2a482/e2e-lifecycle-model-18c2a482-kserve-7586f5c8bc-6257s" Jul 23 09:50:31.225452 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:50:31.225410 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acdc9e8e-27e5-42dd-ac49-44ab50a33044-tls-certs\") pod \"e2e-lifecycle-model-18c2a482-kserve-7586f5c8bc-6257s\" (UID: \"acdc9e8e-27e5-42dd-ac49-44ab50a33044\") " pod="ai-tenant-e2e-mt-18c2a482/e2e-lifecycle-model-18c2a482-kserve-7586f5c8bc-6257s" Jul 23 09:50:31.225452 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:50:31.225442 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plc7g\" (UniqueName: \"kubernetes.io/projected/acdc9e8e-27e5-42dd-ac49-44ab50a33044-kube-api-access-plc7g\") pod \"e2e-lifecycle-model-18c2a482-kserve-7586f5c8bc-6257s\" (UID: \"acdc9e8e-27e5-42dd-ac49-44ab50a33044\") " pod="ai-tenant-e2e-mt-18c2a482/e2e-lifecycle-model-18c2a482-kserve-7586f5c8bc-6257s" Jul 23 09:50:31.225597 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:50:31.225521 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acdc9e8e-27e5-42dd-ac49-44ab50a33044-model-cache\") pod \"e2e-lifecycle-model-18c2a482-kserve-7586f5c8bc-6257s\" (UID: \"acdc9e8e-27e5-42dd-ac49-44ab50a33044\") " pod="ai-tenant-e2e-mt-18c2a482/e2e-lifecycle-model-18c2a482-kserve-7586f5c8bc-6257s" Jul 23 09:50:31.225674 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:50:31.225652 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acdc9e8e-27e5-42dd-ac49-44ab50a33044-tmp-dir\") pod \"e2e-lifecycle-model-18c2a482-kserve-7586f5c8bc-6257s\" (UID: \"acdc9e8e-27e5-42dd-ac49-44ab50a33044\") " pod="ai-tenant-e2e-mt-18c2a482/e2e-lifecycle-model-18c2a482-kserve-7586f5c8bc-6257s" Jul 23 09:50:31.225732 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:50:31.225696 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acdc9e8e-27e5-42dd-ac49-44ab50a33044-home\") pod \"e2e-lifecycle-model-18c2a482-kserve-7586f5c8bc-6257s\" (UID: \"acdc9e8e-27e5-42dd-ac49-44ab50a33044\") " pod="ai-tenant-e2e-mt-18c2a482/e2e-lifecycle-model-18c2a482-kserve-7586f5c8bc-6257s" Jul 23 09:50:31.227450 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:50:31.227434 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acdc9e8e-27e5-42dd-ac49-44ab50a33044-dshm\") pod \"e2e-lifecycle-model-18c2a482-kserve-7586f5c8bc-6257s\" (UID: \"acdc9e8e-27e5-42dd-ac49-44ab50a33044\") " pod="ai-tenant-e2e-mt-18c2a482/e2e-lifecycle-model-18c2a482-kserve-7586f5c8bc-6257s" Jul 23 09:50:31.227752 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:50:31.227735 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acdc9e8e-27e5-42dd-ac49-44ab50a33044-tls-certs\") pod \"e2e-lifecycle-model-18c2a482-kserve-7586f5c8bc-6257s\" (UID: \"acdc9e8e-27e5-42dd-ac49-44ab50a33044\") " pod="ai-tenant-e2e-mt-18c2a482/e2e-lifecycle-model-18c2a482-kserve-7586f5c8bc-6257s" Jul 23 09:50:31.233423 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:50:31.233402 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plc7g\" (UniqueName: \"kubernetes.io/projected/acdc9e8e-27e5-42dd-ac49-44ab50a33044-kube-api-access-plc7g\") pod \"e2e-lifecycle-model-18c2a482-kserve-7586f5c8bc-6257s\" (UID: \"acdc9e8e-27e5-42dd-ac49-44ab50a33044\") " pod="ai-tenant-e2e-mt-18c2a482/e2e-lifecycle-model-18c2a482-kserve-7586f5c8bc-6257s" Jul 23 09:50:31.373001 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:50:31.372975 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-18c2a482/e2e-lifecycle-model-18c2a482-kserve-7586f5c8bc-6257s" Jul 23 09:50:31.507552 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:50:31.507524 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-18c2a482/e2e-lifecycle-model-18c2a482-kserve-7586f5c8bc-6257s"] Jul 23 09:50:31.509572 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:50:31.509546 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podacdc9e8e_27e5_42dd_ac49_44ab50a33044.slice/crio-7f8dbeba6083c73f2a1c4dfbbafd871c3ffea5531e2f320e6e7d89147834061a WatchSource:0}: Error finding container 7f8dbeba6083c73f2a1c4dfbbafd871c3ffea5531e2f320e6e7d89147834061a: Status 404 returned error can't find the container with id 7f8dbeba6083c73f2a1c4dfbbafd871c3ffea5531e2f320e6e7d89147834061a Jul 23 09:50:31.536982 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:50:31.536943 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-18c2a482/e2e-lifecycle-model-18c2a482-kserve-7586f5c8bc-6257s" event={"ID":"acdc9e8e-27e5-42dd-ac49-44ab50a33044","Type":"ContainerStarted","Data":"7f8dbeba6083c73f2a1c4dfbbafd871c3ffea5531e2f320e6e7d89147834061a"} Jul 23 09:50:32.543803 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:50:32.543763 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-18c2a482/e2e-lifecycle-model-18c2a482-kserve-7586f5c8bc-6257s" event={"ID":"acdc9e8e-27e5-42dd-ac49-44ab50a33044","Type":"ContainerStarted","Data":"65b24ee4ba11a3172c6e5d173b7d43743aea41a7e1faf36d2c113943ffa87e6d"} Jul 23 09:50:32.571018 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:50:32.570966 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-18c2a482/e2e-lifecycle-model-18c2a482-kserve-7586f5c8bc-6257s" podStartSLOduration=1.570951704 podStartE2EDuration="1.570951704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:50:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:50:32.568970706 +0000 UTC m=+2021.108249284" watchObservedRunningTime="2026-07-23 09:50:32.570951704 +0000 UTC m=+2021.110230281" Jul 23 09:50:36.585882 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:50:36.585846 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:50:41.373568 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:50:41.373528 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-18c2a482/e2e-lifecycle-model-18c2a482-kserve-7586f5c8bc-6257s" Jul 23 09:50:41.373568 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:50:41.373574 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-18c2a482/e2e-lifecycle-model-18c2a482-kserve-7586f5c8bc-6257s" Jul 23 09:50:41.386652 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:50:41.386621 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-18c2a482/e2e-lifecycle-model-18c2a482-kserve-7586f5c8bc-6257s" Jul 23 09:50:41.605607 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:50:41.605576 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-18c2a482/e2e-lifecycle-model-18c2a482-kserve-7586f5c8bc-6257s" Jul 23 09:50:51.893798 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:50:51.893760 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:51:08.401366 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:51:08.401310 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:51:08.505129 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:51:08.505095 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:51:09.207508 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:51:09.207473 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:51:09.304111 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:51:09.304065 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:51:12.989055 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:51:12.989020 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:51:14.465745 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:51:14.465651 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:51:18.736412 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:51:18.736367 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:51:21.026356 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:51:21.026302 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:51:22.697208 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:51:22.697169 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:51:26.012649 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:51:26.012614 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-18c2a482/e2e-lifecycle-model-18c2a482-kserve-7586f5c8bc-6257s"] Jul 23 09:51:26.069575 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:51:26.069545 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:51:41.032812 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:51:41.032746 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-18c2a482/e2e-lifecycle-model-18c2a482-kserve-7586f5c8bc-6257s" podUID="acdc9e8e-27e5-42dd-ac49-44ab50a33044" containerName="main" containerID="cri-o://65b24ee4ba11a3172c6e5d173b7d43743aea41a7e1faf36d2c113943ffa87e6d" gracePeriod=45 Jul 23 09:51:41.293559 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:51:41.293496 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-18c2a482/e2e-lifecycle-model-18c2a482-kserve-7586f5c8bc-6257s" Jul 23 09:51:41.348279 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:51:41.348230 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acdc9e8e-27e5-42dd-ac49-44ab50a33044-dshm\") pod \"acdc9e8e-27e5-42dd-ac49-44ab50a33044\" (UID: \"acdc9e8e-27e5-42dd-ac49-44ab50a33044\") " Jul 23 09:51:41.348485 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:51:41.348291 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acdc9e8e-27e5-42dd-ac49-44ab50a33044-home\") pod \"acdc9e8e-27e5-42dd-ac49-44ab50a33044\" (UID: \"acdc9e8e-27e5-42dd-ac49-44ab50a33044\") " Jul 23 09:51:41.348485 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:51:41.348321 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-plc7g\" (UniqueName: \"kubernetes.io/projected/acdc9e8e-27e5-42dd-ac49-44ab50a33044-kube-api-access-plc7g\") pod \"acdc9e8e-27e5-42dd-ac49-44ab50a33044\" (UID: \"acdc9e8e-27e5-42dd-ac49-44ab50a33044\") " Jul 23 09:51:41.348485 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:51:41.348390 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acdc9e8e-27e5-42dd-ac49-44ab50a33044-tls-certs\") pod \"acdc9e8e-27e5-42dd-ac49-44ab50a33044\" (UID: \"acdc9e8e-27e5-42dd-ac49-44ab50a33044\") " Jul 23 09:51:41.348485 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:51:41.348426 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acdc9e8e-27e5-42dd-ac49-44ab50a33044-tmp-dir\") pod \"acdc9e8e-27e5-42dd-ac49-44ab50a33044\" (UID: \"acdc9e8e-27e5-42dd-ac49-44ab50a33044\") " Jul 23 09:51:41.348485 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:51:41.348442 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acdc9e8e-27e5-42dd-ac49-44ab50a33044-model-cache\") pod \"acdc9e8e-27e5-42dd-ac49-44ab50a33044\" (UID: \"acdc9e8e-27e5-42dd-ac49-44ab50a33044\") " Jul 23 09:51:41.348764 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:51:41.348548 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acdc9e8e-27e5-42dd-ac49-44ab50a33044-home" (OuterVolumeSpecName: "home") pod "acdc9e8e-27e5-42dd-ac49-44ab50a33044" (UID: "acdc9e8e-27e5-42dd-ac49-44ab50a33044"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 09:51:41.348764 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:51:41.348710 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acdc9e8e-27e5-42dd-ac49-44ab50a33044-home\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:51:41.348874 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:51:41.348759 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acdc9e8e-27e5-42dd-ac49-44ab50a33044-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "acdc9e8e-27e5-42dd-ac49-44ab50a33044" (UID: "acdc9e8e-27e5-42dd-ac49-44ab50a33044"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 09:51:41.348964 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:51:41.348939 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acdc9e8e-27e5-42dd-ac49-44ab50a33044-model-cache" (OuterVolumeSpecName: "model-cache") pod "acdc9e8e-27e5-42dd-ac49-44ab50a33044" (UID: "acdc9e8e-27e5-42dd-ac49-44ab50a33044"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 09:51:41.350632 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:51:41.350607 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acdc9e8e-27e5-42dd-ac49-44ab50a33044-dshm" (OuterVolumeSpecName: "dshm") pod "acdc9e8e-27e5-42dd-ac49-44ab50a33044" (UID: "acdc9e8e-27e5-42dd-ac49-44ab50a33044"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 09:51:41.350791 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:51:41.350771 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acdc9e8e-27e5-42dd-ac49-44ab50a33044-kube-api-access-plc7g" (OuterVolumeSpecName: "kube-api-access-plc7g") pod "acdc9e8e-27e5-42dd-ac49-44ab50a33044" (UID: "acdc9e8e-27e5-42dd-ac49-44ab50a33044"). InnerVolumeSpecName "kube-api-access-plc7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 09:51:41.350791 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:51:41.350776 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acdc9e8e-27e5-42dd-ac49-44ab50a33044-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "acdc9e8e-27e5-42dd-ac49-44ab50a33044" (UID: "acdc9e8e-27e5-42dd-ac49-44ab50a33044"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 09:51:41.449849 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:51:41.449818 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plc7g\" (UniqueName: \"kubernetes.io/projected/acdc9e8e-27e5-42dd-ac49-44ab50a33044-kube-api-access-plc7g\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:51:41.449849 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:51:41.449846 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acdc9e8e-27e5-42dd-ac49-44ab50a33044-tls-certs\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:51:41.449849 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:51:41.449856 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acdc9e8e-27e5-42dd-ac49-44ab50a33044-tmp-dir\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:51:41.450093 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:51:41.449869 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acdc9e8e-27e5-42dd-ac49-44ab50a33044-model-cache\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:51:41.450093 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:51:41.449882 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acdc9e8e-27e5-42dd-ac49-44ab50a33044-dshm\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:51:41.920518 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:51:41.920479 2576 generic.go:356] "Generic (PLEG): container finished" podID="acdc9e8e-27e5-42dd-ac49-44ab50a33044" containerID="65b24ee4ba11a3172c6e5d173b7d43743aea41a7e1faf36d2c113943ffa87e6d" exitCode=0 Jul 23 09:51:41.920696 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:51:41.920524 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-18c2a482/e2e-lifecycle-model-18c2a482-kserve-7586f5c8bc-6257s" Jul 23 09:51:41.920696 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:51:41.920549 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-18c2a482/e2e-lifecycle-model-18c2a482-kserve-7586f5c8bc-6257s" event={"ID":"acdc9e8e-27e5-42dd-ac49-44ab50a33044","Type":"ContainerDied","Data":"65b24ee4ba11a3172c6e5d173b7d43743aea41a7e1faf36d2c113943ffa87e6d"} Jul 23 09:51:41.920696 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:51:41.920586 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-18c2a482/e2e-lifecycle-model-18c2a482-kserve-7586f5c8bc-6257s" event={"ID":"acdc9e8e-27e5-42dd-ac49-44ab50a33044","Type":"ContainerDied","Data":"7f8dbeba6083c73f2a1c4dfbbafd871c3ffea5531e2f320e6e7d89147834061a"} Jul 23 09:51:41.920696 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:51:41.920608 2576 scope.go:117] "RemoveContainer" containerID="65b24ee4ba11a3172c6e5d173b7d43743aea41a7e1faf36d2c113943ffa87e6d" Jul 23 09:51:41.932952 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:51:41.932848 2576 scope.go:117] "RemoveContainer" containerID="65b24ee4ba11a3172c6e5d173b7d43743aea41a7e1faf36d2c113943ffa87e6d" Jul 23 09:51:41.933391 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:51:41.933369 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65b24ee4ba11a3172c6e5d173b7d43743aea41a7e1faf36d2c113943ffa87e6d\": container with ID starting with 65b24ee4ba11a3172c6e5d173b7d43743aea41a7e1faf36d2c113943ffa87e6d not found: ID does not exist" containerID="65b24ee4ba11a3172c6e5d173b7d43743aea41a7e1faf36d2c113943ffa87e6d" Jul 23 09:51:41.933478 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:51:41.933402 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65b24ee4ba11a3172c6e5d173b7d43743aea41a7e1faf36d2c113943ffa87e6d"} err="failed to get container status \"65b24ee4ba11a3172c6e5d173b7d43743aea41a7e1faf36d2c113943ffa87e6d\": rpc error: code = NotFound desc = could not find container \"65b24ee4ba11a3172c6e5d173b7d43743aea41a7e1faf36d2c113943ffa87e6d\": container with ID starting with 65b24ee4ba11a3172c6e5d173b7d43743aea41a7e1faf36d2c113943ffa87e6d not found: ID does not exist" Jul 23 09:51:41.940965 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:51:41.940941 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-18c2a482/e2e-lifecycle-model-18c2a482-kserve-7586f5c8bc-6257s"] Jul 23 09:51:41.944641 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:51:41.944622 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-18c2a482/e2e-lifecycle-model-18c2a482-kserve-7586f5c8bc-6257s"] Jul 23 09:51:42.985148 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:51:42.985115 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="acdc9e8e-27e5-42dd-ac49-44ab50a33044" path="/var/lib/kubelet/pods/acdc9e8e-27e5-42dd-ac49-44ab50a33044/volumes" Jul 23 09:51:52.045098 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:51:52.045067 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-5szxg_4908aff3-196e-44fa-b8e6-c0e81b476433/console-operator/2.log" Jul 23 09:51:52.049788 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:51:52.049763 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-5szxg_4908aff3-196e-44fa-b8e6-c0e81b476433/console-operator/2.log" Jul 23 09:52:02.835814 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:52:02.835784 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:52:03.041718 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:52:03.041688 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:52:08.541976 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:52:08.541941 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:52:08.614266 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:52:08.614213 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:52:08.830150 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:52:08.830059 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:52:09.067200 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:52:09.067160 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:52:10.170603 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:52:10.170570 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:52:10.281982 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:52:10.281954 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:52:10.404074 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:52:10.404024 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:52:11.128302 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:52:11.128270 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:52:11.229347 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:52:11.229317 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:52:11.331071 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:52:11.331029 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:52:12.826108 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:52:12.826070 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:52:18.461466 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:52:18.461428 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:52:18.562185 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:52:18.562149 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:52:18.689052 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:52:18.689020 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:52:19.141931 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:52:19.141899 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:52:24.936561 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:52:24.936525 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:52:24.996976 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:52:24.996940 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:52:25.456986 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:52:25.456947 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:52:25.526704 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:52:25.526676 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:52:26.796385 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:52:26.796353 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:52:35.666535 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:52:35.666491 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:52:36.011312 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:52:36.011198 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:52:37.475780 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:52:37.475749 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:52:52.124062 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:52:52.123977 2576 scope.go:117] "RemoveContainer" containerID="490bf085af706004fb6f578302be16d690d1d2709261b48823de04125e08c098" Jul 23 09:52:52.132501 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:52:52.132477 2576 scope.go:117] "RemoveContainer" containerID="cf317fdc14f97d931660d1d31e1cb1225f601e9fbaced12ddda21c7ed7af2c13" Jul 23 09:52:56.975163 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:52:56.975127 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:53:03.024278 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:03.024170 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:53:03.123994 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:03.123947 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:53:03.385211 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:03.385177 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:53:03.702856 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:03.702770 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:53:03.778505 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:03.778451 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:53:06.338636 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:06.338606 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:53:06.431059 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:06.430996 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:53:06.497536 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:06.497506 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:53:06.599049 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:06.598927 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:53:07.002988 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:07.002958 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:53:07.102676 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:07.102645 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:53:10.850379 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:10.850309 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:53:11.185440 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:11.185381 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:53:11.504172 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:11.504056 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:53:11.599739 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:11.599703 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:53:11.785473 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:11.785344 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:53:14.646827 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:14.646777 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:53:14.748584 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:14.748552 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:53:14.949418 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:14.949332 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:53:15.131009 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:15.130980 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:53:15.442278 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:15.442234 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:53:15.543393 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:15.543357 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:53:19.159868 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:19.159825 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:53:25.279100 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:25.279068 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-16017a/test-model-16017a-kserve-788f5c766f-xrmsc"] Jul 23 09:53:25.279594 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:25.279438 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acdc9e8e-27e5-42dd-ac49-44ab50a33044" containerName="main" Jul 23 09:53:25.279594 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:25.279450 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="acdc9e8e-27e5-42dd-ac49-44ab50a33044" containerName="main" Jul 23 09:53:25.279594 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:25.279508 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="acdc9e8e-27e5-42dd-ac49-44ab50a33044" containerName="main" Jul 23 09:53:25.282804 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:25.282788 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-16017a/test-model-16017a-kserve-788f5c766f-xrmsc" Jul 23 09:53:25.285746 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:25.285724 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-16017a\"/\"openshift-service-ca.crt\"" Jul 23 09:53:25.285746 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:25.285743 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-16017a\"/\"kube-root-ca.crt\"" Jul 23 09:53:25.285931 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:25.285725 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-16017a\"/\"test-model-16017a-kserve-self-signed-certs\"" Jul 23 09:53:25.285931 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:25.285743 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-16017a\"/\"default-dockercfg-9vbqv\"" Jul 23 09:53:25.292628 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:25.292606 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-16017a/test-model-16017a-kserve-788f5c766f-xrmsc"] Jul 23 09:53:25.296663 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:25.296642 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:53:25.339758 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:25.339735 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7-dshm\") pod \"test-model-16017a-kserve-788f5c766f-xrmsc\" (UID: \"54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7\") " pod="ai-tenant-e2e-inf-a-16017a/test-model-16017a-kserve-788f5c766f-xrmsc" Jul 23 09:53:25.339886 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:25.339781 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zj6v\" (UniqueName: \"kubernetes.io/projected/54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7-kube-api-access-7zj6v\") pod \"test-model-16017a-kserve-788f5c766f-xrmsc\" (UID: \"54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7\") " pod="ai-tenant-e2e-inf-a-16017a/test-model-16017a-kserve-788f5c766f-xrmsc" Jul 23 09:53:25.339886 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:25.339824 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7-tmp-dir\") pod \"test-model-16017a-kserve-788f5c766f-xrmsc\" (UID: \"54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7\") " pod="ai-tenant-e2e-inf-a-16017a/test-model-16017a-kserve-788f5c766f-xrmsc" Jul 23 09:53:25.339886 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:25.339849 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7-model-cache\") pod \"test-model-16017a-kserve-788f5c766f-xrmsc\" (UID: \"54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7\") " pod="ai-tenant-e2e-inf-a-16017a/test-model-16017a-kserve-788f5c766f-xrmsc" Jul 23 09:53:25.339886 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:25.339872 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7-tls-certs\") pod \"test-model-16017a-kserve-788f5c766f-xrmsc\" (UID: \"54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7\") " pod="ai-tenant-e2e-inf-a-16017a/test-model-16017a-kserve-788f5c766f-xrmsc" Jul 23 09:53:25.340040 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:25.339909 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7-home\") pod \"test-model-16017a-kserve-788f5c766f-xrmsc\" (UID: \"54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7\") " pod="ai-tenant-e2e-inf-a-16017a/test-model-16017a-kserve-788f5c766f-xrmsc" Jul 23 09:53:25.440766 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:25.440741 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7-home\") pod \"test-model-16017a-kserve-788f5c766f-xrmsc\" (UID: \"54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7\") " pod="ai-tenant-e2e-inf-a-16017a/test-model-16017a-kserve-788f5c766f-xrmsc" Jul 23 09:53:25.440877 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:25.440789 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7-dshm\") pod \"test-model-16017a-kserve-788f5c766f-xrmsc\" (UID: \"54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7\") " pod="ai-tenant-e2e-inf-a-16017a/test-model-16017a-kserve-788f5c766f-xrmsc" Jul 23 09:53:25.440877 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:25.440829 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zj6v\" (UniqueName: \"kubernetes.io/projected/54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7-kube-api-access-7zj6v\") pod \"test-model-16017a-kserve-788f5c766f-xrmsc\" (UID: \"54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7\") " pod="ai-tenant-e2e-inf-a-16017a/test-model-16017a-kserve-788f5c766f-xrmsc" Jul 23 09:53:25.440877 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:25.440860 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7-tmp-dir\") pod \"test-model-16017a-kserve-788f5c766f-xrmsc\" (UID: \"54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7\") " pod="ai-tenant-e2e-inf-a-16017a/test-model-16017a-kserve-788f5c766f-xrmsc" Jul 23 09:53:25.441030 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:25.440881 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7-model-cache\") pod \"test-model-16017a-kserve-788f5c766f-xrmsc\" (UID: \"54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7\") " pod="ai-tenant-e2e-inf-a-16017a/test-model-16017a-kserve-788f5c766f-xrmsc" Jul 23 09:53:25.441030 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:25.440917 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7-tls-certs\") pod \"test-model-16017a-kserve-788f5c766f-xrmsc\" (UID: \"54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7\") " pod="ai-tenant-e2e-inf-a-16017a/test-model-16017a-kserve-788f5c766f-xrmsc" Jul 23 09:53:25.441186 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:25.441162 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7-home\") pod \"test-model-16017a-kserve-788f5c766f-xrmsc\" (UID: \"54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7\") " pod="ai-tenant-e2e-inf-a-16017a/test-model-16017a-kserve-788f5c766f-xrmsc" Jul 23 09:53:25.441275 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:25.441171 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7-tmp-dir\") pod \"test-model-16017a-kserve-788f5c766f-xrmsc\" (UID: \"54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7\") " pod="ai-tenant-e2e-inf-a-16017a/test-model-16017a-kserve-788f5c766f-xrmsc" Jul 23 09:53:25.441329 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:25.441279 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7-model-cache\") pod \"test-model-16017a-kserve-788f5c766f-xrmsc\" (UID: \"54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7\") " pod="ai-tenant-e2e-inf-a-16017a/test-model-16017a-kserve-788f5c766f-xrmsc" Jul 23 09:53:25.443138 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:25.443113 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7-dshm\") pod \"test-model-16017a-kserve-788f5c766f-xrmsc\" (UID: \"54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7\") " pod="ai-tenant-e2e-inf-a-16017a/test-model-16017a-kserve-788f5c766f-xrmsc" Jul 23 09:53:25.443266 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:25.443233 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7-tls-certs\") pod \"test-model-16017a-kserve-788f5c766f-xrmsc\" (UID: \"54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7\") " pod="ai-tenant-e2e-inf-a-16017a/test-model-16017a-kserve-788f5c766f-xrmsc" Jul 23 09:53:25.449042 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:25.449018 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zj6v\" (UniqueName: \"kubernetes.io/projected/54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7-kube-api-access-7zj6v\") pod \"test-model-16017a-kserve-788f5c766f-xrmsc\" (UID: \"54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7\") " pod="ai-tenant-e2e-inf-a-16017a/test-model-16017a-kserve-788f5c766f-xrmsc" Jul 23 09:53:25.593075 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:25.593004 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-16017a/test-model-16017a-kserve-788f5c766f-xrmsc" Jul 23 09:53:25.726154 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:25.726095 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-16017a/test-model-16017a-kserve-788f5c766f-xrmsc"] Jul 23 09:53:25.728846 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:53:25.728788 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod54f1d4f9_2ad1_4bdf_ae52_36b3a9e8c3b7.slice/crio-8948639a753cd5d062621c823460079860d6c44d8440d6191d1a7abae671144f WatchSource:0}: Error finding container 8948639a753cd5d062621c823460079860d6c44d8440d6191d1a7abae671144f: Status 404 returned error can't find the container with id 8948639a753cd5d062621c823460079860d6c44d8440d6191d1a7abae671144f Jul 23 09:53:26.476487 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:26.476449 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-16017a/test-model-16017a-kserve-788f5c766f-xrmsc" event={"ID":"54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7","Type":"ContainerStarted","Data":"d05a1d5436c9d85dab887167a931804cf38869131e2da11a2a646fe2737aa6ce"} Jul 23 09:53:26.476856 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:26.476497 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-16017a/test-model-16017a-kserve-788f5c766f-xrmsc" event={"ID":"54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7","Type":"ContainerStarted","Data":"8948639a753cd5d062621c823460079860d6c44d8440d6191d1a7abae671144f"} Jul 23 09:53:26.485650 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:26.485624 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:53:26.494945 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:26.494906 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-16017a/test-model-16017a-kserve-788f5c766f-xrmsc" podStartSLOduration=1.494891924 podStartE2EDuration="1.494891924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:53:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:53:26.494419419 +0000 UTC m=+2195.033698000" watchObservedRunningTime="2026-07-23 09:53:26.494891924 +0000 UTC m=+2195.034170502" Jul 23 09:53:27.859224 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:27.859188 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:53:27.960618 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:27.960582 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:53:35.593531 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:35.593492 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-16017a/test-model-16017a-kserve-788f5c766f-xrmsc" Jul 23 09:53:35.593531 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:35.593538 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-16017a/test-model-16017a-kserve-788f5c766f-xrmsc" Jul 23 09:53:35.606033 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:35.605993 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-16017a/test-model-16017a-kserve-788f5c766f-xrmsc" Jul 23 09:53:35.697835 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:35.697785 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:53:36.235841 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:36.235800 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:53:36.336708 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:36.336648 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:53:36.539641 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:36.539552 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-16017a/test-model-16017a-kserve-788f5c766f-xrmsc" Jul 23 09:53:52.158867 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:52.158835 2576 scope.go:117] "RemoveContainer" containerID="398a26f7f4d6852a52570dec201e00cd063811e0b15c33dd92618907f5b28e54" Jul 23 09:53:54.060983 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:54.060948 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-64f44e/test-model-64f44e-kserve-69c49975b7-nhwhz"] Jul 23 09:53:54.065556 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:54.065539 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-64f44e/test-model-64f44e-kserve-69c49975b7-nhwhz" Jul 23 09:53:54.068065 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:54.068045 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-64f44e\"/\"kube-root-ca.crt\"" Jul 23 09:53:54.068888 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:54.068870 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-64f44e\"/\"openshift-service-ca.crt\"" Jul 23 09:53:54.069111 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:54.069098 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-64f44e\"/\"default-dockercfg-52k7b\"" Jul 23 09:53:54.069167 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:54.069142 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-64f44e\"/\"test-model-64f44e-kserve-self-signed-certs\"" Jul 23 09:53:54.078425 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:54.078405 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-64f44e/test-model-64f44e-kserve-69c49975b7-nhwhz"] Jul 23 09:53:54.180295 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:54.180250 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:53:54.202487 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:54.202448 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvhq8\" (UniqueName: \"kubernetes.io/projected/9571b1ec-c7da-4873-9c09-9b3afe0cabd5-kube-api-access-kvhq8\") pod \"test-model-64f44e-kserve-69c49975b7-nhwhz\" (UID: \"9571b1ec-c7da-4873-9c09-9b3afe0cabd5\") " pod="ai-tenant-e2e-inf-b-64f44e/test-model-64f44e-kserve-69c49975b7-nhwhz" Jul 23 09:53:54.202648 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:54.202510 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9571b1ec-c7da-4873-9c09-9b3afe0cabd5-home\") pod \"test-model-64f44e-kserve-69c49975b7-nhwhz\" (UID: \"9571b1ec-c7da-4873-9c09-9b3afe0cabd5\") " pod="ai-tenant-e2e-inf-b-64f44e/test-model-64f44e-kserve-69c49975b7-nhwhz" Jul 23 09:53:54.202648 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:54.202535 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9571b1ec-c7da-4873-9c09-9b3afe0cabd5-tmp-dir\") pod \"test-model-64f44e-kserve-69c49975b7-nhwhz\" (UID: \"9571b1ec-c7da-4873-9c09-9b3afe0cabd5\") " pod="ai-tenant-e2e-inf-b-64f44e/test-model-64f44e-kserve-69c49975b7-nhwhz" Jul 23 09:53:54.202648 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:54.202570 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9571b1ec-c7da-4873-9c09-9b3afe0cabd5-dshm\") pod \"test-model-64f44e-kserve-69c49975b7-nhwhz\" (UID: \"9571b1ec-c7da-4873-9c09-9b3afe0cabd5\") " pod="ai-tenant-e2e-inf-b-64f44e/test-model-64f44e-kserve-69c49975b7-nhwhz" Jul 23 09:53:54.202648 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:54.202627 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9571b1ec-c7da-4873-9c09-9b3afe0cabd5-model-cache\") pod \"test-model-64f44e-kserve-69c49975b7-nhwhz\" (UID: \"9571b1ec-c7da-4873-9c09-9b3afe0cabd5\") " pod="ai-tenant-e2e-inf-b-64f44e/test-model-64f44e-kserve-69c49975b7-nhwhz" Jul 23 09:53:54.202807 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:54.202651 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9571b1ec-c7da-4873-9c09-9b3afe0cabd5-tls-certs\") pod \"test-model-64f44e-kserve-69c49975b7-nhwhz\" (UID: \"9571b1ec-c7da-4873-9c09-9b3afe0cabd5\") " pod="ai-tenant-e2e-inf-b-64f44e/test-model-64f44e-kserve-69c49975b7-nhwhz" Jul 23 09:53:54.304146 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:54.304096 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9571b1ec-c7da-4873-9c09-9b3afe0cabd5-dshm\") pod \"test-model-64f44e-kserve-69c49975b7-nhwhz\" (UID: \"9571b1ec-c7da-4873-9c09-9b3afe0cabd5\") " pod="ai-tenant-e2e-inf-b-64f44e/test-model-64f44e-kserve-69c49975b7-nhwhz" Jul 23 09:53:54.304348 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:54.304165 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9571b1ec-c7da-4873-9c09-9b3afe0cabd5-model-cache\") pod \"test-model-64f44e-kserve-69c49975b7-nhwhz\" (UID: \"9571b1ec-c7da-4873-9c09-9b3afe0cabd5\") " pod="ai-tenant-e2e-inf-b-64f44e/test-model-64f44e-kserve-69c49975b7-nhwhz" Jul 23 09:53:54.304348 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:54.304192 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9571b1ec-c7da-4873-9c09-9b3afe0cabd5-tls-certs\") pod \"test-model-64f44e-kserve-69c49975b7-nhwhz\" (UID: \"9571b1ec-c7da-4873-9c09-9b3afe0cabd5\") " pod="ai-tenant-e2e-inf-b-64f44e/test-model-64f44e-kserve-69c49975b7-nhwhz" Jul 23 09:53:54.304348 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:54.304228 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvhq8\" (UniqueName: \"kubernetes.io/projected/9571b1ec-c7da-4873-9c09-9b3afe0cabd5-kube-api-access-kvhq8\") pod \"test-model-64f44e-kserve-69c49975b7-nhwhz\" (UID: \"9571b1ec-c7da-4873-9c09-9b3afe0cabd5\") " pod="ai-tenant-e2e-inf-b-64f44e/test-model-64f44e-kserve-69c49975b7-nhwhz" Jul 23 09:53:54.304348 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:54.304295 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9571b1ec-c7da-4873-9c09-9b3afe0cabd5-home\") pod \"test-model-64f44e-kserve-69c49975b7-nhwhz\" (UID: \"9571b1ec-c7da-4873-9c09-9b3afe0cabd5\") " pod="ai-tenant-e2e-inf-b-64f44e/test-model-64f44e-kserve-69c49975b7-nhwhz" Jul 23 09:53:54.304348 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:54.304312 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9571b1ec-c7da-4873-9c09-9b3afe0cabd5-tmp-dir\") pod \"test-model-64f44e-kserve-69c49975b7-nhwhz\" (UID: \"9571b1ec-c7da-4873-9c09-9b3afe0cabd5\") " pod="ai-tenant-e2e-inf-b-64f44e/test-model-64f44e-kserve-69c49975b7-nhwhz" Jul 23 09:53:54.304650 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:54.304615 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9571b1ec-c7da-4873-9c09-9b3afe0cabd5-model-cache\") pod \"test-model-64f44e-kserve-69c49975b7-nhwhz\" (UID: \"9571b1ec-c7da-4873-9c09-9b3afe0cabd5\") " pod="ai-tenant-e2e-inf-b-64f44e/test-model-64f44e-kserve-69c49975b7-nhwhz" Jul 23 09:53:54.304722 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:54.304659 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9571b1ec-c7da-4873-9c09-9b3afe0cabd5-tmp-dir\") pod \"test-model-64f44e-kserve-69c49975b7-nhwhz\" (UID: \"9571b1ec-c7da-4873-9c09-9b3afe0cabd5\") " pod="ai-tenant-e2e-inf-b-64f44e/test-model-64f44e-kserve-69c49975b7-nhwhz" Jul 23 09:53:54.304722 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:54.304711 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9571b1ec-c7da-4873-9c09-9b3afe0cabd5-home\") pod \"test-model-64f44e-kserve-69c49975b7-nhwhz\" (UID: \"9571b1ec-c7da-4873-9c09-9b3afe0cabd5\") " pod="ai-tenant-e2e-inf-b-64f44e/test-model-64f44e-kserve-69c49975b7-nhwhz" Jul 23 09:53:54.306532 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:54.306510 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9571b1ec-c7da-4873-9c09-9b3afe0cabd5-dshm\") pod \"test-model-64f44e-kserve-69c49975b7-nhwhz\" (UID: \"9571b1ec-c7da-4873-9c09-9b3afe0cabd5\") " pod="ai-tenant-e2e-inf-b-64f44e/test-model-64f44e-kserve-69c49975b7-nhwhz" Jul 23 09:53:54.306690 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:54.306673 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9571b1ec-c7da-4873-9c09-9b3afe0cabd5-tls-certs\") pod \"test-model-64f44e-kserve-69c49975b7-nhwhz\" (UID: \"9571b1ec-c7da-4873-9c09-9b3afe0cabd5\") " pod="ai-tenant-e2e-inf-b-64f44e/test-model-64f44e-kserve-69c49975b7-nhwhz" Jul 23 09:53:54.313605 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:54.313548 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvhq8\" (UniqueName: \"kubernetes.io/projected/9571b1ec-c7da-4873-9c09-9b3afe0cabd5-kube-api-access-kvhq8\") pod \"test-model-64f44e-kserve-69c49975b7-nhwhz\" (UID: \"9571b1ec-c7da-4873-9c09-9b3afe0cabd5\") " pod="ai-tenant-e2e-inf-b-64f44e/test-model-64f44e-kserve-69c49975b7-nhwhz" Jul 23 09:53:54.374165 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:54.374131 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-64f44e/test-model-64f44e-kserve-69c49975b7-nhwhz" Jul 23 09:53:54.507261 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:54.507215 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-64f44e/test-model-64f44e-kserve-69c49975b7-nhwhz"] Jul 23 09:53:54.508813 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:53:54.508789 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9571b1ec_c7da_4873_9c09_9b3afe0cabd5.slice/crio-bc222a888c6dc959db62a75e955d09e341f03801c69e9ec5006e9bdefd12971d WatchSource:0}: Error finding container bc222a888c6dc959db62a75e955d09e341f03801c69e9ec5006e9bdefd12971d: Status 404 returned error can't find the container with id bc222a888c6dc959db62a75e955d09e341f03801c69e9ec5006e9bdefd12971d Jul 23 09:53:54.622136 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:54.622090 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-64f44e/test-model-64f44e-kserve-69c49975b7-nhwhz" event={"ID":"9571b1ec-c7da-4873-9c09-9b3afe0cabd5","Type":"ContainerStarted","Data":"8513a2164a06a26f7414344ac64602190fef7a739c3bed959905391afb2a9cdd"} Jul 23 09:53:54.622136 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:54.622133 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-64f44e/test-model-64f44e-kserve-69c49975b7-nhwhz" event={"ID":"9571b1ec-c7da-4873-9c09-9b3afe0cabd5","Type":"ContainerStarted","Data":"bc222a888c6dc959db62a75e955d09e341f03801c69e9ec5006e9bdefd12971d"} Jul 23 09:53:54.642793 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:54.642743 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-64f44e/test-model-64f44e-kserve-69c49975b7-nhwhz" podStartSLOduration=0.642728833 podStartE2EDuration="642.728833ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:53:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:53:54.640308467 +0000 UTC m=+2223.179587043" watchObservedRunningTime="2026-07-23 09:53:54.642728833 +0000 UTC m=+2223.182007447" Jul 23 09:53:58.186421 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:53:58.186386 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:54:04.374863 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:04.374822 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-64f44e/test-model-64f44e-kserve-69c49975b7-nhwhz" Jul 23 09:54:04.374863 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:04.374869 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-64f44e/test-model-64f44e-kserve-69c49975b7-nhwhz" Jul 23 09:54:04.388050 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:04.388018 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-64f44e/test-model-64f44e-kserve-69c49975b7-nhwhz" Jul 23 09:54:04.687864 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:04.687782 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-64f44e/test-model-64f44e-kserve-69c49975b7-nhwhz" Jul 23 09:54:12.086675 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:12.086635 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:54:34.064646 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:34.064608 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-16017a/test-model-16017a-kserve-788f5c766f-xrmsc"] Jul 23 09:54:34.082371 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:34.082336 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:54:34.977003 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:34.976970 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-64f44e/test-model-64f44e-kserve-69c49975b7-nhwhz"] Jul 23 09:54:35.002050 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:35.002016 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:54:35.303225 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:35.303138 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:54:35.784255 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:35.784218 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:54:35.880161 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:35.880131 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:54:38.820390 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:38.820355 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:54:45.583586 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:45.583552 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:54:49.085602 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:49.085556 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-16017a/test-model-16017a-kserve-788f5c766f-xrmsc" podUID="54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7" containerName="main" containerID="cri-o://d05a1d5436c9d85dab887167a931804cf38869131e2da11a2a646fe2737aa6ce" gracePeriod=45 Jul 23 09:54:49.340714 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:49.340657 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-16017a/test-model-16017a-kserve-788f5c766f-xrmsc" Jul 23 09:54:49.416591 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:49.416559 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7-dshm\") pod \"54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7\" (UID: \"54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7\") " Jul 23 09:54:49.416778 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:49.416661 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7-model-cache\") pod \"54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7\" (UID: \"54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7\") " Jul 23 09:54:49.416778 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:49.416689 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7zj6v\" (UniqueName: \"kubernetes.io/projected/54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7-kube-api-access-7zj6v\") pod \"54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7\" (UID: \"54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7\") " Jul 23 09:54:49.416778 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:49.416710 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7-tls-certs\") pod \"54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7\" (UID: \"54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7\") " Jul 23 09:54:49.416778 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:49.416730 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7-home\") pod \"54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7\" (UID: \"54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7\") " Jul 23 09:54:49.416778 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:49.416769 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7-tmp-dir\") pod \"54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7\" (UID: \"54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7\") " Jul 23 09:54:49.417038 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:49.416977 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7-model-cache" (OuterVolumeSpecName: "model-cache") pod "54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7" (UID: "54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 09:54:49.417038 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:49.417003 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7-home" (OuterVolumeSpecName: "home") pod "54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7" (UID: "54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 09:54:49.417166 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:49.417143 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7" (UID: "54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 09:54:49.418972 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:49.418942 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7-dshm" (OuterVolumeSpecName: "dshm") pod "54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7" (UID: "54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 09:54:49.419096 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:49.419000 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7-kube-api-access-7zj6v" (OuterVolumeSpecName: "kube-api-access-7zj6v") pod "54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7" (UID: "54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7"). InnerVolumeSpecName "kube-api-access-7zj6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 09:54:49.419096 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:49.419001 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7" (UID: "54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 09:54:49.518364 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:49.518323 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7-model-cache\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:54:49.518364 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:49.518356 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zj6v\" (UniqueName: \"kubernetes.io/projected/54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7-kube-api-access-7zj6v\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:54:49.518364 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:49.518371 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7-tls-certs\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:54:49.518620 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:49.518385 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7-home\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:54:49.518620 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:49.518397 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7-tmp-dir\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:54:49.518620 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:49.518406 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7-dshm\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:54:49.923011 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:49.922977 2576 generic.go:356] "Generic (PLEG): container finished" podID="54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7" containerID="d05a1d5436c9d85dab887167a931804cf38869131e2da11a2a646fe2737aa6ce" exitCode=0 Jul 23 09:54:49.923230 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:49.923027 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-16017a/test-model-16017a-kserve-788f5c766f-xrmsc" Jul 23 09:54:49.923230 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:49.923066 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-16017a/test-model-16017a-kserve-788f5c766f-xrmsc" event={"ID":"54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7","Type":"ContainerDied","Data":"d05a1d5436c9d85dab887167a931804cf38869131e2da11a2a646fe2737aa6ce"} Jul 23 09:54:49.923230 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:49.923117 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-16017a/test-model-16017a-kserve-788f5c766f-xrmsc" event={"ID":"54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7","Type":"ContainerDied","Data":"8948639a753cd5d062621c823460079860d6c44d8440d6191d1a7abae671144f"} Jul 23 09:54:49.923230 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:49.923142 2576 scope.go:117] "RemoveContainer" containerID="d05a1d5436c9d85dab887167a931804cf38869131e2da11a2a646fe2737aa6ce" Jul 23 09:54:49.934169 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:49.934150 2576 scope.go:117] "RemoveContainer" containerID="d05a1d5436c9d85dab887167a931804cf38869131e2da11a2a646fe2737aa6ce" Jul 23 09:54:49.934453 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:54:49.934430 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d05a1d5436c9d85dab887167a931804cf38869131e2da11a2a646fe2737aa6ce\": container with ID starting with d05a1d5436c9d85dab887167a931804cf38869131e2da11a2a646fe2737aa6ce not found: ID does not exist" containerID="d05a1d5436c9d85dab887167a931804cf38869131e2da11a2a646fe2737aa6ce" Jul 23 09:54:49.934547 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:49.934459 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d05a1d5436c9d85dab887167a931804cf38869131e2da11a2a646fe2737aa6ce"} err="failed to get container status \"d05a1d5436c9d85dab887167a931804cf38869131e2da11a2a646fe2737aa6ce\": rpc error: code = NotFound desc = could not find container \"d05a1d5436c9d85dab887167a931804cf38869131e2da11a2a646fe2737aa6ce\": container with ID starting with d05a1d5436c9d85dab887167a931804cf38869131e2da11a2a646fe2737aa6ce not found: ID does not exist" Jul 23 09:54:49.956643 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:49.956611 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-16017a/test-model-16017a-kserve-788f5c766f-xrmsc"] Jul 23 09:54:49.960936 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:49.960912 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-16017a/test-model-16017a-kserve-788f5c766f-xrmsc"] Jul 23 09:54:49.997997 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:49.997966 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-64f44e/test-model-64f44e-kserve-69c49975b7-nhwhz" podUID="9571b1ec-c7da-4873-9c09-9b3afe0cabd5" containerName="main" containerID="cri-o://8513a2164a06a26f7414344ac64602190fef7a739c3bed959905391afb2a9cdd" gracePeriod=45 Jul 23 09:54:50.251782 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:50.251755 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:54:50.260759 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:50.260737 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-64f44e/test-model-64f44e-kserve-69c49975b7-nhwhz" Jul 23 09:54:50.325610 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:50.325571 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9571b1ec-c7da-4873-9c09-9b3afe0cabd5-home\") pod \"9571b1ec-c7da-4873-9c09-9b3afe0cabd5\" (UID: \"9571b1ec-c7da-4873-9c09-9b3afe0cabd5\") " Jul 23 09:54:50.325800 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:50.325646 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9571b1ec-c7da-4873-9c09-9b3afe0cabd5-tmp-dir\") pod \"9571b1ec-c7da-4873-9c09-9b3afe0cabd5\" (UID: \"9571b1ec-c7da-4873-9c09-9b3afe0cabd5\") " Jul 23 09:54:50.325800 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:50.325690 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9571b1ec-c7da-4873-9c09-9b3afe0cabd5-tls-certs\") pod \"9571b1ec-c7da-4873-9c09-9b3afe0cabd5\" (UID: \"9571b1ec-c7da-4873-9c09-9b3afe0cabd5\") " Jul 23 09:54:50.325800 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:50.325715 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9571b1ec-c7da-4873-9c09-9b3afe0cabd5-dshm\") pod \"9571b1ec-c7da-4873-9c09-9b3afe0cabd5\" (UID: \"9571b1ec-c7da-4873-9c09-9b3afe0cabd5\") " Jul 23 09:54:50.325800 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:50.325752 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9571b1ec-c7da-4873-9c09-9b3afe0cabd5-model-cache\") pod \"9571b1ec-c7da-4873-9c09-9b3afe0cabd5\" (UID: \"9571b1ec-c7da-4873-9c09-9b3afe0cabd5\") " Jul 23 09:54:50.325800 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:50.325796 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kvhq8\" (UniqueName: \"kubernetes.io/projected/9571b1ec-c7da-4873-9c09-9b3afe0cabd5-kube-api-access-kvhq8\") pod \"9571b1ec-c7da-4873-9c09-9b3afe0cabd5\" (UID: \"9571b1ec-c7da-4873-9c09-9b3afe0cabd5\") " Jul 23 09:54:50.326107 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:50.325889 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9571b1ec-c7da-4873-9c09-9b3afe0cabd5-home" (OuterVolumeSpecName: "home") pod "9571b1ec-c7da-4873-9c09-9b3afe0cabd5" (UID: "9571b1ec-c7da-4873-9c09-9b3afe0cabd5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 09:54:50.326107 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:50.325931 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9571b1ec-c7da-4873-9c09-9b3afe0cabd5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9571b1ec-c7da-4873-9c09-9b3afe0cabd5" (UID: "9571b1ec-c7da-4873-9c09-9b3afe0cabd5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 09:54:50.326107 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:50.326039 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9571b1ec-c7da-4873-9c09-9b3afe0cabd5-model-cache" (OuterVolumeSpecName: "model-cache") pod "9571b1ec-c7da-4873-9c09-9b3afe0cabd5" (UID: "9571b1ec-c7da-4873-9c09-9b3afe0cabd5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 09:54:50.326280 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:50.326128 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9571b1ec-c7da-4873-9c09-9b3afe0cabd5-model-cache\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:54:50.326280 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:50.326144 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9571b1ec-c7da-4873-9c09-9b3afe0cabd5-home\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:54:50.326280 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:50.326161 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9571b1ec-c7da-4873-9c09-9b3afe0cabd5-tmp-dir\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:54:50.328031 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:50.328000 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9571b1ec-c7da-4873-9c09-9b3afe0cabd5-kube-api-access-kvhq8" (OuterVolumeSpecName: "kube-api-access-kvhq8") pod "9571b1ec-c7da-4873-9c09-9b3afe0cabd5" (UID: "9571b1ec-c7da-4873-9c09-9b3afe0cabd5"). InnerVolumeSpecName "kube-api-access-kvhq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 09:54:50.328152 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:50.328034 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9571b1ec-c7da-4873-9c09-9b3afe0cabd5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9571b1ec-c7da-4873-9c09-9b3afe0cabd5" (UID: "9571b1ec-c7da-4873-9c09-9b3afe0cabd5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 09:54:50.328152 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:50.328034 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9571b1ec-c7da-4873-9c09-9b3afe0cabd5-dshm" (OuterVolumeSpecName: "dshm") pod "9571b1ec-c7da-4873-9c09-9b3afe0cabd5" (UID: "9571b1ec-c7da-4873-9c09-9b3afe0cabd5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 09:54:50.426647 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:50.426614 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9571b1ec-c7da-4873-9c09-9b3afe0cabd5-tls-certs\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:54:50.426647 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:50.426646 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9571b1ec-c7da-4873-9c09-9b3afe0cabd5-dshm\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:54:50.426858 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:50.426659 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvhq8\" (UniqueName: \"kubernetes.io/projected/9571b1ec-c7da-4873-9c09-9b3afe0cabd5-kube-api-access-kvhq8\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:54:50.696392 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:50.696351 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:54:50.929149 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:50.929115 2576 generic.go:356] "Generic (PLEG): container finished" podID="9571b1ec-c7da-4873-9c09-9b3afe0cabd5" containerID="8513a2164a06a26f7414344ac64602190fef7a739c3bed959905391afb2a9cdd" exitCode=0 Jul 23 09:54:50.929358 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:50.929164 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-64f44e/test-model-64f44e-kserve-69c49975b7-nhwhz" Jul 23 09:54:50.929358 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:50.929200 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-64f44e/test-model-64f44e-kserve-69c49975b7-nhwhz" event={"ID":"9571b1ec-c7da-4873-9c09-9b3afe0cabd5","Type":"ContainerDied","Data":"8513a2164a06a26f7414344ac64602190fef7a739c3bed959905391afb2a9cdd"} Jul 23 09:54:50.929358 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:50.929267 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-64f44e/test-model-64f44e-kserve-69c49975b7-nhwhz" event={"ID":"9571b1ec-c7da-4873-9c09-9b3afe0cabd5","Type":"ContainerDied","Data":"bc222a888c6dc959db62a75e955d09e341f03801c69e9ec5006e9bdefd12971d"} Jul 23 09:54:50.929358 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:50.929293 2576 scope.go:117] "RemoveContainer" containerID="8513a2164a06a26f7414344ac64602190fef7a739c3bed959905391afb2a9cdd" Jul 23 09:54:50.939099 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:50.939082 2576 scope.go:117] "RemoveContainer" containerID="8513a2164a06a26f7414344ac64602190fef7a739c3bed959905391afb2a9cdd" Jul 23 09:54:50.939368 ip-10-0-135-145 kubenswrapper[2576]: E0723 09:54:50.939349 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8513a2164a06a26f7414344ac64602190fef7a739c3bed959905391afb2a9cdd\": container with ID starting with 8513a2164a06a26f7414344ac64602190fef7a739c3bed959905391afb2a9cdd not found: ID does not exist" containerID="8513a2164a06a26f7414344ac64602190fef7a739c3bed959905391afb2a9cdd" Jul 23 09:54:50.939446 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:50.939379 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8513a2164a06a26f7414344ac64602190fef7a739c3bed959905391afb2a9cdd"} err="failed to get container status \"8513a2164a06a26f7414344ac64602190fef7a739c3bed959905391afb2a9cdd\": rpc error: code = NotFound desc = could not find container \"8513a2164a06a26f7414344ac64602190fef7a739c3bed959905391afb2a9cdd\": container with ID starting with 8513a2164a06a26f7414344ac64602190fef7a739c3bed959905391afb2a9cdd not found: ID does not exist" Jul 23 09:54:50.958119 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:50.958058 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-64f44e/test-model-64f44e-kserve-69c49975b7-nhwhz"] Jul 23 09:54:50.963104 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:50.963080 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-64f44e/test-model-64f44e-kserve-69c49975b7-nhwhz"] Jul 23 09:54:50.985132 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:50.985106 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7" path="/var/lib/kubelet/pods/54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7/volumes" Jul 23 09:54:50.985560 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:50.985545 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9571b1ec-c7da-4873-9c09-9b3afe0cabd5" path="/var/lib/kubelet/pods/9571b1ec-c7da-4873-9c09-9b3afe0cabd5/volumes" Jul 23 09:54:54.509863 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:54.509826 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:54:57.659904 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:57.659851 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:54:57.781348 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:57.781307 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:54:58.055506 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:58.055411 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:54:58.277427 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:58.277391 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:54:58.372046 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:58.371964 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-64f44e-pbbms"] Jul 23 09:54:58.372347 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:58.372334 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7" containerName="main" Jul 23 09:54:58.372392 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:58.372348 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7" containerName="main" Jul 23 09:54:58.372392 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:58.372361 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9571b1ec-c7da-4873-9c09-9b3afe0cabd5" containerName="main" Jul 23 09:54:58.372392 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:58.372367 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9571b1ec-c7da-4873-9c09-9b3afe0cabd5" containerName="main" Jul 23 09:54:58.372496 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:58.372433 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9571b1ec-c7da-4873-9c09-9b3afe0cabd5" containerName="main" Jul 23 09:54:58.372496 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:58.372446 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="54f1d4f9-2ad1-4bdf-ae52-36b3a9e8c3b7" containerName="main" Jul 23 09:54:58.375477 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:58.375459 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-64f44e-pbbms" Jul 23 09:54:58.379036 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:58.379015 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n4hmw\"" Jul 23 09:54:58.379157 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:58.379015 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 09:54:58.390624 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:58.390592 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-64f44e-pbbms"] Jul 23 09:54:58.500668 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:58.500628 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86984fe0-b332-486a-a2f7-137e703b5171-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-64f44e-pbbms\" (UID: \"86984fe0-b332-486a-a2f7-137e703b5171\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-64f44e-pbbms" Jul 23 09:54:58.601208 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:58.601163 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86984fe0-b332-486a-a2f7-137e703b5171-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-64f44e-pbbms\" (UID: \"86984fe0-b332-486a-a2f7-137e703b5171\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-64f44e-pbbms" Jul 23 09:54:58.601837 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:58.601815 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86984fe0-b332-486a-a2f7-137e703b5171-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-64f44e-pbbms\" (UID: \"86984fe0-b332-486a-a2f7-137e703b5171\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-64f44e-pbbms" Jul 23 09:54:58.686064 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:58.686004 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-64f44e-pbbms" Jul 23 09:54:58.854397 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:58.854366 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-64f44e-pbbms"] Jul 23 09:54:58.856457 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:54:58.856425 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86984fe0_b332_486a_a2f7_137e703b5171.slice/crio-b7eaed11485f9b19adb685faf0f57c78de846ffd1d3152491be0f6d133c5c8e2 WatchSource:0}: Error finding container b7eaed11485f9b19adb685faf0f57c78de846ffd1d3152491be0f6d133c5c8e2: Status 404 returned error can't find the container with id b7eaed11485f9b19adb685faf0f57c78de846ffd1d3152491be0f6d133c5c8e2 Jul 23 09:54:58.986271 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:58.986166 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-64f44e-pbbms" event={"ID":"86984fe0-b332-486a-a2f7-137e703b5171","Type":"ContainerStarted","Data":"b7eaed11485f9b19adb685faf0f57c78de846ffd1d3152491be0f6d133c5c8e2"} Jul 23 09:54:59.339638 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:59.339534 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:54:59.442075 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:59.442037 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:54:59.991472 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:59.991429 2576 generic.go:356] "Generic (PLEG): container finished" podID="86984fe0-b332-486a-a2f7-137e703b5171" containerID="d7bd770a2c989acd8919e6bef7d0e2f8596bfb5dd3a68e96fc2a0349c852cc31" exitCode=0 Jul 23 09:54:59.991876 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:54:59.991494 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-64f44e-pbbms" event={"ID":"86984fe0-b332-486a-a2f7-137e703b5171","Type":"ContainerDied","Data":"d7bd770a2c989acd8919e6bef7d0e2f8596bfb5dd3a68e96fc2a0349c852cc31"} Jul 23 09:55:01.136573 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:55:01.136542 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-64f44e-pbbms" Jul 23 09:55:01.227813 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:55:01.227767 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86984fe0-b332-486a-a2f7-137e703b5171-maas-api-service-ca\") pod \"86984fe0-b332-486a-a2f7-137e703b5171\" (UID: \"86984fe0-b332-486a-a2f7-137e703b5171\") " Jul 23 09:55:01.228127 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:55:01.228102 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86984fe0-b332-486a-a2f7-137e703b5171-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "86984fe0-b332-486a-a2f7-137e703b5171" (UID: "86984fe0-b332-486a-a2f7-137e703b5171"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 09:55:01.329028 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:55:01.328930 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86984fe0-b332-486a-a2f7-137e703b5171-maas-api-service-ca\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:55:02.002899 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:55:02.002868 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-64f44e-pbbms" Jul 23 09:55:02.003085 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:55:02.002908 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-64f44e-pbbms" event={"ID":"86984fe0-b332-486a-a2f7-137e703b5171","Type":"ContainerDied","Data":"b7eaed11485f9b19adb685faf0f57c78de846ffd1d3152491be0f6d133c5c8e2"} Jul 23 09:55:02.003085 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:55:02.002955 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7eaed11485f9b19adb685faf0f57c78de846ffd1d3152491be0f6d133c5c8e2" Jul 23 09:55:09.156841 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:55:09.156806 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:55:11.827642 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:55:11.827609 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:55:13.490096 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:55:13.490065 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-64f44e-pbbms"] Jul 23 09:55:13.490517 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:55:13.490173 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:55:13.494726 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:55:13.494703 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-64f44e-pbbms"] Jul 23 09:55:13.906861 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:55:13.906818 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:55:14.078918 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:55:14.078886 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:55:14.985497 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:55:14.985460 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86984fe0-b332-486a-a2f7-137e703b5171" path="/var/lib/kubelet/pods/86984fe0-b332-486a-a2f7-137e703b5171/volumes" Jul 23 09:55:20.522741 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:55:20.522711 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:55:20.659157 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:55:20.659129 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:55:20.863833 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:55:20.863745 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:55:21.081921 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:55:21.081882 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:55:21.435787 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:55:21.435759 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:55:21.818170 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:55:21.818094 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:55:21.913566 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:55:21.913531 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:55:25.261055 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:55:25.261021 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:55:25.532434 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:55:25.532362 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:55:25.719848 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:55:25.719811 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:55:25.887230 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:55:25.887201 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:55:26.891933 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:55:26.891899 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:55:27.060115 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:55:27.060088 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:55:29.812744 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:55:29.812705 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:55:30.323647 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:55:30.323616 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:55:30.801891 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:55:30.801864 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:55:30.961426 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:55:30.961383 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:55:32.033953 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:55:32.033917 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:55:33.543110 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:55:33.543074 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:55:33.673980 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:55:33.673951 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:55:33.952194 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:55:33.952160 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:55:34.244147 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:55:34.244069 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:55:34.885517 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:55:34.885489 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:55:35.067140 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:55:35.067113 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:55:37.996324 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:55:37.996294 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:55:47.035786 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:55:47.035714 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:55:47.242958 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:55:47.242925 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:55:48.965849 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:55:48.965816 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:55:56.099929 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:55:56.099888 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:55:56.214011 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:55:56.213977 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:56:01.476808 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:56:01.476765 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:56:05.232212 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:56:05.232170 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:56:06.276444 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:56:06.276408 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:56:06.705552 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:56:06.705523 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:56:09.360256 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:56:09.360212 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:56:09.456365 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:56:09.456330 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:56:12.791878 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:56:12.791848 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:56:12.893036 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:56:12.893000 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:56:13.054233 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:56:13.054153 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:56:13.430953 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:56:13.430918 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-10b365-l9j69"] Jul 23 09:56:13.431297 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:56:13.431285 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86984fe0-b332-486a-a2f7-137e703b5171" containerName="revoke-keys" Jul 23 09:56:13.431350 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:56:13.431299 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="86984fe0-b332-486a-a2f7-137e703b5171" containerName="revoke-keys" Jul 23 09:56:13.431385 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:56:13.431380 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="86984fe0-b332-486a-a2f7-137e703b5171" containerName="revoke-keys" Jul 23 09:56:13.434441 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:56:13.434423 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-10b365-l9j69" Jul 23 09:56:13.436914 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:56:13.436894 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 09:56:13.437133 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:56:13.437077 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n4hmw\"" Jul 23 09:56:13.443117 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:56:13.443092 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-10b365-l9j69"] Jul 23 09:56:13.465808 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:56:13.465770 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/208f21bf-81a3-4b77-aa9b-76f41b16325d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-10b365-l9j69\" (UID: \"208f21bf-81a3-4b77-aa9b-76f41b16325d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-10b365-l9j69" Jul 23 09:56:13.566518 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:56:13.566479 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/208f21bf-81a3-4b77-aa9b-76f41b16325d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-10b365-l9j69\" (UID: \"208f21bf-81a3-4b77-aa9b-76f41b16325d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-10b365-l9j69" Jul 23 09:56:13.567159 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:56:13.567138 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/208f21bf-81a3-4b77-aa9b-76f41b16325d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-10b365-l9j69\" (UID: \"208f21bf-81a3-4b77-aa9b-76f41b16325d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-10b365-l9j69" Jul 23 09:56:13.704969 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:56:13.704886 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:56:13.746032 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:56:13.746000 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-10b365-l9j69" Jul 23 09:56:13.881143 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:56:13.881116 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-10b365-l9j69"] Jul 23 09:56:13.883192 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:56:13.883151 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod208f21bf_81a3_4b77_aa9b_76f41b16325d.slice/crio-3d5489b0f3a8f4163335f1c7b08a38790c3b62ef94fd200ee38996a8352f9c7f WatchSource:0}: Error finding container 3d5489b0f3a8f4163335f1c7b08a38790c3b62ef94fd200ee38996a8352f9c7f: Status 404 returned error can't find the container with id 3d5489b0f3a8f4163335f1c7b08a38790c3b62ef94fd200ee38996a8352f9c7f Jul 23 09:56:13.954421 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:56:13.954378 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:56:14.053970 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:56:14.053847 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:56:14.375318 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:56:14.375289 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-10b365-l9j69" event={"ID":"208f21bf-81a3-4b77-aa9b-76f41b16325d","Type":"ContainerStarted","Data":"31c6051180864f4fb268ef5279f7cbde35de5faba3d5bb55e43736b0760bf46b"} Jul 23 09:56:14.375494 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:56:14.375337 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-10b365-l9j69" event={"ID":"208f21bf-81a3-4b77-aa9b-76f41b16325d","Type":"ContainerStarted","Data":"3d5489b0f3a8f4163335f1c7b08a38790c3b62ef94fd200ee38996a8352f9c7f"} Jul 23 09:56:14.392292 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:56:14.392227 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-10b365-l9j69" podStartSLOduration=1.3922124390000001 podStartE2EDuration="1.392212439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:56:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:56:14.390289486 +0000 UTC m=+2362.929568062" watchObservedRunningTime="2026-07-23 09:56:14.392212439 +0000 UTC m=+2362.931491016" Jul 23 09:56:15.382483 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:56:15.382447 2576 generic.go:356] "Generic (PLEG): container finished" podID="208f21bf-81a3-4b77-aa9b-76f41b16325d" containerID="31c6051180864f4fb268ef5279f7cbde35de5faba3d5bb55e43736b0760bf46b" exitCode=0 Jul 23 09:56:15.382889 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:56:15.382541 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-10b365-l9j69" event={"ID":"208f21bf-81a3-4b77-aa9b-76f41b16325d","Type":"ContainerDied","Data":"31c6051180864f4fb268ef5279f7cbde35de5faba3d5bb55e43736b0760bf46b"} Jul 23 09:56:16.521439 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:56:16.521413 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-10b365-l9j69" Jul 23 09:56:16.588718 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:56:16.588682 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/208f21bf-81a3-4b77-aa9b-76f41b16325d-maas-api-service-ca\") pod \"208f21bf-81a3-4b77-aa9b-76f41b16325d\" (UID: \"208f21bf-81a3-4b77-aa9b-76f41b16325d\") " Jul 23 09:56:16.589082 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:56:16.589060 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/208f21bf-81a3-4b77-aa9b-76f41b16325d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "208f21bf-81a3-4b77-aa9b-76f41b16325d" (UID: "208f21bf-81a3-4b77-aa9b-76f41b16325d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 09:56:16.689742 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:56:16.689654 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/208f21bf-81a3-4b77-aa9b-76f41b16325d-maas-api-service-ca\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 09:56:17.398005 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:56:17.397972 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-10b365-l9j69" Jul 23 09:56:17.398203 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:56:17.398005 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-10b365-l9j69" event={"ID":"208f21bf-81a3-4b77-aa9b-76f41b16325d","Type":"ContainerDied","Data":"3d5489b0f3a8f4163335f1c7b08a38790c3b62ef94fd200ee38996a8352f9c7f"} Jul 23 09:56:17.398203 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:56:17.398051 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d5489b0f3a8f4163335f1c7b08a38790c3b62ef94fd200ee38996a8352f9c7f" Jul 23 09:56:28.842791 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:56:28.842753 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:56:33.495528 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:56:33.495495 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:56:33.502254 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:56:33.502212 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-10b365-l9j69"] Jul 23 09:56:33.504385 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:56:33.504366 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-10b365-l9j69"] Jul 23 09:56:33.584644 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:56:33.584608 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:56:33.902423 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:56:33.902389 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:56:34.986936 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:56:34.986891 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="208f21bf-81a3-4b77-aa9b-76f41b16325d" path="/var/lib/kubelet/pods/208f21bf-81a3-4b77-aa9b-76f41b16325d/volumes" Jul 23 09:56:37.383003 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:56:37.382970 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:56:37.462398 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:56:37.462370 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:56:39.672188 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:56:39.672151 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:56:39.773005 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:56:39.772965 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:56:39.968776 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:56:39.968689 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:56:40.206165 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:56:40.206132 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:56:40.569798 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:56:40.569765 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:56:40.915682 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:56:40.915653 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:56:41.225323 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:56:41.225197 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:56:43.529748 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:56:43.529710 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:56:43.630077 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:56:43.630045 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:56:43.733645 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:56:43.733584 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:56:44.207966 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:56:44.207932 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:56:44.303187 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:56:44.303158 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:56:44.408770 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:56:44.408742 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:56:47.996397 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:56:47.996368 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:56:48.047760 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:56:48.047732 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:56:48.389446 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:56:48.389414 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:56:48.721472 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:56:48.721390 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:56:48.960260 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:56:48.960210 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:56:51.670280 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:56:51.670222 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:56:51.761589 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:56:51.761522 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:56:52.040385 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:56:52.040296 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:56:52.080728 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:56:52.080703 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-5szxg_4908aff3-196e-44fa-b8e6-c0e81b476433/console-operator/2.log" Jul 23 09:56:52.087175 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:56:52.087150 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-5szxg_4908aff3-196e-44fa-b8e6-c0e81b476433/console-operator/2.log" Jul 23 09:56:52.500004 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:56:52.499972 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:56:52.600981 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:56:52.600947 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:56:52.701415 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:56:52.701383 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:56:56.343119 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:56:56.343086 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:56:58.901226 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:56:58.901186 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:57:05.030627 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:05.030595 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:57:05.130827 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:05.130795 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:57:14.165442 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:14.165353 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-efdc11/auth-test-model-efdc11-kserve-65c9899bcf-n4k7p"] Jul 23 09:57:14.165808 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:14.165725 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="208f21bf-81a3-4b77-aa9b-76f41b16325d" containerName="revoke-keys" Jul 23 09:57:14.165808 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:14.165736 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="208f21bf-81a3-4b77-aa9b-76f41b16325d" containerName="revoke-keys" Jul 23 09:57:14.165808 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:14.165801 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="208f21bf-81a3-4b77-aa9b-76f41b16325d" containerName="revoke-keys" Jul 23 09:57:14.169011 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:14.168993 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-efdc11/auth-test-model-efdc11-kserve-65c9899bcf-n4k7p" Jul 23 09:57:14.171404 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:14.171377 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-efdc11\"/\"kube-root-ca.crt\"" Jul 23 09:57:14.171519 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:14.171423 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-efdc11\"/\"openshift-service-ca.crt\"" Jul 23 09:57:14.172160 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:14.172144 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-efdc11\"/\"default-dockercfg-ncgz9\"" Jul 23 09:57:14.172260 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:14.172151 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-efdc11\"/\"auth-test-model-efdc11-kserve-self-signed-certs\"" Jul 23 09:57:14.178200 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:14.178163 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-efdc11/auth-test-model-efdc11-kserve-65c9899bcf-n4k7p"] Jul 23 09:57:14.195971 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:14.195947 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:57:14.214174 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:14.214148 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc5d5dc2-b24b-49c4-8a72-335242f92896-home\") pod \"auth-test-model-efdc11-kserve-65c9899bcf-n4k7p\" (UID: \"fc5d5dc2-b24b-49c4-8a72-335242f92896\") " pod="ai-tenant-e2e-shared-a-efdc11/auth-test-model-efdc11-kserve-65c9899bcf-n4k7p" Jul 23 09:57:14.214329 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:14.214183 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc5d5dc2-b24b-49c4-8a72-335242f92896-tls-certs\") pod \"auth-test-model-efdc11-kserve-65c9899bcf-n4k7p\" (UID: \"fc5d5dc2-b24b-49c4-8a72-335242f92896\") " pod="ai-tenant-e2e-shared-a-efdc11/auth-test-model-efdc11-kserve-65c9899bcf-n4k7p" Jul 23 09:57:14.214329 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:14.214225 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc5d5dc2-b24b-49c4-8a72-335242f92896-model-cache\") pod \"auth-test-model-efdc11-kserve-65c9899bcf-n4k7p\" (UID: \"fc5d5dc2-b24b-49c4-8a72-335242f92896\") " pod="ai-tenant-e2e-shared-a-efdc11/auth-test-model-efdc11-kserve-65c9899bcf-n4k7p" Jul 23 09:57:14.214329 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:14.214285 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc5d5dc2-b24b-49c4-8a72-335242f92896-tmp-dir\") pod \"auth-test-model-efdc11-kserve-65c9899bcf-n4k7p\" (UID: \"fc5d5dc2-b24b-49c4-8a72-335242f92896\") " pod="ai-tenant-e2e-shared-a-efdc11/auth-test-model-efdc11-kserve-65c9899bcf-n4k7p" Jul 23 09:57:14.214466 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:14.214333 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnrgj\" (UniqueName: \"kubernetes.io/projected/fc5d5dc2-b24b-49c4-8a72-335242f92896-kube-api-access-nnrgj\") pod \"auth-test-model-efdc11-kserve-65c9899bcf-n4k7p\" (UID: \"fc5d5dc2-b24b-49c4-8a72-335242f92896\") " pod="ai-tenant-e2e-shared-a-efdc11/auth-test-model-efdc11-kserve-65c9899bcf-n4k7p" Jul 23 09:57:14.214466 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:14.214361 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc5d5dc2-b24b-49c4-8a72-335242f92896-dshm\") pod \"auth-test-model-efdc11-kserve-65c9899bcf-n4k7p\" (UID: \"fc5d5dc2-b24b-49c4-8a72-335242f92896\") " pod="ai-tenant-e2e-shared-a-efdc11/auth-test-model-efdc11-kserve-65c9899bcf-n4k7p" Jul 23 09:57:14.314832 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:14.314798 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc5d5dc2-b24b-49c4-8a72-335242f92896-home\") pod \"auth-test-model-efdc11-kserve-65c9899bcf-n4k7p\" (UID: \"fc5d5dc2-b24b-49c4-8a72-335242f92896\") " pod="ai-tenant-e2e-shared-a-efdc11/auth-test-model-efdc11-kserve-65c9899bcf-n4k7p" Jul 23 09:57:14.314832 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:14.314833 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc5d5dc2-b24b-49c4-8a72-335242f92896-tls-certs\") pod \"auth-test-model-efdc11-kserve-65c9899bcf-n4k7p\" (UID: \"fc5d5dc2-b24b-49c4-8a72-335242f92896\") " pod="ai-tenant-e2e-shared-a-efdc11/auth-test-model-efdc11-kserve-65c9899bcf-n4k7p" Jul 23 09:57:14.315072 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:14.314883 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc5d5dc2-b24b-49c4-8a72-335242f92896-model-cache\") pod \"auth-test-model-efdc11-kserve-65c9899bcf-n4k7p\" (UID: \"fc5d5dc2-b24b-49c4-8a72-335242f92896\") " pod="ai-tenant-e2e-shared-a-efdc11/auth-test-model-efdc11-kserve-65c9899bcf-n4k7p" Jul 23 09:57:14.315072 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:14.314913 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc5d5dc2-b24b-49c4-8a72-335242f92896-tmp-dir\") pod \"auth-test-model-efdc11-kserve-65c9899bcf-n4k7p\" (UID: \"fc5d5dc2-b24b-49c4-8a72-335242f92896\") " pod="ai-tenant-e2e-shared-a-efdc11/auth-test-model-efdc11-kserve-65c9899bcf-n4k7p" Jul 23 09:57:14.315072 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:14.314944 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnrgj\" (UniqueName: \"kubernetes.io/projected/fc5d5dc2-b24b-49c4-8a72-335242f92896-kube-api-access-nnrgj\") pod \"auth-test-model-efdc11-kserve-65c9899bcf-n4k7p\" (UID: \"fc5d5dc2-b24b-49c4-8a72-335242f92896\") " pod="ai-tenant-e2e-shared-a-efdc11/auth-test-model-efdc11-kserve-65c9899bcf-n4k7p" Jul 23 09:57:14.315072 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:14.314966 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc5d5dc2-b24b-49c4-8a72-335242f92896-dshm\") pod \"auth-test-model-efdc11-kserve-65c9899bcf-n4k7p\" (UID: \"fc5d5dc2-b24b-49c4-8a72-335242f92896\") " pod="ai-tenant-e2e-shared-a-efdc11/auth-test-model-efdc11-kserve-65c9899bcf-n4k7p" Jul 23 09:57:14.315294 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:14.315208 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc5d5dc2-b24b-49c4-8a72-335242f92896-home\") pod \"auth-test-model-efdc11-kserve-65c9899bcf-n4k7p\" (UID: \"fc5d5dc2-b24b-49c4-8a72-335242f92896\") " pod="ai-tenant-e2e-shared-a-efdc11/auth-test-model-efdc11-kserve-65c9899bcf-n4k7p" Jul 23 09:57:14.315346 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:14.315302 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc5d5dc2-b24b-49c4-8a72-335242f92896-model-cache\") pod \"auth-test-model-efdc11-kserve-65c9899bcf-n4k7p\" (UID: \"fc5d5dc2-b24b-49c4-8a72-335242f92896\") " pod="ai-tenant-e2e-shared-a-efdc11/auth-test-model-efdc11-kserve-65c9899bcf-n4k7p" Jul 23 09:57:14.315346 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:14.315334 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc5d5dc2-b24b-49c4-8a72-335242f92896-tmp-dir\") pod \"auth-test-model-efdc11-kserve-65c9899bcf-n4k7p\" (UID: \"fc5d5dc2-b24b-49c4-8a72-335242f92896\") " pod="ai-tenant-e2e-shared-a-efdc11/auth-test-model-efdc11-kserve-65c9899bcf-n4k7p" Jul 23 09:57:14.317332 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:14.317311 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc5d5dc2-b24b-49c4-8a72-335242f92896-dshm\") pod \"auth-test-model-efdc11-kserve-65c9899bcf-n4k7p\" (UID: \"fc5d5dc2-b24b-49c4-8a72-335242f92896\") " pod="ai-tenant-e2e-shared-a-efdc11/auth-test-model-efdc11-kserve-65c9899bcf-n4k7p" Jul 23 09:57:14.317583 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:14.317565 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc5d5dc2-b24b-49c4-8a72-335242f92896-tls-certs\") pod \"auth-test-model-efdc11-kserve-65c9899bcf-n4k7p\" (UID: \"fc5d5dc2-b24b-49c4-8a72-335242f92896\") " pod="ai-tenant-e2e-shared-a-efdc11/auth-test-model-efdc11-kserve-65c9899bcf-n4k7p" Jul 23 09:57:14.330408 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:14.330386 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnrgj\" (UniqueName: \"kubernetes.io/projected/fc5d5dc2-b24b-49c4-8a72-335242f92896-kube-api-access-nnrgj\") pod \"auth-test-model-efdc11-kserve-65c9899bcf-n4k7p\" (UID: \"fc5d5dc2-b24b-49c4-8a72-335242f92896\") " pod="ai-tenant-e2e-shared-a-efdc11/auth-test-model-efdc11-kserve-65c9899bcf-n4k7p" Jul 23 09:57:14.366676 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:14.366642 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:57:14.480011 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:14.479917 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-efdc11/auth-test-model-efdc11-kserve-65c9899bcf-n4k7p" Jul 23 09:57:14.480967 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:14.480939 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:57:14.619052 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:14.619024 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-efdc11/auth-test-model-efdc11-kserve-65c9899bcf-n4k7p"] Jul 23 09:57:14.621033 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:57:14.621004 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfc5d5dc2_b24b_49c4_8a72_335242f92896.slice/crio-af8098a78ae5e472b3f2b2eba9852eb7fc38bd0ee93f6f03bbf8a32c280a6faf WatchSource:0}: Error finding container af8098a78ae5e472b3f2b2eba9852eb7fc38bd0ee93f6f03bbf8a32c280a6faf: Status 404 returned error can't find the container with id af8098a78ae5e472b3f2b2eba9852eb7fc38bd0ee93f6f03bbf8a32c280a6faf Jul 23 09:57:14.701776 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:14.701734 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-efdc11/auth-test-model-efdc11-kserve-65c9899bcf-n4k7p" event={"ID":"fc5d5dc2-b24b-49c4-8a72-335242f92896","Type":"ContainerStarted","Data":"f57202fb73309afbe5665dfad03d45b2077276c74f2a877a289090b22ff5d7bf"} Jul 23 09:57:14.701965 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:14.701784 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-efdc11/auth-test-model-efdc11-kserve-65c9899bcf-n4k7p" event={"ID":"fc5d5dc2-b24b-49c4-8a72-335242f92896","Type":"ContainerStarted","Data":"af8098a78ae5e472b3f2b2eba9852eb7fc38bd0ee93f6f03bbf8a32c280a6faf"} Jul 23 09:57:14.719412 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:14.719254 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-efdc11/auth-test-model-efdc11-kserve-65c9899bcf-n4k7p" podStartSLOduration=0.719220553 podStartE2EDuration="719.220553ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:57:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:57:14.716352296 +0000 UTC m=+2423.255630872" watchObservedRunningTime="2026-07-23 09:57:14.719220553 +0000 UTC m=+2423.258499129" Jul 23 09:57:17.391796 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:17.391757 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:57:24.480720 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:24.480676 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-efdc11/auth-test-model-efdc11-kserve-65c9899bcf-n4k7p" Jul 23 09:57:24.480720 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:24.480727 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-efdc11/auth-test-model-efdc11-kserve-65c9899bcf-n4k7p" Jul 23 09:57:24.493471 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:24.493437 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-efdc11/auth-test-model-efdc11-kserve-65c9899bcf-n4k7p" Jul 23 09:57:24.773060 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:24.772971 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-efdc11/auth-test-model-efdc11-kserve-65c9899bcf-n4k7p" Jul 23 09:57:26.691765 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:26.691716 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:57:29.503997 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:29.503959 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-0dc985/auth-test-model-0dc985-kserve-7c56c45d5f-fw74q"] Jul 23 09:57:29.507378 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:29.507360 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0dc985/auth-test-model-0dc985-kserve-7c56c45d5f-fw74q" Jul 23 09:57:29.510487 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:29.510459 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-0dc985\"/\"auth-test-model-0dc985-kserve-self-signed-certs\"" Jul 23 09:57:29.510487 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:29.510475 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-0dc985\"/\"openshift-service-ca.crt\"" Jul 23 09:57:29.510673 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:29.510479 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-0dc985\"/\"kube-root-ca.crt\"" Jul 23 09:57:29.510673 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:29.510483 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-0dc985\"/\"default-dockercfg-47rx4\"" Jul 23 09:57:29.518315 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:29.518289 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-0dc985/auth-test-model-0dc985-kserve-7c56c45d5f-fw74q"] Jul 23 09:57:29.531749 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:29.531718 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:57:29.653590 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:29.653548 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7-tls-certs\") pod \"auth-test-model-0dc985-kserve-7c56c45d5f-fw74q\" (UID: \"b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7\") " pod="ai-tenant-e2e-shared-b-0dc985/auth-test-model-0dc985-kserve-7c56c45d5f-fw74q" Jul 23 09:57:29.653787 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:29.653656 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7-model-cache\") pod \"auth-test-model-0dc985-kserve-7c56c45d5f-fw74q\" (UID: \"b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7\") " pod="ai-tenant-e2e-shared-b-0dc985/auth-test-model-0dc985-kserve-7c56c45d5f-fw74q" Jul 23 09:57:29.653787 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:29.653677 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7-dshm\") pod \"auth-test-model-0dc985-kserve-7c56c45d5f-fw74q\" (UID: \"b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7\") " pod="ai-tenant-e2e-shared-b-0dc985/auth-test-model-0dc985-kserve-7c56c45d5f-fw74q" Jul 23 09:57:29.653787 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:29.653697 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n7rr\" (UniqueName: \"kubernetes.io/projected/b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7-kube-api-access-4n7rr\") pod \"auth-test-model-0dc985-kserve-7c56c45d5f-fw74q\" (UID: \"b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7\") " pod="ai-tenant-e2e-shared-b-0dc985/auth-test-model-0dc985-kserve-7c56c45d5f-fw74q" Jul 23 09:57:29.653787 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:29.653737 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7-home\") pod \"auth-test-model-0dc985-kserve-7c56c45d5f-fw74q\" (UID: \"b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7\") " pod="ai-tenant-e2e-shared-b-0dc985/auth-test-model-0dc985-kserve-7c56c45d5f-fw74q" Jul 23 09:57:29.653932 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:29.653798 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7-tmp-dir\") pod \"auth-test-model-0dc985-kserve-7c56c45d5f-fw74q\" (UID: \"b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7\") " pod="ai-tenant-e2e-shared-b-0dc985/auth-test-model-0dc985-kserve-7c56c45d5f-fw74q" Jul 23 09:57:29.755193 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:29.755089 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7-tmp-dir\") pod \"auth-test-model-0dc985-kserve-7c56c45d5f-fw74q\" (UID: \"b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7\") " pod="ai-tenant-e2e-shared-b-0dc985/auth-test-model-0dc985-kserve-7c56c45d5f-fw74q" Jul 23 09:57:29.755193 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:29.755137 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7-tls-certs\") pod \"auth-test-model-0dc985-kserve-7c56c45d5f-fw74q\" (UID: \"b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7\") " pod="ai-tenant-e2e-shared-b-0dc985/auth-test-model-0dc985-kserve-7c56c45d5f-fw74q" Jul 23 09:57:29.755438 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:29.755225 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7-model-cache\") pod \"auth-test-model-0dc985-kserve-7c56c45d5f-fw74q\" (UID: \"b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7\") " pod="ai-tenant-e2e-shared-b-0dc985/auth-test-model-0dc985-kserve-7c56c45d5f-fw74q" Jul 23 09:57:29.755438 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:29.755283 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7-dshm\") pod \"auth-test-model-0dc985-kserve-7c56c45d5f-fw74q\" (UID: \"b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7\") " pod="ai-tenant-e2e-shared-b-0dc985/auth-test-model-0dc985-kserve-7c56c45d5f-fw74q" Jul 23 09:57:29.755438 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:29.755307 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n7rr\" (UniqueName: \"kubernetes.io/projected/b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7-kube-api-access-4n7rr\") pod \"auth-test-model-0dc985-kserve-7c56c45d5f-fw74q\" (UID: \"b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7\") " pod="ai-tenant-e2e-shared-b-0dc985/auth-test-model-0dc985-kserve-7c56c45d5f-fw74q" Jul 23 09:57:29.755438 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:29.755355 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7-home\") pod \"auth-test-model-0dc985-kserve-7c56c45d5f-fw74q\" (UID: \"b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7\") " pod="ai-tenant-e2e-shared-b-0dc985/auth-test-model-0dc985-kserve-7c56c45d5f-fw74q" Jul 23 09:57:29.755654 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:29.755628 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7-tmp-dir\") pod \"auth-test-model-0dc985-kserve-7c56c45d5f-fw74q\" (UID: \"b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7\") " pod="ai-tenant-e2e-shared-b-0dc985/auth-test-model-0dc985-kserve-7c56c45d5f-fw74q" Jul 23 09:57:29.755708 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:29.755668 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7-model-cache\") pod \"auth-test-model-0dc985-kserve-7c56c45d5f-fw74q\" (UID: \"b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7\") " pod="ai-tenant-e2e-shared-b-0dc985/auth-test-model-0dc985-kserve-7c56c45d5f-fw74q" Jul 23 09:57:29.755761 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:29.755708 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7-home\") pod \"auth-test-model-0dc985-kserve-7c56c45d5f-fw74q\" (UID: \"b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7\") " pod="ai-tenant-e2e-shared-b-0dc985/auth-test-model-0dc985-kserve-7c56c45d5f-fw74q" Jul 23 09:57:29.757708 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:29.757685 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7-dshm\") pod \"auth-test-model-0dc985-kserve-7c56c45d5f-fw74q\" (UID: \"b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7\") " pod="ai-tenant-e2e-shared-b-0dc985/auth-test-model-0dc985-kserve-7c56c45d5f-fw74q" Jul 23 09:57:29.757890 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:29.757873 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7-tls-certs\") pod \"auth-test-model-0dc985-kserve-7c56c45d5f-fw74q\" (UID: \"b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7\") " pod="ai-tenant-e2e-shared-b-0dc985/auth-test-model-0dc985-kserve-7c56c45d5f-fw74q" Jul 23 09:57:29.765004 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:29.764979 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n7rr\" (UniqueName: \"kubernetes.io/projected/b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7-kube-api-access-4n7rr\") pod \"auth-test-model-0dc985-kserve-7c56c45d5f-fw74q\" (UID: \"b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7\") " pod="ai-tenant-e2e-shared-b-0dc985/auth-test-model-0dc985-kserve-7c56c45d5f-fw74q" Jul 23 09:57:29.819096 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:29.819053 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0dc985/auth-test-model-0dc985-kserve-7c56c45d5f-fw74q" Jul 23 09:57:29.966320 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:29.966294 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-0dc985/auth-test-model-0dc985-kserve-7c56c45d5f-fw74q"] Jul 23 09:57:29.968504 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:57:29.968465 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb1d9c0dc_a7f8_4830_b071_9a59e8dbc0c7.slice/crio-5df26e6536c3b7fadec239569cc8a07a8d3af077725432a88cee592654b5041b WatchSource:0}: Error finding container 5df26e6536c3b7fadec239569cc8a07a8d3af077725432a88cee592654b5041b: Status 404 returned error can't find the container with id 5df26e6536c3b7fadec239569cc8a07a8d3af077725432a88cee592654b5041b Jul 23 09:57:30.791863 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:30.791821 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0dc985/auth-test-model-0dc985-kserve-7c56c45d5f-fw74q" event={"ID":"b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7","Type":"ContainerStarted","Data":"f314480e87469f9884a9fd6d7134e5fda1aad6e45b0c6a047f1228edf41862a5"} Jul 23 09:57:30.792279 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:30.791871 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0dc985/auth-test-model-0dc985-kserve-7c56c45d5f-fw74q" event={"ID":"b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7","Type":"ContainerStarted","Data":"5df26e6536c3b7fadec239569cc8a07a8d3af077725432a88cee592654b5041b"} Jul 23 09:57:30.809487 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:30.809432 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-0dc985/auth-test-model-0dc985-kserve-7c56c45d5f-fw74q" podStartSLOduration=1.809416084 podStartE2EDuration="1.809416084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:57:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:57:30.807811084 +0000 UTC m=+2439.347089677" watchObservedRunningTime="2026-07-23 09:57:30.809416084 +0000 UTC m=+2439.348694661" Jul 23 09:57:33.197499 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:33.197459 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:57:39.819416 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:39.819371 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-0dc985/auth-test-model-0dc985-kserve-7c56c45d5f-fw74q" Jul 23 09:57:39.819833 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:39.819431 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-0dc985/auth-test-model-0dc985-kserve-7c56c45d5f-fw74q" Jul 23 09:57:39.831992 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:39.831965 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-0dc985/auth-test-model-0dc985-kserve-7c56c45d5f-fw74q" Jul 23 09:57:39.850525 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:39.850497 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-0dc985/auth-test-model-0dc985-kserve-7c56c45d5f-fw74q" Jul 23 09:57:50.193608 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:57:50.193570 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:58:07.496564 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:58:07.496532 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:58:23.331659 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:58:23.331620 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:58:42.616758 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:58:42.616719 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:59:00.800186 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:59:00.800148 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:59:10.908082 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:59:10.908048 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:59:10.980000 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:59:10.979966 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:59:17.006590 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:59:17.006551 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:59:31.485225 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:59:31.485189 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:59:46.063392 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:59:46.063354 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-efdc11/auth-test-model-efdc11-kserve-65c9899bcf-n4k7p"] Jul 23 09:59:46.086109 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:59:46.086076 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:59:46.893689 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:59:46.893653 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-0dc985/auth-test-model-0dc985-kserve-7c56c45d5f-fw74q"] Jul 23 09:59:46.995876 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:59:46.995650 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:59:48.190985 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:59:48.190951 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 09:59:51.996616 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:59:51.996579 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-efdc11/sub-test-model-efdc11-kserve-69dfc4965f-fzz9q"] Jul 23 09:59:52.000356 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:59:52.000332 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-efdc11/sub-test-model-efdc11-kserve-69dfc4965f-fzz9q" Jul 23 09:59:52.002874 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:59:52.002844 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-efdc11\"/\"sub-test-model-efdc11-kserve-self-signed-certs\"" Jul 23 09:59:52.009577 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:59:52.009555 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-efdc11/sub-test-model-efdc11-kserve-69dfc4965f-fzz9q"] Jul 23 09:59:52.023477 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:59:52.023441 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e33e441d-9f34-45d5-aa67-01e269bf52d2-home\") pod \"sub-test-model-efdc11-kserve-69dfc4965f-fzz9q\" (UID: \"e33e441d-9f34-45d5-aa67-01e269bf52d2\") " pod="ai-tenant-e2e-shared-a-efdc11/sub-test-model-efdc11-kserve-69dfc4965f-fzz9q" Jul 23 09:59:52.023477 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:59:52.023470 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e33e441d-9f34-45d5-aa67-01e269bf52d2-dshm\") pod \"sub-test-model-efdc11-kserve-69dfc4965f-fzz9q\" (UID: \"e33e441d-9f34-45d5-aa67-01e269bf52d2\") " pod="ai-tenant-e2e-shared-a-efdc11/sub-test-model-efdc11-kserve-69dfc4965f-fzz9q" Jul 23 09:59:52.023695 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:59:52.023486 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e33e441d-9f34-45d5-aa67-01e269bf52d2-model-cache\") pod \"sub-test-model-efdc11-kserve-69dfc4965f-fzz9q\" (UID: \"e33e441d-9f34-45d5-aa67-01e269bf52d2\") " pod="ai-tenant-e2e-shared-a-efdc11/sub-test-model-efdc11-kserve-69dfc4965f-fzz9q" Jul 23 09:59:52.023695 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:59:52.023538 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e33e441d-9f34-45d5-aa67-01e269bf52d2-tmp-dir\") pod \"sub-test-model-efdc11-kserve-69dfc4965f-fzz9q\" (UID: \"e33e441d-9f34-45d5-aa67-01e269bf52d2\") " pod="ai-tenant-e2e-shared-a-efdc11/sub-test-model-efdc11-kserve-69dfc4965f-fzz9q" Jul 23 09:59:52.023695 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:59:52.023587 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnpn4\" (UniqueName: \"kubernetes.io/projected/e33e441d-9f34-45d5-aa67-01e269bf52d2-kube-api-access-cnpn4\") pod \"sub-test-model-efdc11-kserve-69dfc4965f-fzz9q\" (UID: \"e33e441d-9f34-45d5-aa67-01e269bf52d2\") " pod="ai-tenant-e2e-shared-a-efdc11/sub-test-model-efdc11-kserve-69dfc4965f-fzz9q" Jul 23 09:59:52.023695 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:59:52.023663 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e33e441d-9f34-45d5-aa67-01e269bf52d2-tls-certs\") pod \"sub-test-model-efdc11-kserve-69dfc4965f-fzz9q\" (UID: \"e33e441d-9f34-45d5-aa67-01e269bf52d2\") " pod="ai-tenant-e2e-shared-a-efdc11/sub-test-model-efdc11-kserve-69dfc4965f-fzz9q" Jul 23 09:59:52.079930 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:59:52.079886 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 09:59:52.124848 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:59:52.124816 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnpn4\" (UniqueName: \"kubernetes.io/projected/e33e441d-9f34-45d5-aa67-01e269bf52d2-kube-api-access-cnpn4\") pod \"sub-test-model-efdc11-kserve-69dfc4965f-fzz9q\" (UID: \"e33e441d-9f34-45d5-aa67-01e269bf52d2\") " pod="ai-tenant-e2e-shared-a-efdc11/sub-test-model-efdc11-kserve-69dfc4965f-fzz9q" Jul 23 09:59:52.125001 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:59:52.124861 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e33e441d-9f34-45d5-aa67-01e269bf52d2-tls-certs\") pod \"sub-test-model-efdc11-kserve-69dfc4965f-fzz9q\" (UID: \"e33e441d-9f34-45d5-aa67-01e269bf52d2\") " pod="ai-tenant-e2e-shared-a-efdc11/sub-test-model-efdc11-kserve-69dfc4965f-fzz9q" Jul 23 09:59:52.125001 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:59:52.124894 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e33e441d-9f34-45d5-aa67-01e269bf52d2-home\") pod \"sub-test-model-efdc11-kserve-69dfc4965f-fzz9q\" (UID: \"e33e441d-9f34-45d5-aa67-01e269bf52d2\") " pod="ai-tenant-e2e-shared-a-efdc11/sub-test-model-efdc11-kserve-69dfc4965f-fzz9q" Jul 23 09:59:52.125001 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:59:52.124909 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e33e441d-9f34-45d5-aa67-01e269bf52d2-dshm\") pod \"sub-test-model-efdc11-kserve-69dfc4965f-fzz9q\" (UID: \"e33e441d-9f34-45d5-aa67-01e269bf52d2\") " pod="ai-tenant-e2e-shared-a-efdc11/sub-test-model-efdc11-kserve-69dfc4965f-fzz9q" Jul 23 09:59:52.125001 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:59:52.124924 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e33e441d-9f34-45d5-aa67-01e269bf52d2-model-cache\") pod \"sub-test-model-efdc11-kserve-69dfc4965f-fzz9q\" (UID: \"e33e441d-9f34-45d5-aa67-01e269bf52d2\") " pod="ai-tenant-e2e-shared-a-efdc11/sub-test-model-efdc11-kserve-69dfc4965f-fzz9q" Jul 23 09:59:52.125001 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:59:52.124947 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e33e441d-9f34-45d5-aa67-01e269bf52d2-tmp-dir\") pod \"sub-test-model-efdc11-kserve-69dfc4965f-fzz9q\" (UID: \"e33e441d-9f34-45d5-aa67-01e269bf52d2\") " pod="ai-tenant-e2e-shared-a-efdc11/sub-test-model-efdc11-kserve-69dfc4965f-fzz9q" Jul 23 09:59:52.125314 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:59:52.125288 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e33e441d-9f34-45d5-aa67-01e269bf52d2-home\") pod \"sub-test-model-efdc11-kserve-69dfc4965f-fzz9q\" (UID: \"e33e441d-9f34-45d5-aa67-01e269bf52d2\") " pod="ai-tenant-e2e-shared-a-efdc11/sub-test-model-efdc11-kserve-69dfc4965f-fzz9q" Jul 23 09:59:52.125398 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:59:52.125374 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e33e441d-9f34-45d5-aa67-01e269bf52d2-tmp-dir\") pod \"sub-test-model-efdc11-kserve-69dfc4965f-fzz9q\" (UID: \"e33e441d-9f34-45d5-aa67-01e269bf52d2\") " pod="ai-tenant-e2e-shared-a-efdc11/sub-test-model-efdc11-kserve-69dfc4965f-fzz9q" Jul 23 09:59:52.125450 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:59:52.125405 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e33e441d-9f34-45d5-aa67-01e269bf52d2-model-cache\") pod \"sub-test-model-efdc11-kserve-69dfc4965f-fzz9q\" (UID: \"e33e441d-9f34-45d5-aa67-01e269bf52d2\") " pod="ai-tenant-e2e-shared-a-efdc11/sub-test-model-efdc11-kserve-69dfc4965f-fzz9q" Jul 23 09:59:52.127307 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:59:52.127284 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e33e441d-9f34-45d5-aa67-01e269bf52d2-dshm\") pod \"sub-test-model-efdc11-kserve-69dfc4965f-fzz9q\" (UID: \"e33e441d-9f34-45d5-aa67-01e269bf52d2\") " pod="ai-tenant-e2e-shared-a-efdc11/sub-test-model-efdc11-kserve-69dfc4965f-fzz9q" Jul 23 09:59:52.127520 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:59:52.127501 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e33e441d-9f34-45d5-aa67-01e269bf52d2-tls-certs\") pod \"sub-test-model-efdc11-kserve-69dfc4965f-fzz9q\" (UID: \"e33e441d-9f34-45d5-aa67-01e269bf52d2\") " pod="ai-tenant-e2e-shared-a-efdc11/sub-test-model-efdc11-kserve-69dfc4965f-fzz9q" Jul 23 09:59:52.137227 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:59:52.137207 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnpn4\" (UniqueName: \"kubernetes.io/projected/e33e441d-9f34-45d5-aa67-01e269bf52d2-kube-api-access-cnpn4\") pod \"sub-test-model-efdc11-kserve-69dfc4965f-fzz9q\" (UID: \"e33e441d-9f34-45d5-aa67-01e269bf52d2\") " pod="ai-tenant-e2e-shared-a-efdc11/sub-test-model-efdc11-kserve-69dfc4965f-fzz9q" Jul 23 09:59:52.310581 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:59:52.310494 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-efdc11/sub-test-model-efdc11-kserve-69dfc4965f-fzz9q" Jul 23 09:59:52.467728 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:59:52.467696 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-efdc11/sub-test-model-efdc11-kserve-69dfc4965f-fzz9q"] Jul 23 09:59:52.469716 ip-10-0-135-145 kubenswrapper[2576]: W0723 09:59:52.469683 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode33e441d_9f34_45d5_aa67_01e269bf52d2.slice/crio-4824541c54e0be64485e576a80a2cfc81b4befd280a9236cd133e78509a543cf WatchSource:0}: Error finding container 4824541c54e0be64485e576a80a2cfc81b4befd280a9236cd133e78509a543cf: Status 404 returned error can't find the container with id 4824541c54e0be64485e576a80a2cfc81b4befd280a9236cd133e78509a543cf Jul 23 09:59:52.541121 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:59:52.541094 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-efdc11/sub-test-model-efdc11-kserve-69dfc4965f-fzz9q" event={"ID":"e33e441d-9f34-45d5-aa67-01e269bf52d2","Type":"ContainerStarted","Data":"8f6acbefa45b24a3d7ae04f87090a92d1bc49282b4354fc6023aeeedee9e8e2b"} Jul 23 09:59:52.541227 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:59:52.541133 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-efdc11/sub-test-model-efdc11-kserve-69dfc4965f-fzz9q" event={"ID":"e33e441d-9f34-45d5-aa67-01e269bf52d2","Type":"ContainerStarted","Data":"4824541c54e0be64485e576a80a2cfc81b4befd280a9236cd133e78509a543cf"} Jul 23 09:59:52.567406 ip-10-0-135-145 kubenswrapper[2576]: I0723 09:59:52.567321 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-efdc11/sub-test-model-efdc11-kserve-69dfc4965f-fzz9q" podStartSLOduration=1.567306117 podStartE2EDuration="1.567306117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:59:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:59:52.566011963 +0000 UTC m=+2581.105290541" watchObservedRunningTime="2026-07-23 09:59:52.567306117 +0000 UTC m=+2581.106584695" Jul 23 10:00:01.001963 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:01.001923 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 10:00:01.083815 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:01.083775 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-efdc11/auth-test-model-efdc11-kserve-65c9899bcf-n4k7p" podUID="fc5d5dc2-b24b-49c4-8a72-335242f92896" containerName="main" containerID="cri-o://f57202fb73309afbe5665dfad03d45b2077276c74f2a877a289090b22ff5d7bf" gracePeriod=45 Jul 23 10:00:01.340217 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:01.340193 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-efdc11/auth-test-model-efdc11-kserve-65c9899bcf-n4k7p" Jul 23 10:00:01.410118 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:01.410085 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc5d5dc2-b24b-49c4-8a72-335242f92896-tls-certs\") pod \"fc5d5dc2-b24b-49c4-8a72-335242f92896\" (UID: \"fc5d5dc2-b24b-49c4-8a72-335242f92896\") " Jul 23 10:00:01.410320 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:01.410130 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc5d5dc2-b24b-49c4-8a72-335242f92896-tmp-dir\") pod \"fc5d5dc2-b24b-49c4-8a72-335242f92896\" (UID: \"fc5d5dc2-b24b-49c4-8a72-335242f92896\") " Jul 23 10:00:01.410320 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:01.410312 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc5d5dc2-b24b-49c4-8a72-335242f92896-home\") pod \"fc5d5dc2-b24b-49c4-8a72-335242f92896\" (UID: \"fc5d5dc2-b24b-49c4-8a72-335242f92896\") " Jul 23 10:00:01.410500 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:01.410348 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc5d5dc2-b24b-49c4-8a72-335242f92896-dshm\") pod \"fc5d5dc2-b24b-49c4-8a72-335242f92896\" (UID: \"fc5d5dc2-b24b-49c4-8a72-335242f92896\") " Jul 23 10:00:01.410500 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:01.410392 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc5d5dc2-b24b-49c4-8a72-335242f92896-model-cache\") pod \"fc5d5dc2-b24b-49c4-8a72-335242f92896\" (UID: \"fc5d5dc2-b24b-49c4-8a72-335242f92896\") " Jul 23 10:00:01.410500 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:01.410416 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nnrgj\" (UniqueName: \"kubernetes.io/projected/fc5d5dc2-b24b-49c4-8a72-335242f92896-kube-api-access-nnrgj\") pod \"fc5d5dc2-b24b-49c4-8a72-335242f92896\" (UID: \"fc5d5dc2-b24b-49c4-8a72-335242f92896\") " Jul 23 10:00:01.410500 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:01.410422 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc5d5dc2-b24b-49c4-8a72-335242f92896-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fc5d5dc2-b24b-49c4-8a72-335242f92896" (UID: "fc5d5dc2-b24b-49c4-8a72-335242f92896"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 10:00:01.410649 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:01.410550 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc5d5dc2-b24b-49c4-8a72-335242f92896-home" (OuterVolumeSpecName: "home") pod "fc5d5dc2-b24b-49c4-8a72-335242f92896" (UID: "fc5d5dc2-b24b-49c4-8a72-335242f92896"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 10:00:01.410694 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:01.410651 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc5d5dc2-b24b-49c4-8a72-335242f92896-model-cache" (OuterVolumeSpecName: "model-cache") pod "fc5d5dc2-b24b-49c4-8a72-335242f92896" (UID: "fc5d5dc2-b24b-49c4-8a72-335242f92896"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 10:00:01.410747 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:01.410733 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc5d5dc2-b24b-49c4-8a72-335242f92896-home\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 10:00:01.410791 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:01.410753 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc5d5dc2-b24b-49c4-8a72-335242f92896-tmp-dir\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 10:00:01.412541 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:01.412516 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc5d5dc2-b24b-49c4-8a72-335242f92896-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fc5d5dc2-b24b-49c4-8a72-335242f92896" (UID: "fc5d5dc2-b24b-49c4-8a72-335242f92896"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 10:00:01.412541 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:01.412528 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc5d5dc2-b24b-49c4-8a72-335242f92896-dshm" (OuterVolumeSpecName: "dshm") pod "fc5d5dc2-b24b-49c4-8a72-335242f92896" (UID: "fc5d5dc2-b24b-49c4-8a72-335242f92896"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 10:00:01.412686 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:01.412549 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc5d5dc2-b24b-49c4-8a72-335242f92896-kube-api-access-nnrgj" (OuterVolumeSpecName: "kube-api-access-nnrgj") pod "fc5d5dc2-b24b-49c4-8a72-335242f92896" (UID: "fc5d5dc2-b24b-49c4-8a72-335242f92896"). InnerVolumeSpecName "kube-api-access-nnrgj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 10:00:01.512130 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:01.512078 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc5d5dc2-b24b-49c4-8a72-335242f92896-dshm\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 10:00:01.512130 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:01.512121 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc5d5dc2-b24b-49c4-8a72-335242f92896-model-cache\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 10:00:01.512130 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:01.512136 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnrgj\" (UniqueName: \"kubernetes.io/projected/fc5d5dc2-b24b-49c4-8a72-335242f92896-kube-api-access-nnrgj\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 10:00:01.512399 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:01.512148 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc5d5dc2-b24b-49c4-8a72-335242f92896-tls-certs\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 10:00:01.589749 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:01.589663 2576 generic.go:356] "Generic (PLEG): container finished" podID="fc5d5dc2-b24b-49c4-8a72-335242f92896" containerID="f57202fb73309afbe5665dfad03d45b2077276c74f2a877a289090b22ff5d7bf" exitCode=0 Jul 23 10:00:01.589749 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:01.589706 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-efdc11/auth-test-model-efdc11-kserve-65c9899bcf-n4k7p" Jul 23 10:00:01.589950 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:01.589752 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-efdc11/auth-test-model-efdc11-kserve-65c9899bcf-n4k7p" event={"ID":"fc5d5dc2-b24b-49c4-8a72-335242f92896","Type":"ContainerDied","Data":"f57202fb73309afbe5665dfad03d45b2077276c74f2a877a289090b22ff5d7bf"} Jul 23 10:00:01.589950 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:01.589795 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-efdc11/auth-test-model-efdc11-kserve-65c9899bcf-n4k7p" event={"ID":"fc5d5dc2-b24b-49c4-8a72-335242f92896","Type":"ContainerDied","Data":"af8098a78ae5e472b3f2b2eba9852eb7fc38bd0ee93f6f03bbf8a32c280a6faf"} Jul 23 10:00:01.589950 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:01.589813 2576 scope.go:117] "RemoveContainer" containerID="f57202fb73309afbe5665dfad03d45b2077276c74f2a877a289090b22ff5d7bf" Jul 23 10:00:01.599609 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:01.599590 2576 scope.go:117] "RemoveContainer" containerID="f57202fb73309afbe5665dfad03d45b2077276c74f2a877a289090b22ff5d7bf" Jul 23 10:00:01.599857 ip-10-0-135-145 kubenswrapper[2576]: E0723 10:00:01.599836 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f57202fb73309afbe5665dfad03d45b2077276c74f2a877a289090b22ff5d7bf\": container with ID starting with f57202fb73309afbe5665dfad03d45b2077276c74f2a877a289090b22ff5d7bf not found: ID does not exist" containerID="f57202fb73309afbe5665dfad03d45b2077276c74f2a877a289090b22ff5d7bf" Jul 23 10:00:01.599900 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:01.599866 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f57202fb73309afbe5665dfad03d45b2077276c74f2a877a289090b22ff5d7bf"} err="failed to get container status \"f57202fb73309afbe5665dfad03d45b2077276c74f2a877a289090b22ff5d7bf\": rpc error: code = NotFound desc = could not find container \"f57202fb73309afbe5665dfad03d45b2077276c74f2a877a289090b22ff5d7bf\": container with ID starting with f57202fb73309afbe5665dfad03d45b2077276c74f2a877a289090b22ff5d7bf not found: ID does not exist" Jul 23 10:00:01.612116 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:01.612094 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-efdc11/auth-test-model-efdc11-kserve-65c9899bcf-n4k7p"] Jul 23 10:00:01.616288 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:01.616267 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-efdc11/auth-test-model-efdc11-kserve-65c9899bcf-n4k7p"] Jul 23 10:00:01.913275 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:01.913217 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-0dc985/auth-test-model-0dc985-kserve-7c56c45d5f-fw74q" podUID="b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7" containerName="main" containerID="cri-o://f314480e87469f9884a9fd6d7134e5fda1aad6e45b0c6a047f1228edf41862a5" gracePeriod=45 Jul 23 10:00:02.171852 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:02.171790 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0dc985/auth-test-model-0dc985-kserve-7c56c45d5f-fw74q" Jul 23 10:00:02.217803 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:02.217771 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7-tls-certs\") pod \"b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7\" (UID: \"b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7\") " Jul 23 10:00:02.217803 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:02.217805 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7-dshm\") pod \"b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7\" (UID: \"b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7\") " Jul 23 10:00:02.218375 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:02.217825 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7-model-cache\") pod \"b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7\" (UID: \"b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7\") " Jul 23 10:00:02.218375 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:02.217875 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4n7rr\" (UniqueName: \"kubernetes.io/projected/b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7-kube-api-access-4n7rr\") pod \"b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7\" (UID: \"b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7\") " Jul 23 10:00:02.218375 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:02.217917 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7-home\") pod \"b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7\" (UID: \"b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7\") " Jul 23 10:00:02.218375 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:02.217944 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7-tmp-dir\") pod \"b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7\" (UID: \"b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7\") " Jul 23 10:00:02.218375 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:02.218271 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7-model-cache" (OuterVolumeSpecName: "model-cache") pod "b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7" (UID: "b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 10:00:02.218375 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:02.218291 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7-home" (OuterVolumeSpecName: "home") pod "b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7" (UID: "b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 10:00:02.218375 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:02.218291 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7" (UID: "b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 10:00:02.220150 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:02.220124 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7-dshm" (OuterVolumeSpecName: "dshm") pod "b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7" (UID: "b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 10:00:02.220150 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:02.220140 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7" (UID: "b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 10:00:02.220525 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:02.220505 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7-kube-api-access-4n7rr" (OuterVolumeSpecName: "kube-api-access-4n7rr") pod "b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7" (UID: "b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7"). InnerVolumeSpecName "kube-api-access-4n7rr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 10:00:02.311514 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:02.311473 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-efdc11/sub-test-model-efdc11-kserve-69dfc4965f-fzz9q" Jul 23 10:00:02.311800 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:02.311772 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-efdc11/sub-test-model-efdc11-kserve-69dfc4965f-fzz9q" Jul 23 10:00:02.318569 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:02.318547 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7-home\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 10:00:02.318569 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:02.318570 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7-tmp-dir\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 10:00:02.318736 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:02.318582 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7-tls-certs\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 10:00:02.318736 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:02.318595 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7-dshm\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 10:00:02.318736 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:02.318607 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7-model-cache\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 10:00:02.318736 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:02.318618 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4n7rr\" (UniqueName: \"kubernetes.io/projected/b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7-kube-api-access-4n7rr\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 10:00:02.324099 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:02.324078 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-efdc11/sub-test-model-efdc11-kserve-69dfc4965f-fzz9q" Jul 23 10:00:02.596835 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:02.596745 2576 generic.go:356] "Generic (PLEG): container finished" podID="b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7" containerID="f314480e87469f9884a9fd6d7134e5fda1aad6e45b0c6a047f1228edf41862a5" exitCode=0 Jul 23 10:00:02.596835 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:02.596791 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0dc985/auth-test-model-0dc985-kserve-7c56c45d5f-fw74q" Jul 23 10:00:02.597062 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:02.596830 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0dc985/auth-test-model-0dc985-kserve-7c56c45d5f-fw74q" event={"ID":"b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7","Type":"ContainerDied","Data":"f314480e87469f9884a9fd6d7134e5fda1aad6e45b0c6a047f1228edf41862a5"} Jul 23 10:00:02.597062 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:02.596872 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0dc985/auth-test-model-0dc985-kserve-7c56c45d5f-fw74q" event={"ID":"b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7","Type":"ContainerDied","Data":"5df26e6536c3b7fadec239569cc8a07a8d3af077725432a88cee592654b5041b"} Jul 23 10:00:02.597062 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:02.596889 2576 scope.go:117] "RemoveContainer" containerID="f314480e87469f9884a9fd6d7134e5fda1aad6e45b0c6a047f1228edf41862a5" Jul 23 10:00:02.612543 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:02.612519 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-efdc11/sub-test-model-efdc11-kserve-69dfc4965f-fzz9q" Jul 23 10:00:02.613965 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:02.613948 2576 scope.go:117] "RemoveContainer" containerID="f314480e87469f9884a9fd6d7134e5fda1aad6e45b0c6a047f1228edf41862a5" Jul 23 10:00:02.614273 ip-10-0-135-145 kubenswrapper[2576]: E0723 10:00:02.614230 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f314480e87469f9884a9fd6d7134e5fda1aad6e45b0c6a047f1228edf41862a5\": container with ID starting with f314480e87469f9884a9fd6d7134e5fda1aad6e45b0c6a047f1228edf41862a5 not found: ID does not exist" containerID="f314480e87469f9884a9fd6d7134e5fda1aad6e45b0c6a047f1228edf41862a5" Jul 23 10:00:02.614345 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:02.614286 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f314480e87469f9884a9fd6d7134e5fda1aad6e45b0c6a047f1228edf41862a5"} err="failed to get container status \"f314480e87469f9884a9fd6d7134e5fda1aad6e45b0c6a047f1228edf41862a5\": rpc error: code = NotFound desc = could not find container \"f314480e87469f9884a9fd6d7134e5fda1aad6e45b0c6a047f1228edf41862a5\": container with ID starting with f314480e87469f9884a9fd6d7134e5fda1aad6e45b0c6a047f1228edf41862a5 not found: ID does not exist" Jul 23 10:00:02.629452 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:02.629430 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-0dc985/auth-test-model-0dc985-kserve-7c56c45d5f-fw74q"] Jul 23 10:00:02.637095 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:02.637074 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-0dc985/auth-test-model-0dc985-kserve-7c56c45d5f-fw74q"] Jul 23 10:00:02.987197 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:02.987164 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7" path="/var/lib/kubelet/pods/b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7/volumes" Jul 23 10:00:02.987591 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:02.987576 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc5d5dc2-b24b-49c4-8a72-335242f92896" path="/var/lib/kubelet/pods/fc5d5dc2-b24b-49c4-8a72-335242f92896/volumes" Jul 23 10:00:26.284880 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:26.284797 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-0dc985/sub-test-model-0dc985-kserve-69f6b8d85-8gvsz"] Jul 23 10:00:26.285365 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:26.285344 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc5d5dc2-b24b-49c4-8a72-335242f92896" containerName="main" Jul 23 10:00:26.285365 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:26.285364 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc5d5dc2-b24b-49c4-8a72-335242f92896" containerName="main" Jul 23 10:00:26.285444 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:26.285401 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7" containerName="main" Jul 23 10:00:26.285444 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:26.285411 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7" containerName="main" Jul 23 10:00:26.285525 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:26.285512 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1d9c0dc-a7f8-4830-b071-9a59e8dbc0c7" containerName="main" Jul 23 10:00:26.285560 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:26.285530 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc5d5dc2-b24b-49c4-8a72-335242f92896" containerName="main" Jul 23 10:00:26.290435 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:26.290415 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0dc985/sub-test-model-0dc985-kserve-69f6b8d85-8gvsz" Jul 23 10:00:26.294711 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:26.294685 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-0dc985\"/\"sub-test-model-0dc985-kserve-self-signed-certs\"" Jul 23 10:00:26.301489 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:26.301457 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-0dc985/sub-test-model-0dc985-kserve-69f6b8d85-8gvsz"] Jul 23 10:00:26.305419 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:26.305398 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-0dc985\"/\"default-dockercfg-47rx4\"" Jul 23 10:00:26.305510 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:26.305463 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-0dc985\"/\"openshift-service-ca.crt\"" Jul 23 10:00:26.305510 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:26.305468 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-0dc985\"/\"kube-root-ca.crt\"" Jul 23 10:00:26.332756 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:26.332726 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 10:00:26.339803 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:26.339780 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa3bde2d-0fe6-4207-9697-c3d95a78fd59-home\") pod \"sub-test-model-0dc985-kserve-69f6b8d85-8gvsz\" (UID: \"fa3bde2d-0fe6-4207-9697-c3d95a78fd59\") " pod="ai-tenant-e2e-shared-b-0dc985/sub-test-model-0dc985-kserve-69f6b8d85-8gvsz" Jul 23 10:00:26.339885 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:26.339815 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gc2b\" (UniqueName: \"kubernetes.io/projected/fa3bde2d-0fe6-4207-9697-c3d95a78fd59-kube-api-access-2gc2b\") pod \"sub-test-model-0dc985-kserve-69f6b8d85-8gvsz\" (UID: \"fa3bde2d-0fe6-4207-9697-c3d95a78fd59\") " pod="ai-tenant-e2e-shared-b-0dc985/sub-test-model-0dc985-kserve-69f6b8d85-8gvsz" Jul 23 10:00:26.339885 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:26.339840 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa3bde2d-0fe6-4207-9697-c3d95a78fd59-tmp-dir\") pod \"sub-test-model-0dc985-kserve-69f6b8d85-8gvsz\" (UID: \"fa3bde2d-0fe6-4207-9697-c3d95a78fd59\") " pod="ai-tenant-e2e-shared-b-0dc985/sub-test-model-0dc985-kserve-69f6b8d85-8gvsz" Jul 23 10:00:26.339966 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:26.339907 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa3bde2d-0fe6-4207-9697-c3d95a78fd59-model-cache\") pod \"sub-test-model-0dc985-kserve-69f6b8d85-8gvsz\" (UID: \"fa3bde2d-0fe6-4207-9697-c3d95a78fd59\") " pod="ai-tenant-e2e-shared-b-0dc985/sub-test-model-0dc985-kserve-69f6b8d85-8gvsz" Jul 23 10:00:26.340002 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:26.339979 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa3bde2d-0fe6-4207-9697-c3d95a78fd59-tls-certs\") pod \"sub-test-model-0dc985-kserve-69f6b8d85-8gvsz\" (UID: \"fa3bde2d-0fe6-4207-9697-c3d95a78fd59\") " pod="ai-tenant-e2e-shared-b-0dc985/sub-test-model-0dc985-kserve-69f6b8d85-8gvsz" Jul 23 10:00:26.340036 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:26.340021 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa3bde2d-0fe6-4207-9697-c3d95a78fd59-dshm\") pod \"sub-test-model-0dc985-kserve-69f6b8d85-8gvsz\" (UID: \"fa3bde2d-0fe6-4207-9697-c3d95a78fd59\") " pod="ai-tenant-e2e-shared-b-0dc985/sub-test-model-0dc985-kserve-69f6b8d85-8gvsz" Jul 23 10:00:26.441308 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:26.441267 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa3bde2d-0fe6-4207-9697-c3d95a78fd59-tls-certs\") pod \"sub-test-model-0dc985-kserve-69f6b8d85-8gvsz\" (UID: \"fa3bde2d-0fe6-4207-9697-c3d95a78fd59\") " pod="ai-tenant-e2e-shared-b-0dc985/sub-test-model-0dc985-kserve-69f6b8d85-8gvsz" Jul 23 10:00:26.441513 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:26.441336 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa3bde2d-0fe6-4207-9697-c3d95a78fd59-dshm\") pod \"sub-test-model-0dc985-kserve-69f6b8d85-8gvsz\" (UID: \"fa3bde2d-0fe6-4207-9697-c3d95a78fd59\") " pod="ai-tenant-e2e-shared-b-0dc985/sub-test-model-0dc985-kserve-69f6b8d85-8gvsz" Jul 23 10:00:26.441513 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:26.441366 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa3bde2d-0fe6-4207-9697-c3d95a78fd59-home\") pod \"sub-test-model-0dc985-kserve-69f6b8d85-8gvsz\" (UID: \"fa3bde2d-0fe6-4207-9697-c3d95a78fd59\") " pod="ai-tenant-e2e-shared-b-0dc985/sub-test-model-0dc985-kserve-69f6b8d85-8gvsz" Jul 23 10:00:26.441513 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:26.441394 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gc2b\" (UniqueName: \"kubernetes.io/projected/fa3bde2d-0fe6-4207-9697-c3d95a78fd59-kube-api-access-2gc2b\") pod \"sub-test-model-0dc985-kserve-69f6b8d85-8gvsz\" (UID: \"fa3bde2d-0fe6-4207-9697-c3d95a78fd59\") " pod="ai-tenant-e2e-shared-b-0dc985/sub-test-model-0dc985-kserve-69f6b8d85-8gvsz" Jul 23 10:00:26.441712 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:26.441538 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa3bde2d-0fe6-4207-9697-c3d95a78fd59-tmp-dir\") pod \"sub-test-model-0dc985-kserve-69f6b8d85-8gvsz\" (UID: \"fa3bde2d-0fe6-4207-9697-c3d95a78fd59\") " pod="ai-tenant-e2e-shared-b-0dc985/sub-test-model-0dc985-kserve-69f6b8d85-8gvsz" Jul 23 10:00:26.441712 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:26.441584 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa3bde2d-0fe6-4207-9697-c3d95a78fd59-model-cache\") pod \"sub-test-model-0dc985-kserve-69f6b8d85-8gvsz\" (UID: \"fa3bde2d-0fe6-4207-9697-c3d95a78fd59\") " pod="ai-tenant-e2e-shared-b-0dc985/sub-test-model-0dc985-kserve-69f6b8d85-8gvsz" Jul 23 10:00:26.441859 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:26.441831 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa3bde2d-0fe6-4207-9697-c3d95a78fd59-home\") pod \"sub-test-model-0dc985-kserve-69f6b8d85-8gvsz\" (UID: \"fa3bde2d-0fe6-4207-9697-c3d95a78fd59\") " pod="ai-tenant-e2e-shared-b-0dc985/sub-test-model-0dc985-kserve-69f6b8d85-8gvsz" Jul 23 10:00:26.441977 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:26.441916 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa3bde2d-0fe6-4207-9697-c3d95a78fd59-tmp-dir\") pod \"sub-test-model-0dc985-kserve-69f6b8d85-8gvsz\" (UID: \"fa3bde2d-0fe6-4207-9697-c3d95a78fd59\") " pod="ai-tenant-e2e-shared-b-0dc985/sub-test-model-0dc985-kserve-69f6b8d85-8gvsz" Jul 23 10:00:26.441977 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:26.441918 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa3bde2d-0fe6-4207-9697-c3d95a78fd59-model-cache\") pod \"sub-test-model-0dc985-kserve-69f6b8d85-8gvsz\" (UID: \"fa3bde2d-0fe6-4207-9697-c3d95a78fd59\") " pod="ai-tenant-e2e-shared-b-0dc985/sub-test-model-0dc985-kserve-69f6b8d85-8gvsz" Jul 23 10:00:26.443666 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:26.443646 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa3bde2d-0fe6-4207-9697-c3d95a78fd59-dshm\") pod \"sub-test-model-0dc985-kserve-69f6b8d85-8gvsz\" (UID: \"fa3bde2d-0fe6-4207-9697-c3d95a78fd59\") " pod="ai-tenant-e2e-shared-b-0dc985/sub-test-model-0dc985-kserve-69f6b8d85-8gvsz" Jul 23 10:00:26.444005 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:26.443985 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa3bde2d-0fe6-4207-9697-c3d95a78fd59-tls-certs\") pod \"sub-test-model-0dc985-kserve-69f6b8d85-8gvsz\" (UID: \"fa3bde2d-0fe6-4207-9697-c3d95a78fd59\") " pod="ai-tenant-e2e-shared-b-0dc985/sub-test-model-0dc985-kserve-69f6b8d85-8gvsz" Jul 23 10:00:26.449586 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:26.449564 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gc2b\" (UniqueName: \"kubernetes.io/projected/fa3bde2d-0fe6-4207-9697-c3d95a78fd59-kube-api-access-2gc2b\") pod \"sub-test-model-0dc985-kserve-69f6b8d85-8gvsz\" (UID: \"fa3bde2d-0fe6-4207-9697-c3d95a78fd59\") " pod="ai-tenant-e2e-shared-b-0dc985/sub-test-model-0dc985-kserve-69f6b8d85-8gvsz" Jul 23 10:00:26.601543 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:26.601449 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0dc985/sub-test-model-0dc985-kserve-69f6b8d85-8gvsz" Jul 23 10:00:26.739363 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:26.739339 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-0dc985/sub-test-model-0dc985-kserve-69f6b8d85-8gvsz"] Jul 23 10:00:26.741337 ip-10-0-135-145 kubenswrapper[2576]: W0723 10:00:26.741306 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfa3bde2d_0fe6_4207_9697_c3d95a78fd59.slice/crio-56d2ca62b7c3035bacd70c695b3fc0d782064782e4ef576212661f5b6f2b1eb7 WatchSource:0}: Error finding container 56d2ca62b7c3035bacd70c695b3fc0d782064782e4ef576212661f5b6f2b1eb7: Status 404 returned error can't find the container with id 56d2ca62b7c3035bacd70c695b3fc0d782064782e4ef576212661f5b6f2b1eb7 Jul 23 10:00:27.731763 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:27.731726 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0dc985/sub-test-model-0dc985-kserve-69f6b8d85-8gvsz" event={"ID":"fa3bde2d-0fe6-4207-9697-c3d95a78fd59","Type":"ContainerStarted","Data":"67c55f2041735d893281ebd07ccfca5b8d9a671a0face87ce3ac7ed82e6665f8"} Jul 23 10:00:27.731763 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:27.731769 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0dc985/sub-test-model-0dc985-kserve-69f6b8d85-8gvsz" event={"ID":"fa3bde2d-0fe6-4207-9697-c3d95a78fd59","Type":"ContainerStarted","Data":"56d2ca62b7c3035bacd70c695b3fc0d782064782e4ef576212661f5b6f2b1eb7"} Jul 23 10:00:27.751432 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:27.751372 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-0dc985/sub-test-model-0dc985-kserve-69f6b8d85-8gvsz" podStartSLOduration=1.7513557 podStartE2EDuration="1.7513557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 10:00:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 10:00:27.749842853 +0000 UTC m=+2616.289121475" watchObservedRunningTime="2026-07-23 10:00:27.7513557 +0000 UTC m=+2616.290634255" Jul 23 10:00:29.800765 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:29.800721 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 10:00:36.601670 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:36.601634 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-0dc985/sub-test-model-0dc985-kserve-69f6b8d85-8gvsz" Jul 23 10:00:36.601670 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:36.601681 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-0dc985/sub-test-model-0dc985-kserve-69f6b8d85-8gvsz" Jul 23 10:00:36.614175 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:36.614146 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-0dc985/sub-test-model-0dc985-kserve-69f6b8d85-8gvsz" Jul 23 10:00:36.791782 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:36.791752 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-0dc985/sub-test-model-0dc985-kserve-69f6b8d85-8gvsz" Jul 23 10:00:57.500984 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:00:57.500952 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 10:01:10.249303 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:10.249269 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 10:01:10.800277 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:10.800226 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 10:01:11.190048 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:11.190012 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 10:01:17.517402 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:17.517358 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-efdc11/sub-test-model-efdc11-kserve-69dfc4965f-fzz9q"] Jul 23 10:01:17.538718 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:17.538680 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 10:01:18.386763 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:18.386729 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-0dc985/sub-test-model-0dc985-kserve-69f6b8d85-8gvsz"] Jul 23 10:01:18.443772 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:18.443740 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 10:01:28.501753 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:28.501719 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-efdc11/rate-test-model-efdc11-kserve-c7f66d5f-pg8jm"] Jul 23 10:01:28.505334 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:28.505314 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-efdc11/rate-test-model-efdc11-kserve-c7f66d5f-pg8jm" Jul 23 10:01:28.507988 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:28.507966 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-efdc11\"/\"rate-test-model-efdc11-kserve-self-signed-certs\"" Jul 23 10:01:28.509179 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:28.509157 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 10:01:28.520516 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:28.520483 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-efdc11/rate-test-model-efdc11-kserve-c7f66d5f-pg8jm"] Jul 23 10:01:28.550707 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:28.550674 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 10:01:28.588776 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:28.588748 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b33d295e-fd2b-404b-81b6-11d12c07e750-dshm\") pod \"rate-test-model-efdc11-kserve-c7f66d5f-pg8jm\" (UID: \"b33d295e-fd2b-404b-81b6-11d12c07e750\") " pod="ai-tenant-e2e-shared-a-efdc11/rate-test-model-efdc11-kserve-c7f66d5f-pg8jm" Jul 23 10:01:28.588885 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:28.588787 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b33d295e-fd2b-404b-81b6-11d12c07e750-model-cache\") pod \"rate-test-model-efdc11-kserve-c7f66d5f-pg8jm\" (UID: \"b33d295e-fd2b-404b-81b6-11d12c07e750\") " pod="ai-tenant-e2e-shared-a-efdc11/rate-test-model-efdc11-kserve-c7f66d5f-pg8jm" Jul 23 10:01:28.588885 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:28.588805 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b33d295e-fd2b-404b-81b6-11d12c07e750-home\") pod \"rate-test-model-efdc11-kserve-c7f66d5f-pg8jm\" (UID: \"b33d295e-fd2b-404b-81b6-11d12c07e750\") " pod="ai-tenant-e2e-shared-a-efdc11/rate-test-model-efdc11-kserve-c7f66d5f-pg8jm" Jul 23 10:01:28.588997 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:28.588911 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b33d295e-fd2b-404b-81b6-11d12c07e750-tmp-dir\") pod \"rate-test-model-efdc11-kserve-c7f66d5f-pg8jm\" (UID: \"b33d295e-fd2b-404b-81b6-11d12c07e750\") " pod="ai-tenant-e2e-shared-a-efdc11/rate-test-model-efdc11-kserve-c7f66d5f-pg8jm" Jul 23 10:01:28.588997 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:28.588945 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hh99\" (UniqueName: \"kubernetes.io/projected/b33d295e-fd2b-404b-81b6-11d12c07e750-kube-api-access-6hh99\") pod \"rate-test-model-efdc11-kserve-c7f66d5f-pg8jm\" (UID: \"b33d295e-fd2b-404b-81b6-11d12c07e750\") " pod="ai-tenant-e2e-shared-a-efdc11/rate-test-model-efdc11-kserve-c7f66d5f-pg8jm" Jul 23 10:01:28.589101 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:28.589042 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b33d295e-fd2b-404b-81b6-11d12c07e750-tls-certs\") pod \"rate-test-model-efdc11-kserve-c7f66d5f-pg8jm\" (UID: \"b33d295e-fd2b-404b-81b6-11d12c07e750\") " pod="ai-tenant-e2e-shared-a-efdc11/rate-test-model-efdc11-kserve-c7f66d5f-pg8jm" Jul 23 10:01:28.689658 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:28.689619 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b33d295e-fd2b-404b-81b6-11d12c07e750-model-cache\") pod \"rate-test-model-efdc11-kserve-c7f66d5f-pg8jm\" (UID: \"b33d295e-fd2b-404b-81b6-11d12c07e750\") " pod="ai-tenant-e2e-shared-a-efdc11/rate-test-model-efdc11-kserve-c7f66d5f-pg8jm" Jul 23 10:01:28.689658 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:28.689658 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b33d295e-fd2b-404b-81b6-11d12c07e750-home\") pod \"rate-test-model-efdc11-kserve-c7f66d5f-pg8jm\" (UID: \"b33d295e-fd2b-404b-81b6-11d12c07e750\") " pod="ai-tenant-e2e-shared-a-efdc11/rate-test-model-efdc11-kserve-c7f66d5f-pg8jm" Jul 23 10:01:28.689924 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:28.689715 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b33d295e-fd2b-404b-81b6-11d12c07e750-tmp-dir\") pod \"rate-test-model-efdc11-kserve-c7f66d5f-pg8jm\" (UID: \"b33d295e-fd2b-404b-81b6-11d12c07e750\") " pod="ai-tenant-e2e-shared-a-efdc11/rate-test-model-efdc11-kserve-c7f66d5f-pg8jm" Jul 23 10:01:28.689924 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:28.689752 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hh99\" (UniqueName: \"kubernetes.io/projected/b33d295e-fd2b-404b-81b6-11d12c07e750-kube-api-access-6hh99\") pod \"rate-test-model-efdc11-kserve-c7f66d5f-pg8jm\" (UID: \"b33d295e-fd2b-404b-81b6-11d12c07e750\") " pod="ai-tenant-e2e-shared-a-efdc11/rate-test-model-efdc11-kserve-c7f66d5f-pg8jm" Jul 23 10:01:28.689924 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:28.689803 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b33d295e-fd2b-404b-81b6-11d12c07e750-tls-certs\") pod \"rate-test-model-efdc11-kserve-c7f66d5f-pg8jm\" (UID: \"b33d295e-fd2b-404b-81b6-11d12c07e750\") " pod="ai-tenant-e2e-shared-a-efdc11/rate-test-model-efdc11-kserve-c7f66d5f-pg8jm" Jul 23 10:01:28.689924 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:28.689829 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b33d295e-fd2b-404b-81b6-11d12c07e750-dshm\") pod \"rate-test-model-efdc11-kserve-c7f66d5f-pg8jm\" (UID: \"b33d295e-fd2b-404b-81b6-11d12c07e750\") " pod="ai-tenant-e2e-shared-a-efdc11/rate-test-model-efdc11-kserve-c7f66d5f-pg8jm" Jul 23 10:01:28.690124 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:28.690062 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b33d295e-fd2b-404b-81b6-11d12c07e750-model-cache\") pod \"rate-test-model-efdc11-kserve-c7f66d5f-pg8jm\" (UID: \"b33d295e-fd2b-404b-81b6-11d12c07e750\") " pod="ai-tenant-e2e-shared-a-efdc11/rate-test-model-efdc11-kserve-c7f66d5f-pg8jm" Jul 23 10:01:28.690124 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:28.690085 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b33d295e-fd2b-404b-81b6-11d12c07e750-home\") pod \"rate-test-model-efdc11-kserve-c7f66d5f-pg8jm\" (UID: \"b33d295e-fd2b-404b-81b6-11d12c07e750\") " pod="ai-tenant-e2e-shared-a-efdc11/rate-test-model-efdc11-kserve-c7f66d5f-pg8jm" Jul 23 10:01:28.690350 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:28.690331 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b33d295e-fd2b-404b-81b6-11d12c07e750-tmp-dir\") pod \"rate-test-model-efdc11-kserve-c7f66d5f-pg8jm\" (UID: \"b33d295e-fd2b-404b-81b6-11d12c07e750\") " pod="ai-tenant-e2e-shared-a-efdc11/rate-test-model-efdc11-kserve-c7f66d5f-pg8jm" Jul 23 10:01:28.692356 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:28.692332 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b33d295e-fd2b-404b-81b6-11d12c07e750-dshm\") pod \"rate-test-model-efdc11-kserve-c7f66d5f-pg8jm\" (UID: \"b33d295e-fd2b-404b-81b6-11d12c07e750\") " pod="ai-tenant-e2e-shared-a-efdc11/rate-test-model-efdc11-kserve-c7f66d5f-pg8jm" Jul 23 10:01:28.692654 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:28.692635 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b33d295e-fd2b-404b-81b6-11d12c07e750-tls-certs\") pod \"rate-test-model-efdc11-kserve-c7f66d5f-pg8jm\" (UID: \"b33d295e-fd2b-404b-81b6-11d12c07e750\") " pod="ai-tenant-e2e-shared-a-efdc11/rate-test-model-efdc11-kserve-c7f66d5f-pg8jm" Jul 23 10:01:28.699129 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:28.699111 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hh99\" (UniqueName: \"kubernetes.io/projected/b33d295e-fd2b-404b-81b6-11d12c07e750-kube-api-access-6hh99\") pod \"rate-test-model-efdc11-kserve-c7f66d5f-pg8jm\" (UID: \"b33d295e-fd2b-404b-81b6-11d12c07e750\") " pod="ai-tenant-e2e-shared-a-efdc11/rate-test-model-efdc11-kserve-c7f66d5f-pg8jm" Jul 23 10:01:28.817294 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:28.817172 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-efdc11/rate-test-model-efdc11-kserve-c7f66d5f-pg8jm" Jul 23 10:01:28.950738 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:28.950704 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-efdc11/rate-test-model-efdc11-kserve-c7f66d5f-pg8jm"] Jul 23 10:01:28.952666 ip-10-0-135-145 kubenswrapper[2576]: W0723 10:01:28.952631 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb33d295e_fd2b_404b_81b6_11d12c07e750.slice/crio-91827fdaba4afc36c9c170662ea81f0b4c60fa77252be6954374081ac34c99b4 WatchSource:0}: Error finding container 91827fdaba4afc36c9c170662ea81f0b4c60fa77252be6954374081ac34c99b4: Status 404 returned error can't find the container with id 91827fdaba4afc36c9c170662ea81f0b4c60fa77252be6954374081ac34c99b4 Jul 23 10:01:29.059110 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:29.059076 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-efdc11/rate-test-model-efdc11-kserve-c7f66d5f-pg8jm" event={"ID":"b33d295e-fd2b-404b-81b6-11d12c07e750","Type":"ContainerStarted","Data":"0a95b81116c2c140a0c4880f80a6de2ad34c2f281eed9aade602d3e96d242f10"} Jul 23 10:01:29.059253 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:29.059118 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-efdc11/rate-test-model-efdc11-kserve-c7f66d5f-pg8jm" event={"ID":"b33d295e-fd2b-404b-81b6-11d12c07e750","Type":"ContainerStarted","Data":"91827fdaba4afc36c9c170662ea81f0b4c60fa77252be6954374081ac34c99b4"} Jul 23 10:01:29.081083 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:29.080993 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-efdc11/rate-test-model-efdc11-kserve-c7f66d5f-pg8jm" podStartSLOduration=1.080978804 podStartE2EDuration="1.080978804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 10:01:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 10:01:29.080617395 +0000 UTC m=+2677.619895975" watchObservedRunningTime="2026-07-23 10:01:29.080978804 +0000 UTC m=+2677.620257382" Jul 23 10:01:32.538329 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:32.538283 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-efdc11/sub-test-model-efdc11-kserve-69dfc4965f-fzz9q" podUID="e33e441d-9f34-45d5-aa67-01e269bf52d2" containerName="main" containerID="cri-o://8f6acbefa45b24a3d7ae04f87090a92d1bc49282b4354fc6023aeeedee9e8e2b" gracePeriod=45 Jul 23 10:01:32.597646 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:32.597602 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-efdc11/sub-test-model-efdc11-kserve-69dfc4965f-fzz9q" podUID="e33e441d-9f34-45d5-aa67-01e269bf52d2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": dial tcp 10.132.0.60:8000: connect: connection refused" Jul 23 10:01:32.798313 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:32.798229 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-efdc11/sub-test-model-efdc11-kserve-69dfc4965f-fzz9q" Jul 23 10:01:32.928963 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:32.928919 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e33e441d-9f34-45d5-aa67-01e269bf52d2-dshm\") pod \"e33e441d-9f34-45d5-aa67-01e269bf52d2\" (UID: \"e33e441d-9f34-45d5-aa67-01e269bf52d2\") " Jul 23 10:01:32.928963 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:32.928970 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e33e441d-9f34-45d5-aa67-01e269bf52d2-model-cache\") pod \"e33e441d-9f34-45d5-aa67-01e269bf52d2\" (UID: \"e33e441d-9f34-45d5-aa67-01e269bf52d2\") " Jul 23 10:01:32.929234 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:32.929001 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cnpn4\" (UniqueName: \"kubernetes.io/projected/e33e441d-9f34-45d5-aa67-01e269bf52d2-kube-api-access-cnpn4\") pod \"e33e441d-9f34-45d5-aa67-01e269bf52d2\" (UID: \"e33e441d-9f34-45d5-aa67-01e269bf52d2\") " Jul 23 10:01:32.929234 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:32.929036 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e33e441d-9f34-45d5-aa67-01e269bf52d2-tmp-dir\") pod \"e33e441d-9f34-45d5-aa67-01e269bf52d2\" (UID: \"e33e441d-9f34-45d5-aa67-01e269bf52d2\") " Jul 23 10:01:32.929234 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:32.929096 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e33e441d-9f34-45d5-aa67-01e269bf52d2-tls-certs\") pod \"e33e441d-9f34-45d5-aa67-01e269bf52d2\" (UID: \"e33e441d-9f34-45d5-aa67-01e269bf52d2\") " Jul 23 10:01:32.929234 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:32.929137 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e33e441d-9f34-45d5-aa67-01e269bf52d2-home\") pod \"e33e441d-9f34-45d5-aa67-01e269bf52d2\" (UID: \"e33e441d-9f34-45d5-aa67-01e269bf52d2\") " Jul 23 10:01:32.929482 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:32.929234 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e33e441d-9f34-45d5-aa67-01e269bf52d2-model-cache" (OuterVolumeSpecName: "model-cache") pod "e33e441d-9f34-45d5-aa67-01e269bf52d2" (UID: "e33e441d-9f34-45d5-aa67-01e269bf52d2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 10:01:32.929482 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:32.929372 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e33e441d-9f34-45d5-aa67-01e269bf52d2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e33e441d-9f34-45d5-aa67-01e269bf52d2" (UID: "e33e441d-9f34-45d5-aa67-01e269bf52d2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 10:01:32.929584 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:32.929492 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e33e441d-9f34-45d5-aa67-01e269bf52d2-home" (OuterVolumeSpecName: "home") pod "e33e441d-9f34-45d5-aa67-01e269bf52d2" (UID: "e33e441d-9f34-45d5-aa67-01e269bf52d2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 10:01:32.929584 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:32.929508 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e33e441d-9f34-45d5-aa67-01e269bf52d2-model-cache\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 10:01:32.929584 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:32.929526 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e33e441d-9f34-45d5-aa67-01e269bf52d2-tmp-dir\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 10:01:32.931214 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:32.931195 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e33e441d-9f34-45d5-aa67-01e269bf52d2-dshm" (OuterVolumeSpecName: "dshm") pod "e33e441d-9f34-45d5-aa67-01e269bf52d2" (UID: "e33e441d-9f34-45d5-aa67-01e269bf52d2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 10:01:32.931305 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:32.931231 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e33e441d-9f34-45d5-aa67-01e269bf52d2-kube-api-access-cnpn4" (OuterVolumeSpecName: "kube-api-access-cnpn4") pod "e33e441d-9f34-45d5-aa67-01e269bf52d2" (UID: "e33e441d-9f34-45d5-aa67-01e269bf52d2"). InnerVolumeSpecName "kube-api-access-cnpn4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 10:01:32.931348 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:32.931319 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e33e441d-9f34-45d5-aa67-01e269bf52d2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e33e441d-9f34-45d5-aa67-01e269bf52d2" (UID: "e33e441d-9f34-45d5-aa67-01e269bf52d2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 10:01:33.030226 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:33.030192 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e33e441d-9f34-45d5-aa67-01e269bf52d2-tls-certs\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 10:01:33.030226 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:33.030221 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e33e441d-9f34-45d5-aa67-01e269bf52d2-home\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 10:01:33.030226 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:33.030233 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e33e441d-9f34-45d5-aa67-01e269bf52d2-dshm\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 10:01:33.030452 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:33.030260 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnpn4\" (UniqueName: \"kubernetes.io/projected/e33e441d-9f34-45d5-aa67-01e269bf52d2-kube-api-access-cnpn4\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 10:01:33.080893 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:33.080818 2576 generic.go:356] "Generic (PLEG): container finished" podID="e33e441d-9f34-45d5-aa67-01e269bf52d2" containerID="8f6acbefa45b24a3d7ae04f87090a92d1bc49282b4354fc6023aeeedee9e8e2b" exitCode=0 Jul 23 10:01:33.080893 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:33.080866 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-efdc11/sub-test-model-efdc11-kserve-69dfc4965f-fzz9q" Jul 23 10:01:33.081037 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:33.080905 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-efdc11/sub-test-model-efdc11-kserve-69dfc4965f-fzz9q" event={"ID":"e33e441d-9f34-45d5-aa67-01e269bf52d2","Type":"ContainerDied","Data":"8f6acbefa45b24a3d7ae04f87090a92d1bc49282b4354fc6023aeeedee9e8e2b"} Jul 23 10:01:33.081037 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:33.080949 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-efdc11/sub-test-model-efdc11-kserve-69dfc4965f-fzz9q" event={"ID":"e33e441d-9f34-45d5-aa67-01e269bf52d2","Type":"ContainerDied","Data":"4824541c54e0be64485e576a80a2cfc81b4befd280a9236cd133e78509a543cf"} Jul 23 10:01:33.081037 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:33.080971 2576 scope.go:117] "RemoveContainer" containerID="8f6acbefa45b24a3d7ae04f87090a92d1bc49282b4354fc6023aeeedee9e8e2b" Jul 23 10:01:33.089975 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:33.089957 2576 scope.go:117] "RemoveContainer" containerID="8f6acbefa45b24a3d7ae04f87090a92d1bc49282b4354fc6023aeeedee9e8e2b" Jul 23 10:01:33.090228 ip-10-0-135-145 kubenswrapper[2576]: E0723 10:01:33.090206 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f6acbefa45b24a3d7ae04f87090a92d1bc49282b4354fc6023aeeedee9e8e2b\": container with ID starting with 8f6acbefa45b24a3d7ae04f87090a92d1bc49282b4354fc6023aeeedee9e8e2b not found: ID does not exist" containerID="8f6acbefa45b24a3d7ae04f87090a92d1bc49282b4354fc6023aeeedee9e8e2b" Jul 23 10:01:33.090302 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:33.090259 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f6acbefa45b24a3d7ae04f87090a92d1bc49282b4354fc6023aeeedee9e8e2b"} err="failed to get container status \"8f6acbefa45b24a3d7ae04f87090a92d1bc49282b4354fc6023aeeedee9e8e2b\": rpc error: code = NotFound desc = could not find container \"8f6acbefa45b24a3d7ae04f87090a92d1bc49282b4354fc6023aeeedee9e8e2b\": container with ID starting with 8f6acbefa45b24a3d7ae04f87090a92d1bc49282b4354fc6023aeeedee9e8e2b not found: ID does not exist" Jul 23 10:01:33.098262 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:33.098225 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-efdc11/sub-test-model-efdc11-kserve-69dfc4965f-fzz9q"] Jul 23 10:01:33.108801 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:33.108776 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-efdc11/sub-test-model-efdc11-kserve-69dfc4965f-fzz9q"] Jul 23 10:01:33.408884 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:33.408836 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-0dc985/sub-test-model-0dc985-kserve-69f6b8d85-8gvsz" podUID="fa3bde2d-0fe6-4207-9697-c3d95a78fd59" containerName="main" containerID="cri-o://67c55f2041735d893281ebd07ccfca5b8d9a671a0face87ce3ac7ed82e6665f8" gracePeriod=45 Jul 23 10:01:33.664379 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:33.664317 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0dc985/sub-test-model-0dc985-kserve-69f6b8d85-8gvsz" Jul 23 10:01:33.735874 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:33.735841 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa3bde2d-0fe6-4207-9697-c3d95a78fd59-model-cache\") pod \"fa3bde2d-0fe6-4207-9697-c3d95a78fd59\" (UID: \"fa3bde2d-0fe6-4207-9697-c3d95a78fd59\") " Jul 23 10:01:33.736043 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:33.735892 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa3bde2d-0fe6-4207-9697-c3d95a78fd59-tmp-dir\") pod \"fa3bde2d-0fe6-4207-9697-c3d95a78fd59\" (UID: \"fa3bde2d-0fe6-4207-9697-c3d95a78fd59\") " Jul 23 10:01:33.736043 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:33.735990 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2gc2b\" (UniqueName: \"kubernetes.io/projected/fa3bde2d-0fe6-4207-9697-c3d95a78fd59-kube-api-access-2gc2b\") pod \"fa3bde2d-0fe6-4207-9697-c3d95a78fd59\" (UID: \"fa3bde2d-0fe6-4207-9697-c3d95a78fd59\") " Jul 23 10:01:33.736043 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:33.736028 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa3bde2d-0fe6-4207-9697-c3d95a78fd59-dshm\") pod \"fa3bde2d-0fe6-4207-9697-c3d95a78fd59\" (UID: \"fa3bde2d-0fe6-4207-9697-c3d95a78fd59\") " Jul 23 10:01:33.736172 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:33.736062 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa3bde2d-0fe6-4207-9697-c3d95a78fd59-tls-certs\") pod \"fa3bde2d-0fe6-4207-9697-c3d95a78fd59\" (UID: \"fa3bde2d-0fe6-4207-9697-c3d95a78fd59\") " Jul 23 10:01:33.736172 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:33.736115 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa3bde2d-0fe6-4207-9697-c3d95a78fd59-home\") pod \"fa3bde2d-0fe6-4207-9697-c3d95a78fd59\" (UID: \"fa3bde2d-0fe6-4207-9697-c3d95a78fd59\") " Jul 23 10:01:33.736602 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:33.736160 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa3bde2d-0fe6-4207-9697-c3d95a78fd59-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fa3bde2d-0fe6-4207-9697-c3d95a78fd59" (UID: "fa3bde2d-0fe6-4207-9697-c3d95a78fd59"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 10:01:33.736602 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:33.736174 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa3bde2d-0fe6-4207-9697-c3d95a78fd59-model-cache" (OuterVolumeSpecName: "model-cache") pod "fa3bde2d-0fe6-4207-9697-c3d95a78fd59" (UID: "fa3bde2d-0fe6-4207-9697-c3d95a78fd59"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 10:01:33.736602 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:33.736370 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa3bde2d-0fe6-4207-9697-c3d95a78fd59-model-cache\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 10:01:33.736602 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:33.736404 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa3bde2d-0fe6-4207-9697-c3d95a78fd59-tmp-dir\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 10:01:33.736602 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:33.736452 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa3bde2d-0fe6-4207-9697-c3d95a78fd59-home" (OuterVolumeSpecName: "home") pod "fa3bde2d-0fe6-4207-9697-c3d95a78fd59" (UID: "fa3bde2d-0fe6-4207-9697-c3d95a78fd59"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 10:01:33.738225 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:33.738197 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa3bde2d-0fe6-4207-9697-c3d95a78fd59-dshm" (OuterVolumeSpecName: "dshm") pod "fa3bde2d-0fe6-4207-9697-c3d95a78fd59" (UID: "fa3bde2d-0fe6-4207-9697-c3d95a78fd59"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 10:01:33.738361 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:33.738228 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa3bde2d-0fe6-4207-9697-c3d95a78fd59-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fa3bde2d-0fe6-4207-9697-c3d95a78fd59" (UID: "fa3bde2d-0fe6-4207-9697-c3d95a78fd59"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 10:01:33.738361 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:33.738288 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa3bde2d-0fe6-4207-9697-c3d95a78fd59-kube-api-access-2gc2b" (OuterVolumeSpecName: "kube-api-access-2gc2b") pod "fa3bde2d-0fe6-4207-9697-c3d95a78fd59" (UID: "fa3bde2d-0fe6-4207-9697-c3d95a78fd59"). InnerVolumeSpecName "kube-api-access-2gc2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 10:01:33.837800 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:33.837761 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gc2b\" (UniqueName: \"kubernetes.io/projected/fa3bde2d-0fe6-4207-9697-c3d95a78fd59-kube-api-access-2gc2b\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 10:01:33.837800 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:33.837793 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa3bde2d-0fe6-4207-9697-c3d95a78fd59-dshm\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 10:01:33.837800 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:33.837804 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa3bde2d-0fe6-4207-9697-c3d95a78fd59-tls-certs\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 10:01:33.837800 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:33.837813 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa3bde2d-0fe6-4207-9697-c3d95a78fd59-home\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 10:01:34.089094 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:34.089001 2576 generic.go:356] "Generic (PLEG): container finished" podID="fa3bde2d-0fe6-4207-9697-c3d95a78fd59" containerID="67c55f2041735d893281ebd07ccfca5b8d9a671a0face87ce3ac7ed82e6665f8" exitCode=0 Jul 23 10:01:34.089094 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:34.089048 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0dc985/sub-test-model-0dc985-kserve-69f6b8d85-8gvsz" Jul 23 10:01:34.089094 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:34.089084 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0dc985/sub-test-model-0dc985-kserve-69f6b8d85-8gvsz" event={"ID":"fa3bde2d-0fe6-4207-9697-c3d95a78fd59","Type":"ContainerDied","Data":"67c55f2041735d893281ebd07ccfca5b8d9a671a0face87ce3ac7ed82e6665f8"} Jul 23 10:01:34.089348 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:34.089282 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0dc985/sub-test-model-0dc985-kserve-69f6b8d85-8gvsz" event={"ID":"fa3bde2d-0fe6-4207-9697-c3d95a78fd59","Type":"ContainerDied","Data":"56d2ca62b7c3035bacd70c695b3fc0d782064782e4ef576212661f5b6f2b1eb7"} Jul 23 10:01:34.089348 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:34.089311 2576 scope.go:117] "RemoveContainer" containerID="67c55f2041735d893281ebd07ccfca5b8d9a671a0face87ce3ac7ed82e6665f8" Jul 23 10:01:34.098867 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:34.098849 2576 scope.go:117] "RemoveContainer" containerID="67c55f2041735d893281ebd07ccfca5b8d9a671a0face87ce3ac7ed82e6665f8" Jul 23 10:01:34.099113 ip-10-0-135-145 kubenswrapper[2576]: E0723 10:01:34.099095 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67c55f2041735d893281ebd07ccfca5b8d9a671a0face87ce3ac7ed82e6665f8\": container with ID starting with 67c55f2041735d893281ebd07ccfca5b8d9a671a0face87ce3ac7ed82e6665f8 not found: ID does not exist" containerID="67c55f2041735d893281ebd07ccfca5b8d9a671a0face87ce3ac7ed82e6665f8" Jul 23 10:01:34.099159 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:34.099122 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67c55f2041735d893281ebd07ccfca5b8d9a671a0face87ce3ac7ed82e6665f8"} err="failed to get container status \"67c55f2041735d893281ebd07ccfca5b8d9a671a0face87ce3ac7ed82e6665f8\": rpc error: code = NotFound desc = could not find container \"67c55f2041735d893281ebd07ccfca5b8d9a671a0face87ce3ac7ed82e6665f8\": container with ID starting with 67c55f2041735d893281ebd07ccfca5b8d9a671a0face87ce3ac7ed82e6665f8 not found: ID does not exist" Jul 23 10:01:34.110834 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:34.110804 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-0dc985/sub-test-model-0dc985-kserve-69f6b8d85-8gvsz"] Jul 23 10:01:34.113590 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:34.113555 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-0dc985/sub-test-model-0dc985-kserve-69f6b8d85-8gvsz"] Jul 23 10:01:34.987228 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:34.987193 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e33e441d-9f34-45d5-aa67-01e269bf52d2" path="/var/lib/kubelet/pods/e33e441d-9f34-45d5-aa67-01e269bf52d2/volumes" Jul 23 10:01:34.987650 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:34.987604 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa3bde2d-0fe6-4207-9697-c3d95a78fd59" path="/var/lib/kubelet/pods/fa3bde2d-0fe6-4207-9697-c3d95a78fd59/volumes" Jul 23 10:01:37.494563 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:37.494529 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 10:01:38.817491 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:38.817457 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-efdc11/rate-test-model-efdc11-kserve-c7f66d5f-pg8jm" Jul 23 10:01:38.817939 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:38.817508 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-efdc11/rate-test-model-efdc11-kserve-c7f66d5f-pg8jm" Jul 23 10:01:38.829999 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:38.829971 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-efdc11/rate-test-model-efdc11-kserve-c7f66d5f-pg8jm" Jul 23 10:01:39.127745 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:39.127714 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-efdc11/rate-test-model-efdc11-kserve-c7f66d5f-pg8jm" Jul 23 10:01:44.081844 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:44.081747 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 10:01:49.481850 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:49.481817 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-0dc985/rate-test-model-0dc985-kserve-bd7857d7f-j6z5s"] Jul 23 10:01:49.482208 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:49.482166 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa3bde2d-0fe6-4207-9697-c3d95a78fd59" containerName="main" Jul 23 10:01:49.482208 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:49.482176 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa3bde2d-0fe6-4207-9697-c3d95a78fd59" containerName="main" Jul 23 10:01:49.482208 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:49.482200 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e33e441d-9f34-45d5-aa67-01e269bf52d2" containerName="main" Jul 23 10:01:49.482208 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:49.482205 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e33e441d-9f34-45d5-aa67-01e269bf52d2" containerName="main" Jul 23 10:01:49.482373 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:49.482275 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa3bde2d-0fe6-4207-9697-c3d95a78fd59" containerName="main" Jul 23 10:01:49.482373 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:49.482285 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e33e441d-9f34-45d5-aa67-01e269bf52d2" containerName="main" Jul 23 10:01:49.486693 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:49.486670 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0dc985/rate-test-model-0dc985-kserve-bd7857d7f-j6z5s" Jul 23 10:01:49.489084 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:49.489059 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-0dc985\"/\"openshift-service-ca.crt\"" Jul 23 10:01:49.490024 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:49.490001 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-0dc985\"/\"kube-root-ca.crt\"" Jul 23 10:01:49.490123 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:49.490001 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-0dc985\"/\"rate-test-model-0dc985-kserve-self-signed-certs\"" Jul 23 10:01:49.493148 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:49.493031 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-0dc985\"/\"default-dockercfg-47rx4\"" Jul 23 10:01:49.499219 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:49.499195 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-0dc985/rate-test-model-0dc985-kserve-bd7857d7f-j6z5s"] Jul 23 10:01:49.519821 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:49.519791 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 10:01:49.586512 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:49.586476 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ed575ba-c014-4973-b971-443a3b5ef256-tls-certs\") pod \"rate-test-model-0dc985-kserve-bd7857d7f-j6z5s\" (UID: \"7ed575ba-c014-4973-b971-443a3b5ef256\") " pod="ai-tenant-e2e-shared-b-0dc985/rate-test-model-0dc985-kserve-bd7857d7f-j6z5s" Jul 23 10:01:49.586512 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:49.586510 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww26q\" (UniqueName: \"kubernetes.io/projected/7ed575ba-c014-4973-b971-443a3b5ef256-kube-api-access-ww26q\") pod \"rate-test-model-0dc985-kserve-bd7857d7f-j6z5s\" (UID: \"7ed575ba-c014-4973-b971-443a3b5ef256\") " pod="ai-tenant-e2e-shared-b-0dc985/rate-test-model-0dc985-kserve-bd7857d7f-j6z5s" Jul 23 10:01:49.586712 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:49.586532 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ed575ba-c014-4973-b971-443a3b5ef256-dshm\") pod \"rate-test-model-0dc985-kserve-bd7857d7f-j6z5s\" (UID: \"7ed575ba-c014-4973-b971-443a3b5ef256\") " pod="ai-tenant-e2e-shared-b-0dc985/rate-test-model-0dc985-kserve-bd7857d7f-j6z5s" Jul 23 10:01:49.586712 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:49.586643 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ed575ba-c014-4973-b971-443a3b5ef256-tmp-dir\") pod \"rate-test-model-0dc985-kserve-bd7857d7f-j6z5s\" (UID: \"7ed575ba-c014-4973-b971-443a3b5ef256\") " pod="ai-tenant-e2e-shared-b-0dc985/rate-test-model-0dc985-kserve-bd7857d7f-j6z5s" Jul 23 10:01:49.586712 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:49.586670 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ed575ba-c014-4973-b971-443a3b5ef256-model-cache\") pod \"rate-test-model-0dc985-kserve-bd7857d7f-j6z5s\" (UID: \"7ed575ba-c014-4973-b971-443a3b5ef256\") " pod="ai-tenant-e2e-shared-b-0dc985/rate-test-model-0dc985-kserve-bd7857d7f-j6z5s" Jul 23 10:01:49.586712 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:49.586691 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ed575ba-c014-4973-b971-443a3b5ef256-home\") pod \"rate-test-model-0dc985-kserve-bd7857d7f-j6z5s\" (UID: \"7ed575ba-c014-4973-b971-443a3b5ef256\") " pod="ai-tenant-e2e-shared-b-0dc985/rate-test-model-0dc985-kserve-bd7857d7f-j6z5s" Jul 23 10:01:49.687848 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:49.687797 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ed575ba-c014-4973-b971-443a3b5ef256-tls-certs\") pod \"rate-test-model-0dc985-kserve-bd7857d7f-j6z5s\" (UID: \"7ed575ba-c014-4973-b971-443a3b5ef256\") " pod="ai-tenant-e2e-shared-b-0dc985/rate-test-model-0dc985-kserve-bd7857d7f-j6z5s" Jul 23 10:01:49.687848 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:49.687847 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww26q\" (UniqueName: \"kubernetes.io/projected/7ed575ba-c014-4973-b971-443a3b5ef256-kube-api-access-ww26q\") pod \"rate-test-model-0dc985-kserve-bd7857d7f-j6z5s\" (UID: \"7ed575ba-c014-4973-b971-443a3b5ef256\") " pod="ai-tenant-e2e-shared-b-0dc985/rate-test-model-0dc985-kserve-bd7857d7f-j6z5s" Jul 23 10:01:49.688097 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:49.687875 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ed575ba-c014-4973-b971-443a3b5ef256-dshm\") pod \"rate-test-model-0dc985-kserve-bd7857d7f-j6z5s\" (UID: \"7ed575ba-c014-4973-b971-443a3b5ef256\") " pod="ai-tenant-e2e-shared-b-0dc985/rate-test-model-0dc985-kserve-bd7857d7f-j6z5s" Jul 23 10:01:49.688097 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:49.687936 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ed575ba-c014-4973-b971-443a3b5ef256-tmp-dir\") pod \"rate-test-model-0dc985-kserve-bd7857d7f-j6z5s\" (UID: \"7ed575ba-c014-4973-b971-443a3b5ef256\") " pod="ai-tenant-e2e-shared-b-0dc985/rate-test-model-0dc985-kserve-bd7857d7f-j6z5s" Jul 23 10:01:49.688097 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:49.687953 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ed575ba-c014-4973-b971-443a3b5ef256-model-cache\") pod \"rate-test-model-0dc985-kserve-bd7857d7f-j6z5s\" (UID: \"7ed575ba-c014-4973-b971-443a3b5ef256\") " pod="ai-tenant-e2e-shared-b-0dc985/rate-test-model-0dc985-kserve-bd7857d7f-j6z5s" Jul 23 10:01:49.688097 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:49.687974 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ed575ba-c014-4973-b971-443a3b5ef256-home\") pod \"rate-test-model-0dc985-kserve-bd7857d7f-j6z5s\" (UID: \"7ed575ba-c014-4973-b971-443a3b5ef256\") " pod="ai-tenant-e2e-shared-b-0dc985/rate-test-model-0dc985-kserve-bd7857d7f-j6z5s" Jul 23 10:01:49.688441 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:49.688414 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ed575ba-c014-4973-b971-443a3b5ef256-model-cache\") pod \"rate-test-model-0dc985-kserve-bd7857d7f-j6z5s\" (UID: \"7ed575ba-c014-4973-b971-443a3b5ef256\") " pod="ai-tenant-e2e-shared-b-0dc985/rate-test-model-0dc985-kserve-bd7857d7f-j6z5s" Jul 23 10:01:49.688519 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:49.688428 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ed575ba-c014-4973-b971-443a3b5ef256-tmp-dir\") pod \"rate-test-model-0dc985-kserve-bd7857d7f-j6z5s\" (UID: \"7ed575ba-c014-4973-b971-443a3b5ef256\") " pod="ai-tenant-e2e-shared-b-0dc985/rate-test-model-0dc985-kserve-bd7857d7f-j6z5s" Jul 23 10:01:49.688519 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:49.688475 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ed575ba-c014-4973-b971-443a3b5ef256-home\") pod \"rate-test-model-0dc985-kserve-bd7857d7f-j6z5s\" (UID: \"7ed575ba-c014-4973-b971-443a3b5ef256\") " pod="ai-tenant-e2e-shared-b-0dc985/rate-test-model-0dc985-kserve-bd7857d7f-j6z5s" Jul 23 10:01:49.690325 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:49.690303 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ed575ba-c014-4973-b971-443a3b5ef256-dshm\") pod \"rate-test-model-0dc985-kserve-bd7857d7f-j6z5s\" (UID: \"7ed575ba-c014-4973-b971-443a3b5ef256\") " pod="ai-tenant-e2e-shared-b-0dc985/rate-test-model-0dc985-kserve-bd7857d7f-j6z5s" Jul 23 10:01:49.690496 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:49.690479 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ed575ba-c014-4973-b971-443a3b5ef256-tls-certs\") pod \"rate-test-model-0dc985-kserve-bd7857d7f-j6z5s\" (UID: \"7ed575ba-c014-4973-b971-443a3b5ef256\") " pod="ai-tenant-e2e-shared-b-0dc985/rate-test-model-0dc985-kserve-bd7857d7f-j6z5s" Jul 23 10:01:49.696667 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:49.696645 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww26q\" (UniqueName: \"kubernetes.io/projected/7ed575ba-c014-4973-b971-443a3b5ef256-kube-api-access-ww26q\") pod \"rate-test-model-0dc985-kserve-bd7857d7f-j6z5s\" (UID: \"7ed575ba-c014-4973-b971-443a3b5ef256\") " pod="ai-tenant-e2e-shared-b-0dc985/rate-test-model-0dc985-kserve-bd7857d7f-j6z5s" Jul 23 10:01:49.795775 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:49.795678 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0dc985/rate-test-model-0dc985-kserve-bd7857d7f-j6z5s" Jul 23 10:01:49.935339 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:49.935305 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-0dc985/rate-test-model-0dc985-kserve-bd7857d7f-j6z5s"] Jul 23 10:01:49.937143 ip-10-0-135-145 kubenswrapper[2576]: W0723 10:01:49.937119 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7ed575ba_c014_4973_b971_443a3b5ef256.slice/crio-ebc0df65d092e794e0753399b0291cb7e71edca39b9eaa357297cc341b98a939 WatchSource:0}: Error finding container ebc0df65d092e794e0753399b0291cb7e71edca39b9eaa357297cc341b98a939: Status 404 returned error can't find the container with id ebc0df65d092e794e0753399b0291cb7e71edca39b9eaa357297cc341b98a939 Jul 23 10:01:50.186724 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:50.186682 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0dc985/rate-test-model-0dc985-kserve-bd7857d7f-j6z5s" event={"ID":"7ed575ba-c014-4973-b971-443a3b5ef256","Type":"ContainerStarted","Data":"a42c2d7433bdabe8a1724ba70d83d9d299398cef9a6cf8b6ff9a21a0ac202bb0"} Jul 23 10:01:50.186724 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:50.186725 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0dc985/rate-test-model-0dc985-kserve-bd7857d7f-j6z5s" event={"ID":"7ed575ba-c014-4973-b971-443a3b5ef256","Type":"ContainerStarted","Data":"ebc0df65d092e794e0753399b0291cb7e71edca39b9eaa357297cc341b98a939"} Jul 23 10:01:50.205847 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:50.205793 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-0dc985/rate-test-model-0dc985-kserve-bd7857d7f-j6z5s" podStartSLOduration=1.205778339 podStartE2EDuration="1.205778339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 10:01:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 10:01:50.204154361 +0000 UTC m=+2698.743432937" watchObservedRunningTime="2026-07-23 10:01:50.205778339 +0000 UTC m=+2698.745056915" Jul 23 10:01:52.116024 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:52.115994 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-5szxg_4908aff3-196e-44fa-b8e6-c0e81b476433/console-operator/2.log" Jul 23 10:01:52.124699 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:52.124675 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-5szxg_4908aff3-196e-44fa-b8e6-c0e81b476433/console-operator/2.log" Jul 23 10:01:52.260121 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:52.260090 2576 scope.go:117] "RemoveContainer" containerID="d7bd770a2c989acd8919e6bef7d0e2f8596bfb5dd3a68e96fc2a0349c852cc31" Jul 23 10:01:53.488048 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:53.488011 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 10:01:59.796642 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:59.796601 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-0dc985/rate-test-model-0dc985-kserve-bd7857d7f-j6z5s" Jul 23 10:01:59.797337 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:59.796747 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-0dc985/rate-test-model-0dc985-kserve-bd7857d7f-j6z5s" Jul 23 10:01:59.809386 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:01:59.809354 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-0dc985/rate-test-model-0dc985-kserve-bd7857d7f-j6z5s" Jul 23 10:02:00.252447 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:00.252416 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-0dc985/rate-test-model-0dc985-kserve-bd7857d7f-j6z5s" Jul 23 10:02:05.791366 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:05.791329 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 10:02:05.891685 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:05.891649 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 10:02:12.842881 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:12.842849 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 10:02:15.164631 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:15.164578 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 10:02:15.265119 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:15.265070 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 10:02:25.754983 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:25.754941 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 10:02:28.903314 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:28.903273 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 10:02:35.861018 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:35.860984 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-efdc11/rate-test-model-efdc11-kserve-c7f66d5f-pg8jm"] Jul 23 10:02:35.880902 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:35.880875 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 10:02:36.688676 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:36.688640 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-0dc985/rate-test-model-0dc985-kserve-bd7857d7f-j6z5s"] Jul 23 10:02:36.710865 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:36.710838 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 10:02:37.686111 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:37.686074 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7614-140371019447600"] Jul 23 10:02:37.689505 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:37.689487 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7614-140371019447600" Jul 23 10:02:37.692096 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:37.692072 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-7gmkp\"" Jul 23 10:02:37.694909 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:37.694883 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7614-140371019447600"] Jul 23 10:02:37.837381 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:37.837338 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgzq9\" (UniqueName: \"kubernetes.io/projected/738e9395-145c-458e-ac0c-e1d6f3a26188-kube-api-access-qgzq9\") pod \"test-curl-7614-140371019447600\" (UID: \"738e9395-145c-458e-ac0c-e1d6f3a26188\") " pod="opendatahub/test-curl-7614-140371019447600" Jul 23 10:02:37.938169 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:37.938072 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgzq9\" (UniqueName: \"kubernetes.io/projected/738e9395-145c-458e-ac0c-e1d6f3a26188-kube-api-access-qgzq9\") pod \"test-curl-7614-140371019447600\" (UID: \"738e9395-145c-458e-ac0c-e1d6f3a26188\") " pod="opendatahub/test-curl-7614-140371019447600" Jul 23 10:02:37.946830 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:37.946795 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgzq9\" (UniqueName: \"kubernetes.io/projected/738e9395-145c-458e-ac0c-e1d6f3a26188-kube-api-access-qgzq9\") pod \"test-curl-7614-140371019447600\" (UID: \"738e9395-145c-458e-ac0c-e1d6f3a26188\") " pod="opendatahub/test-curl-7614-140371019447600" Jul 23 10:02:37.999704 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:37.999671 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7614-140371019447600" Jul 23 10:02:38.135878 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:38.135849 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7614-140371019447600"] Jul 23 10:02:38.137909 ip-10-0-135-145 kubenswrapper[2576]: W0723 10:02:38.137885 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod738e9395_145c_458e_ac0c_e1d6f3a26188.slice/crio-240da7ccb66ed7cd57f9bba43f8d36a3ad21d8c63f244884306ccfcd3f5bcabb WatchSource:0}: Error finding container 240da7ccb66ed7cd57f9bba43f8d36a3ad21d8c63f244884306ccfcd3f5bcabb: Status 404 returned error can't find the container with id 240da7ccb66ed7cd57f9bba43f8d36a3ad21d8c63f244884306ccfcd3f5bcabb Jul 23 10:02:38.139147 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:38.139126 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 10:02:38.445641 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:38.445603 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7614-140371019447600" event={"ID":"738e9395-145c-458e-ac0c-e1d6f3a26188","Type":"ContainerStarted","Data":"240da7ccb66ed7cd57f9bba43f8d36a3ad21d8c63f244884306ccfcd3f5bcabb"} Jul 23 10:02:39.452370 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:39.452334 2576 generic.go:356] "Generic (PLEG): container finished" podID="738e9395-145c-458e-ac0c-e1d6f3a26188" containerID="1e2918d137c59692df6cd77040ca41a8194ed6291540b7231633eeca77237bce" exitCode=0 Jul 23 10:02:39.452767 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:39.452421 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7614-140371019447600" event={"ID":"738e9395-145c-458e-ac0c-e1d6f3a26188","Type":"ContainerDied","Data":"1e2918d137c59692df6cd77040ca41a8194ed6291540b7231633eeca77237bce"} Jul 23 10:02:40.591024 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:40.590997 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7614-140371019447600" Jul 23 10:02:40.612415 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:40.612382 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7614-140371019447600_738e9395-145c-458e-ac0c-e1d6f3a26188/test-curl-7614-140371019447600/0.log" Jul 23 10:02:40.637535 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:40.637503 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7614-140371019447600"] Jul 23 10:02:40.642403 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:40.642377 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7614-140371019447600"] Jul 23 10:02:40.660619 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:40.660584 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qgzq9\" (UniqueName: \"kubernetes.io/projected/738e9395-145c-458e-ac0c-e1d6f3a26188-kube-api-access-qgzq9\") pod \"738e9395-145c-458e-ac0c-e1d6f3a26188\" (UID: \"738e9395-145c-458e-ac0c-e1d6f3a26188\") " Jul 23 10:02:40.662823 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:40.662790 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/738e9395-145c-458e-ac0c-e1d6f3a26188-kube-api-access-qgzq9" (OuterVolumeSpecName: "kube-api-access-qgzq9") pod "738e9395-145c-458e-ac0c-e1d6f3a26188" (UID: "738e9395-145c-458e-ac0c-e1d6f3a26188"). InnerVolumeSpecName "kube-api-access-qgzq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 10:02:40.761429 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:40.761351 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgzq9\" (UniqueName: \"kubernetes.io/projected/738e9395-145c-458e-ac0c-e1d6f3a26188-kube-api-access-qgzq9\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 10:02:40.762532 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:40.762508 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7614-140371019448112"] Jul 23 10:02:40.762898 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:40.762885 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="738e9395-145c-458e-ac0c-e1d6f3a26188" containerName="test-curl-7614-140371019447600" Jul 23 10:02:40.762939 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:40.762899 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="738e9395-145c-458e-ac0c-e1d6f3a26188" containerName="test-curl-7614-140371019447600" Jul 23 10:02:40.762973 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:40.762959 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="738e9395-145c-458e-ac0c-e1d6f3a26188" containerName="test-curl-7614-140371019447600" Jul 23 10:02:40.765981 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:40.765965 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7614-140371019448112" Jul 23 10:02:40.771422 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:40.771396 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7614-140371019448112"] Jul 23 10:02:40.862551 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:40.862512 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snb9g\" (UniqueName: \"kubernetes.io/projected/f0820fa4-1782-4469-ad03-6933b7875440-kube-api-access-snb9g\") pod \"test-curl-7614-140371019448112\" (UID: \"f0820fa4-1782-4469-ad03-6933b7875440\") " pod="opendatahub/test-curl-7614-140371019448112" Jul 23 10:02:40.963096 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:40.963061 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snb9g\" (UniqueName: \"kubernetes.io/projected/f0820fa4-1782-4469-ad03-6933b7875440-kube-api-access-snb9g\") pod \"test-curl-7614-140371019448112\" (UID: \"f0820fa4-1782-4469-ad03-6933b7875440\") " pod="opendatahub/test-curl-7614-140371019448112" Jul 23 10:02:40.971542 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:40.971511 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-snb9g\" (UniqueName: \"kubernetes.io/projected/f0820fa4-1782-4469-ad03-6933b7875440-kube-api-access-snb9g\") pod \"test-curl-7614-140371019448112\" (UID: \"f0820fa4-1782-4469-ad03-6933b7875440\") " pod="opendatahub/test-curl-7614-140371019448112" Jul 23 10:02:40.985251 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:40.985214 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="738e9395-145c-458e-ac0c-e1d6f3a26188" path="/var/lib/kubelet/pods/738e9395-145c-458e-ac0c-e1d6f3a26188/volumes" Jul 23 10:02:41.078019 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:41.077942 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7614-140371019448112" Jul 23 10:02:41.223546 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:41.223501 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7614-140371019448112"] Jul 23 10:02:41.464569 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:41.464538 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7614-140371019447600" Jul 23 10:02:41.464790 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:41.464584 2576 scope.go:117] "RemoveContainer" containerID="1e2918d137c59692df6cd77040ca41a8194ed6291540b7231633eeca77237bce" Jul 23 10:02:41.465970 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:41.465937 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7614-140371019448112" event={"ID":"f0820fa4-1782-4469-ad03-6933b7875440","Type":"ContainerStarted","Data":"dd08486a41747010eb0156f95283301f26556fdf9100ddc33bbb8e3f2f32ba8e"} Jul 23 10:02:42.473685 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:42.473650 2576 generic.go:356] "Generic (PLEG): container finished" podID="f0820fa4-1782-4469-ad03-6933b7875440" containerID="452b7e3edae4e16d38f7fdc82b609f0486aa2a1f74d774c6ed617c79762c3880" exitCode=0 Jul 23 10:02:42.474098 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:42.473740 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7614-140371019448112" event={"ID":"f0820fa4-1782-4469-ad03-6933b7875440","Type":"ContainerDied","Data":"452b7e3edae4e16d38f7fdc82b609f0486aa2a1f74d774c6ed617c79762c3880"} Jul 23 10:02:43.611176 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:43.611149 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7614-140371019448112" Jul 23 10:02:43.631648 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:43.631616 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7614-140371019448112_f0820fa4-1782-4469-ad03-6933b7875440/test-curl-7614-140371019448112/0.log" Jul 23 10:02:43.656672 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:43.656643 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7614-140371019448112"] Jul 23 10:02:43.664686 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:43.664661 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7614-140371019448112"] Jul 23 10:02:43.685610 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:43.685587 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-snb9g\" (UniqueName: \"kubernetes.io/projected/f0820fa4-1782-4469-ad03-6933b7875440-kube-api-access-snb9g\") pod \"f0820fa4-1782-4469-ad03-6933b7875440\" (UID: \"f0820fa4-1782-4469-ad03-6933b7875440\") " Jul 23 10:02:43.687672 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:43.687650 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0820fa4-1782-4469-ad03-6933b7875440-kube-api-access-snb9g" (OuterVolumeSpecName: "kube-api-access-snb9g") pod "f0820fa4-1782-4469-ad03-6933b7875440" (UID: "f0820fa4-1782-4469-ad03-6933b7875440"). InnerVolumeSpecName "kube-api-access-snb9g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 10:02:43.787349 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:43.787282 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snb9g\" (UniqueName: \"kubernetes.io/projected/f0820fa4-1782-4469-ad03-6933b7875440-kube-api-access-snb9g\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 10:02:43.825961 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:43.825930 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7614-140371019592736"] Jul 23 10:02:43.826330 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:43.826315 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0820fa4-1782-4469-ad03-6933b7875440" containerName="test-curl-7614-140371019448112" Jul 23 10:02:43.826378 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:43.826333 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0820fa4-1782-4469-ad03-6933b7875440" containerName="test-curl-7614-140371019448112" Jul 23 10:02:43.826435 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:43.826425 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0820fa4-1782-4469-ad03-6933b7875440" containerName="test-curl-7614-140371019448112" Jul 23 10:02:43.829416 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:43.829401 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7614-140371019592736" Jul 23 10:02:43.835585 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:43.835551 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7614-140371019592736"] Jul 23 10:02:43.888569 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:43.888537 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zth7\" (UniqueName: \"kubernetes.io/projected/1e9ddf77-cd10-4c4b-9e79-cae5d396da8b-kube-api-access-9zth7\") pod \"test-curl-7614-140371019592736\" (UID: \"1e9ddf77-cd10-4c4b-9e79-cae5d396da8b\") " pod="opendatahub/test-curl-7614-140371019592736" Jul 23 10:02:43.989090 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:43.989054 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zth7\" (UniqueName: \"kubernetes.io/projected/1e9ddf77-cd10-4c4b-9e79-cae5d396da8b-kube-api-access-9zth7\") pod \"test-curl-7614-140371019592736\" (UID: \"1e9ddf77-cd10-4c4b-9e79-cae5d396da8b\") " pod="opendatahub/test-curl-7614-140371019592736" Jul 23 10:02:43.996544 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:43.996524 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zth7\" (UniqueName: \"kubernetes.io/projected/1e9ddf77-cd10-4c4b-9e79-cae5d396da8b-kube-api-access-9zth7\") pod \"test-curl-7614-140371019592736\" (UID: \"1e9ddf77-cd10-4c4b-9e79-cae5d396da8b\") " pod="opendatahub/test-curl-7614-140371019592736" Jul 23 10:02:44.140739 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:44.140707 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7614-140371019592736" Jul 23 10:02:44.477055 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:44.477031 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7614-140371019592736"] Jul 23 10:02:44.478941 ip-10-0-135-145 kubenswrapper[2576]: W0723 10:02:44.478917 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1e9ddf77_cd10_4c4b_9e79_cae5d396da8b.slice/crio-3a3e933b5c39cf7ce28b347ae1eb8d9d916b98c893f333f5bffb37ab9b176f1a WatchSource:0}: Error finding container 3a3e933b5c39cf7ce28b347ae1eb8d9d916b98c893f333f5bffb37ab9b176f1a: Status 404 returned error can't find the container with id 3a3e933b5c39cf7ce28b347ae1eb8d9d916b98c893f333f5bffb37ab9b176f1a Jul 23 10:02:44.485894 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:44.485864 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7614-140371019592736" event={"ID":"1e9ddf77-cd10-4c4b-9e79-cae5d396da8b","Type":"ContainerStarted","Data":"3a3e933b5c39cf7ce28b347ae1eb8d9d916b98c893f333f5bffb37ab9b176f1a"} Jul 23 10:02:44.487183 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:44.487159 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7614-140371019448112" Jul 23 10:02:44.487308 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:44.487198 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd08486a41747010eb0156f95283301f26556fdf9100ddc33bbb8e3f2f32ba8e" Jul 23 10:02:44.497587 ip-10-0-135-145 kubenswrapper[2576]: E0723 10:02:44.497559 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7614-140371019448112\" is forbidden: User \"system:node:ip-10-0-135-145.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-145.ec2.internal' and this object" podUID="f0820fa4-1782-4469-ad03-6933b7875440" pod="opendatahub/test-curl-7614-140371019448112" Jul 23 10:02:44.985847 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:44.985818 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0820fa4-1782-4469-ad03-6933b7875440" path="/var/lib/kubelet/pods/f0820fa4-1782-4469-ad03-6933b7875440/volumes" Jul 23 10:02:45.493441 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:45.493407 2576 generic.go:356] "Generic (PLEG): container finished" podID="1e9ddf77-cd10-4c4b-9e79-cae5d396da8b" containerID="ae83117a7f7dabebd6b073fc28533ef8af3908b5dc0927915c9b1b87ba4cfd06" exitCode=0 Jul 23 10:02:45.493629 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:45.493491 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7614-140371019592736" event={"ID":"1e9ddf77-cd10-4c4b-9e79-cae5d396da8b","Type":"ContainerDied","Data":"ae83117a7f7dabebd6b073fc28533ef8af3908b5dc0927915c9b1b87ba4cfd06"} Jul 23 10:02:46.657862 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:46.657836 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7614-140371019592736" Jul 23 10:02:46.681914 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:46.681884 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7614-140371019592736_1e9ddf77-cd10-4c4b-9e79-cae5d396da8b/test-curl-7614-140371019592736/0.log" Jul 23 10:02:46.708317 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:46.708286 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7614-140371019592736"] Jul 23 10:02:46.712217 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:46.712193 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9zth7\" (UniqueName: \"kubernetes.io/projected/1e9ddf77-cd10-4c4b-9e79-cae5d396da8b-kube-api-access-9zth7\") pod \"1e9ddf77-cd10-4c4b-9e79-cae5d396da8b\" (UID: \"1e9ddf77-cd10-4c4b-9e79-cae5d396da8b\") " Jul 23 10:02:46.712376 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:46.712358 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7614-140371019592736"] Jul 23 10:02:46.714195 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:46.714168 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e9ddf77-cd10-4c4b-9e79-cae5d396da8b-kube-api-access-9zth7" (OuterVolumeSpecName: "kube-api-access-9zth7") pod "1e9ddf77-cd10-4c4b-9e79-cae5d396da8b" (UID: "1e9ddf77-cd10-4c4b-9e79-cae5d396da8b"). InnerVolumeSpecName "kube-api-access-9zth7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 10:02:46.813198 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:46.813141 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zth7\" (UniqueName: \"kubernetes.io/projected/1e9ddf77-cd10-4c4b-9e79-cae5d396da8b-kube-api-access-9zth7\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 10:02:46.844815 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:46.844789 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7614-140371019592736"] Jul 23 10:02:46.845158 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:46.845146 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e9ddf77-cd10-4c4b-9e79-cae5d396da8b" containerName="test-curl-7614-140371019592736" Jul 23 10:02:46.845200 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:46.845160 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e9ddf77-cd10-4c4b-9e79-cae5d396da8b" containerName="test-curl-7614-140371019592736" Jul 23 10:02:46.845271 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:46.845233 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e9ddf77-cd10-4c4b-9e79-cae5d396da8b" containerName="test-curl-7614-140371019592736" Jul 23 10:02:46.848280 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:46.848262 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7614-140371019592736" Jul 23 10:02:46.854512 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:46.854484 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7614-140371019592736"] Jul 23 10:02:46.913900 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:46.913861 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dnq5\" (UniqueName: \"kubernetes.io/projected/c71ed9d8-096e-4e84-9836-c494a39481aa-kube-api-access-7dnq5\") pod \"test-curl-7614-140371019592736\" (UID: \"c71ed9d8-096e-4e84-9836-c494a39481aa\") " pod="opendatahub/test-curl-7614-140371019592736" Jul 23 10:02:46.986719 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:46.986688 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e9ddf77-cd10-4c4b-9e79-cae5d396da8b" path="/var/lib/kubelet/pods/1e9ddf77-cd10-4c4b-9e79-cae5d396da8b/volumes" Jul 23 10:02:47.014830 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:47.014803 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dnq5\" (UniqueName: \"kubernetes.io/projected/c71ed9d8-096e-4e84-9836-c494a39481aa-kube-api-access-7dnq5\") pod \"test-curl-7614-140371019592736\" (UID: \"c71ed9d8-096e-4e84-9836-c494a39481aa\") " pod="opendatahub/test-curl-7614-140371019592736" Jul 23 10:02:47.023148 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:47.023121 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dnq5\" (UniqueName: \"kubernetes.io/projected/c71ed9d8-096e-4e84-9836-c494a39481aa-kube-api-access-7dnq5\") pod \"test-curl-7614-140371019592736\" (UID: \"c71ed9d8-096e-4e84-9836-c494a39481aa\") " pod="opendatahub/test-curl-7614-140371019592736" Jul 23 10:02:47.158962 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:47.158940 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7614-140371019592736" Jul 23 10:02:47.288354 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:47.288327 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7614-140371019592736"] Jul 23 10:02:47.290289 ip-10-0-135-145 kubenswrapper[2576]: W0723 10:02:47.290262 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc71ed9d8_096e_4e84_9836_c494a39481aa.slice/crio-aa5a41a4c46708842510f99c980790e11060c3ca4a7aaedbd1b4816ea15ba9cc WatchSource:0}: Error finding container aa5a41a4c46708842510f99c980790e11060c3ca4a7aaedbd1b4816ea15ba9cc: Status 404 returned error can't find the container with id aa5a41a4c46708842510f99c980790e11060c3ca4a7aaedbd1b4816ea15ba9cc Jul 23 10:02:47.506281 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:47.506193 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7614-140371019592736" Jul 23 10:02:47.506281 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:47.506208 2576 scope.go:117] "RemoveContainer" containerID="ae83117a7f7dabebd6b073fc28533ef8af3908b5dc0927915c9b1b87ba4cfd06" Jul 23 10:02:47.507533 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:47.507401 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7614-140371019592736" event={"ID":"c71ed9d8-096e-4e84-9836-c494a39481aa","Type":"ContainerStarted","Data":"aa5a41a4c46708842510f99c980790e11060c3ca4a7aaedbd1b4816ea15ba9cc"} Jul 23 10:02:48.514995 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:48.514959 2576 generic.go:356] "Generic (PLEG): container finished" podID="c71ed9d8-096e-4e84-9836-c494a39481aa" containerID="f6f9427d85b008037341f63312eab4a2502af0390ee6887727e608281a091dc4" exitCode=0 Jul 23 10:02:48.515401 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:48.515037 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7614-140371019592736" event={"ID":"c71ed9d8-096e-4e84-9836-c494a39481aa","Type":"ContainerDied","Data":"f6f9427d85b008037341f63312eab4a2502af0390ee6887727e608281a091dc4"} Jul 23 10:02:48.605972 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:48.605940 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 10:02:49.652982 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:49.652954 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7614-140371019592736" Jul 23 10:02:49.679256 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:49.679221 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7614-140371019592736_c71ed9d8-096e-4e84-9836-c494a39481aa/test-curl-7614-140371019592736/0.log" Jul 23 10:02:49.709860 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:49.709763 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7614-140371019592736"] Jul 23 10:02:49.711983 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:49.711960 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7614-140371019592736"] Jul 23 10:02:49.739597 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:49.739536 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7dnq5\" (UniqueName: \"kubernetes.io/projected/c71ed9d8-096e-4e84-9836-c494a39481aa-kube-api-access-7dnq5\") pod \"c71ed9d8-096e-4e84-9836-c494a39481aa\" (UID: \"c71ed9d8-096e-4e84-9836-c494a39481aa\") " Jul 23 10:02:49.741813 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:49.741788 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c71ed9d8-096e-4e84-9836-c494a39481aa-kube-api-access-7dnq5" (OuterVolumeSpecName: "kube-api-access-7dnq5") pod "c71ed9d8-096e-4e84-9836-c494a39481aa" (UID: "c71ed9d8-096e-4e84-9836-c494a39481aa"). InnerVolumeSpecName "kube-api-access-7dnq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 10:02:49.840599 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:49.840508 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dnq5\" (UniqueName: \"kubernetes.io/projected/c71ed9d8-096e-4e84-9836-c494a39481aa-kube-api-access-7dnq5\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 10:02:49.841726 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:49.841699 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7614-140371019593744"] Jul 23 10:02:49.842136 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:49.842118 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c71ed9d8-096e-4e84-9836-c494a39481aa" containerName="test-curl-7614-140371019592736" Jul 23 10:02:49.842207 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:49.842138 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c71ed9d8-096e-4e84-9836-c494a39481aa" containerName="test-curl-7614-140371019592736" Jul 23 10:02:49.842290 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:49.842276 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c71ed9d8-096e-4e84-9836-c494a39481aa" containerName="test-curl-7614-140371019592736" Jul 23 10:02:49.845100 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:49.845080 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7614-140371019593744" Jul 23 10:02:49.853180 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:49.853152 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7614-140371019593744"] Jul 23 10:02:49.941146 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:49.941110 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hprm\" (UniqueName: \"kubernetes.io/projected/c8bac22b-94d9-483d-86e3-598af9f0f7c3-kube-api-access-7hprm\") pod \"test-curl-7614-140371019593744\" (UID: \"c8bac22b-94d9-483d-86e3-598af9f0f7c3\") " pod="opendatahub/test-curl-7614-140371019593744" Jul 23 10:02:50.041959 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:50.041921 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hprm\" (UniqueName: \"kubernetes.io/projected/c8bac22b-94d9-483d-86e3-598af9f0f7c3-kube-api-access-7hprm\") pod \"test-curl-7614-140371019593744\" (UID: \"c8bac22b-94d9-483d-86e3-598af9f0f7c3\") " pod="opendatahub/test-curl-7614-140371019593744" Jul 23 10:02:50.049970 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:50.049935 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hprm\" (UniqueName: \"kubernetes.io/projected/c8bac22b-94d9-483d-86e3-598af9f0f7c3-kube-api-access-7hprm\") pod \"test-curl-7614-140371019593744\" (UID: \"c8bac22b-94d9-483d-86e3-598af9f0f7c3\") " pod="opendatahub/test-curl-7614-140371019593744" Jul 23 10:02:50.155837 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:50.155791 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7614-140371019593744" Jul 23 10:02:50.298651 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:50.298622 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7614-140371019593744"] Jul 23 10:02:50.300505 ip-10-0-135-145 kubenswrapper[2576]: W0723 10:02:50.300477 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc8bac22b_94d9_483d_86e3_598af9f0f7c3.slice/crio-94c634ac04956d9a39e02e8f700335cfba9a24c0487dda7d58d5d6a64a9580dd WatchSource:0}: Error finding container 94c634ac04956d9a39e02e8f700335cfba9a24c0487dda7d58d5d6a64a9580dd: Status 404 returned error can't find the container with id 94c634ac04956d9a39e02e8f700335cfba9a24c0487dda7d58d5d6a64a9580dd Jul 23 10:02:50.528251 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:50.528159 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa5a41a4c46708842510f99c980790e11060c3ca4a7aaedbd1b4816ea15ba9cc" Jul 23 10:02:50.528251 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:50.528163 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7614-140371019592736" Jul 23 10:02:50.529316 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:50.529290 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7614-140371019593744" event={"ID":"c8bac22b-94d9-483d-86e3-598af9f0f7c3","Type":"ContainerStarted","Data":"94c634ac04956d9a39e02e8f700335cfba9a24c0487dda7d58d5d6a64a9580dd"} Jul 23 10:02:50.538281 ip-10-0-135-145 kubenswrapper[2576]: E0723 10:02:50.538253 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7614-140371019592736\" is forbidden: User \"system:node:ip-10-0-135-145.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-145.ec2.internal' and this object" podUID="c71ed9d8-096e-4e84-9836-c494a39481aa" pod="opendatahub/test-curl-7614-140371019592736" Jul 23 10:02:50.881382 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:50.881345 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-efdc11/rate-test-model-efdc11-kserve-c7f66d5f-pg8jm" podUID="b33d295e-fd2b-404b-81b6-11d12c07e750" containerName="main" containerID="cri-o://0a95b81116c2c140a0c4880f80a6de2ad34c2f281eed9aade602d3e96d242f10" gracePeriod=45 Jul 23 10:02:50.985636 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:50.985602 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c71ed9d8-096e-4e84-9836-c494a39481aa" path="/var/lib/kubelet/pods/c71ed9d8-096e-4e84-9836-c494a39481aa/volumes" Jul 23 10:02:51.134895 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:51.134825 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-efdc11/rate-test-model-efdc11-kserve-c7f66d5f-pg8jm" Jul 23 10:02:51.253380 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:51.253344 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b33d295e-fd2b-404b-81b6-11d12c07e750-home\") pod \"b33d295e-fd2b-404b-81b6-11d12c07e750\" (UID: \"b33d295e-fd2b-404b-81b6-11d12c07e750\") " Jul 23 10:02:51.253380 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:51.253380 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b33d295e-fd2b-404b-81b6-11d12c07e750-tmp-dir\") pod \"b33d295e-fd2b-404b-81b6-11d12c07e750\" (UID: \"b33d295e-fd2b-404b-81b6-11d12c07e750\") " Jul 23 10:02:51.253634 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:51.253417 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6hh99\" (UniqueName: \"kubernetes.io/projected/b33d295e-fd2b-404b-81b6-11d12c07e750-kube-api-access-6hh99\") pod \"b33d295e-fd2b-404b-81b6-11d12c07e750\" (UID: \"b33d295e-fd2b-404b-81b6-11d12c07e750\") " Jul 23 10:02:51.253634 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:51.253448 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b33d295e-fd2b-404b-81b6-11d12c07e750-dshm\") pod \"b33d295e-fd2b-404b-81b6-11d12c07e750\" (UID: \"b33d295e-fd2b-404b-81b6-11d12c07e750\") " Jul 23 10:02:51.253634 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:51.253474 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b33d295e-fd2b-404b-81b6-11d12c07e750-tls-certs\") pod \"b33d295e-fd2b-404b-81b6-11d12c07e750\" (UID: \"b33d295e-fd2b-404b-81b6-11d12c07e750\") " Jul 23 10:02:51.253634 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:51.253522 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b33d295e-fd2b-404b-81b6-11d12c07e750-model-cache\") pod \"b33d295e-fd2b-404b-81b6-11d12c07e750\" (UID: \"b33d295e-fd2b-404b-81b6-11d12c07e750\") " Jul 23 10:02:51.253780 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:51.253704 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b33d295e-fd2b-404b-81b6-11d12c07e750-home" (OuterVolumeSpecName: "home") pod "b33d295e-fd2b-404b-81b6-11d12c07e750" (UID: "b33d295e-fd2b-404b-81b6-11d12c07e750"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 10:02:51.253780 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:51.253720 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b33d295e-fd2b-404b-81b6-11d12c07e750-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b33d295e-fd2b-404b-81b6-11d12c07e750" (UID: "b33d295e-fd2b-404b-81b6-11d12c07e750"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 10:02:51.253898 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:51.253869 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b33d295e-fd2b-404b-81b6-11d12c07e750-model-cache" (OuterVolumeSpecName: "model-cache") pod "b33d295e-fd2b-404b-81b6-11d12c07e750" (UID: "b33d295e-fd2b-404b-81b6-11d12c07e750"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 10:02:51.255707 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:51.255681 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b33d295e-fd2b-404b-81b6-11d12c07e750-dshm" (OuterVolumeSpecName: "dshm") pod "b33d295e-fd2b-404b-81b6-11d12c07e750" (UID: "b33d295e-fd2b-404b-81b6-11d12c07e750"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 10:02:51.255792 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:51.255725 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b33d295e-fd2b-404b-81b6-11d12c07e750-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b33d295e-fd2b-404b-81b6-11d12c07e750" (UID: "b33d295e-fd2b-404b-81b6-11d12c07e750"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 10:02:51.255792 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:51.255741 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b33d295e-fd2b-404b-81b6-11d12c07e750-kube-api-access-6hh99" (OuterVolumeSpecName: "kube-api-access-6hh99") pod "b33d295e-fd2b-404b-81b6-11d12c07e750" (UID: "b33d295e-fd2b-404b-81b6-11d12c07e750"). InnerVolumeSpecName "kube-api-access-6hh99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 10:02:51.354871 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:51.354821 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b33d295e-fd2b-404b-81b6-11d12c07e750-home\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 10:02:51.354871 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:51.354869 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b33d295e-fd2b-404b-81b6-11d12c07e750-tmp-dir\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 10:02:51.354871 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:51.354883 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hh99\" (UniqueName: \"kubernetes.io/projected/b33d295e-fd2b-404b-81b6-11d12c07e750-kube-api-access-6hh99\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 10:02:51.355131 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:51.354896 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b33d295e-fd2b-404b-81b6-11d12c07e750-dshm\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 10:02:51.355131 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:51.354908 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b33d295e-fd2b-404b-81b6-11d12c07e750-tls-certs\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 10:02:51.355131 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:51.354919 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b33d295e-fd2b-404b-81b6-11d12c07e750-model-cache\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 10:02:51.536497 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:51.536406 2576 generic.go:356] "Generic (PLEG): container finished" podID="c8bac22b-94d9-483d-86e3-598af9f0f7c3" containerID="464c5a058b0a875a5c3bfd094251b739911d4621e3793611d8b6de40dd9b4ad7" exitCode=0 Jul 23 10:02:51.536656 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:51.536494 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7614-140371019593744" event={"ID":"c8bac22b-94d9-483d-86e3-598af9f0f7c3","Type":"ContainerDied","Data":"464c5a058b0a875a5c3bfd094251b739911d4621e3793611d8b6de40dd9b4ad7"} Jul 23 10:02:51.537719 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:51.537688 2576 generic.go:356] "Generic (PLEG): container finished" podID="b33d295e-fd2b-404b-81b6-11d12c07e750" containerID="0a95b81116c2c140a0c4880f80a6de2ad34c2f281eed9aade602d3e96d242f10" exitCode=0 Jul 23 10:02:51.537850 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:51.537734 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-efdc11/rate-test-model-efdc11-kserve-c7f66d5f-pg8jm" Jul 23 10:02:51.537850 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:51.537750 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-efdc11/rate-test-model-efdc11-kserve-c7f66d5f-pg8jm" event={"ID":"b33d295e-fd2b-404b-81b6-11d12c07e750","Type":"ContainerDied","Data":"0a95b81116c2c140a0c4880f80a6de2ad34c2f281eed9aade602d3e96d242f10"} Jul 23 10:02:51.537850 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:51.537782 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-efdc11/rate-test-model-efdc11-kserve-c7f66d5f-pg8jm" event={"ID":"b33d295e-fd2b-404b-81b6-11d12c07e750","Type":"ContainerDied","Data":"91827fdaba4afc36c9c170662ea81f0b4c60fa77252be6954374081ac34c99b4"} Jul 23 10:02:51.537850 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:51.537802 2576 scope.go:117] "RemoveContainer" containerID="0a95b81116c2c140a0c4880f80a6de2ad34c2f281eed9aade602d3e96d242f10" Jul 23 10:02:51.546869 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:51.546856 2576 scope.go:117] "RemoveContainer" containerID="0a95b81116c2c140a0c4880f80a6de2ad34c2f281eed9aade602d3e96d242f10" Jul 23 10:02:51.547105 ip-10-0-135-145 kubenswrapper[2576]: E0723 10:02:51.547086 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a95b81116c2c140a0c4880f80a6de2ad34c2f281eed9aade602d3e96d242f10\": container with ID starting with 0a95b81116c2c140a0c4880f80a6de2ad34c2f281eed9aade602d3e96d242f10 not found: ID does not exist" containerID="0a95b81116c2c140a0c4880f80a6de2ad34c2f281eed9aade602d3e96d242f10" Jul 23 10:02:51.547143 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:51.547113 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a95b81116c2c140a0c4880f80a6de2ad34c2f281eed9aade602d3e96d242f10"} err="failed to get container status \"0a95b81116c2c140a0c4880f80a6de2ad34c2f281eed9aade602d3e96d242f10\": rpc error: code = NotFound desc = could not find container \"0a95b81116c2c140a0c4880f80a6de2ad34c2f281eed9aade602d3e96d242f10\": container with ID starting with 0a95b81116c2c140a0c4880f80a6de2ad34c2f281eed9aade602d3e96d242f10 not found: ID does not exist" Jul 23 10:02:51.564599 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:51.564574 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-efdc11/rate-test-model-efdc11-kserve-c7f66d5f-pg8jm"] Jul 23 10:02:51.570317 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:51.570294 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-efdc11/rate-test-model-efdc11-kserve-c7f66d5f-pg8jm"] Jul 23 10:02:51.709125 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:51.709081 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-0dc985/rate-test-model-0dc985-kserve-bd7857d7f-j6z5s" podUID="7ed575ba-c014-4973-b971-443a3b5ef256" containerName="main" containerID="cri-o://a42c2d7433bdabe8a1724ba70d83d9d299398cef9a6cf8b6ff9a21a0ac202bb0" gracePeriod=45 Jul 23 10:02:51.957445 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:51.957419 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0dc985/rate-test-model-0dc985-kserve-bd7857d7f-j6z5s" Jul 23 10:02:52.061820 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:52.061785 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ed575ba-c014-4973-b971-443a3b5ef256-tls-certs\") pod \"7ed575ba-c014-4973-b971-443a3b5ef256\" (UID: \"7ed575ba-c014-4973-b971-443a3b5ef256\") " Jul 23 10:02:52.061820 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:52.061828 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ed575ba-c014-4973-b971-443a3b5ef256-model-cache\") pod \"7ed575ba-c014-4973-b971-443a3b5ef256\" (UID: \"7ed575ba-c014-4973-b971-443a3b5ef256\") " Jul 23 10:02:52.062086 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:52.061856 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ed575ba-c014-4973-b971-443a3b5ef256-tmp-dir\") pod \"7ed575ba-c014-4973-b971-443a3b5ef256\" (UID: \"7ed575ba-c014-4973-b971-443a3b5ef256\") " Jul 23 10:02:52.062086 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:52.061895 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ed575ba-c014-4973-b971-443a3b5ef256-dshm\") pod \"7ed575ba-c014-4973-b971-443a3b5ef256\" (UID: \"7ed575ba-c014-4973-b971-443a3b5ef256\") " Jul 23 10:02:52.062086 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:52.061952 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ed575ba-c014-4973-b971-443a3b5ef256-home\") pod \"7ed575ba-c014-4973-b971-443a3b5ef256\" (UID: \"7ed575ba-c014-4973-b971-443a3b5ef256\") " Jul 23 10:02:52.062086 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:52.062048 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ww26q\" (UniqueName: \"kubernetes.io/projected/7ed575ba-c014-4973-b971-443a3b5ef256-kube-api-access-ww26q\") pod \"7ed575ba-c014-4973-b971-443a3b5ef256\" (UID: \"7ed575ba-c014-4973-b971-443a3b5ef256\") " Jul 23 10:02:52.062411 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:52.062156 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ed575ba-c014-4973-b971-443a3b5ef256-model-cache" (OuterVolumeSpecName: "model-cache") pod "7ed575ba-c014-4973-b971-443a3b5ef256" (UID: "7ed575ba-c014-4973-b971-443a3b5ef256"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 10:02:52.062411 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:52.062214 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ed575ba-c014-4973-b971-443a3b5ef256-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7ed575ba-c014-4973-b971-443a3b5ef256" (UID: "7ed575ba-c014-4973-b971-443a3b5ef256"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 10:02:52.062411 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:52.062323 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ed575ba-c014-4973-b971-443a3b5ef256-home" (OuterVolumeSpecName: "home") pod "7ed575ba-c014-4973-b971-443a3b5ef256" (UID: "7ed575ba-c014-4973-b971-443a3b5ef256"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 10:02:52.062543 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:52.062425 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ed575ba-c014-4973-b971-443a3b5ef256-model-cache\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 10:02:52.062543 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:52.062443 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ed575ba-c014-4973-b971-443a3b5ef256-tmp-dir\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 10:02:52.062543 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:52.062454 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ed575ba-c014-4973-b971-443a3b5ef256-home\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 10:02:52.064157 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:52.064130 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ed575ba-c014-4973-b971-443a3b5ef256-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7ed575ba-c014-4973-b971-443a3b5ef256" (UID: "7ed575ba-c014-4973-b971-443a3b5ef256"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 10:02:52.064511 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:52.064492 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ed575ba-c014-4973-b971-443a3b5ef256-dshm" (OuterVolumeSpecName: "dshm") pod "7ed575ba-c014-4973-b971-443a3b5ef256" (UID: "7ed575ba-c014-4973-b971-443a3b5ef256"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 10:02:52.064573 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:52.064511 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ed575ba-c014-4973-b971-443a3b5ef256-kube-api-access-ww26q" (OuterVolumeSpecName: "kube-api-access-ww26q") pod "7ed575ba-c014-4973-b971-443a3b5ef256" (UID: "7ed575ba-c014-4973-b971-443a3b5ef256"). InnerVolumeSpecName "kube-api-access-ww26q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 10:02:52.163090 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:52.163052 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ed575ba-c014-4973-b971-443a3b5ef256-dshm\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 10:02:52.163090 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:52.163085 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ww26q\" (UniqueName: \"kubernetes.io/projected/7ed575ba-c014-4973-b971-443a3b5ef256-kube-api-access-ww26q\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 10:02:52.163090 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:52.163097 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ed575ba-c014-4973-b971-443a3b5ef256-tls-certs\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 10:02:52.291045 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:52.291002 2576 scope.go:117] "RemoveContainer" containerID="31c6051180864f4fb268ef5279f7cbde35de5faba3d5bb55e43736b0760bf46b" Jul 23 10:02:52.544487 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:52.544394 2576 generic.go:356] "Generic (PLEG): container finished" podID="7ed575ba-c014-4973-b971-443a3b5ef256" containerID="a42c2d7433bdabe8a1724ba70d83d9d299398cef9a6cf8b6ff9a21a0ac202bb0" exitCode=0 Jul 23 10:02:52.544487 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:52.544435 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0dc985/rate-test-model-0dc985-kserve-bd7857d7f-j6z5s" Jul 23 10:02:52.544487 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:52.544481 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0dc985/rate-test-model-0dc985-kserve-bd7857d7f-j6z5s" event={"ID":"7ed575ba-c014-4973-b971-443a3b5ef256","Type":"ContainerDied","Data":"a42c2d7433bdabe8a1724ba70d83d9d299398cef9a6cf8b6ff9a21a0ac202bb0"} Jul 23 10:02:52.544767 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:52.544520 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0dc985/rate-test-model-0dc985-kserve-bd7857d7f-j6z5s" event={"ID":"7ed575ba-c014-4973-b971-443a3b5ef256","Type":"ContainerDied","Data":"ebc0df65d092e794e0753399b0291cb7e71edca39b9eaa357297cc341b98a939"} Jul 23 10:02:52.544767 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:52.544542 2576 scope.go:117] "RemoveContainer" containerID="a42c2d7433bdabe8a1724ba70d83d9d299398cef9a6cf8b6ff9a21a0ac202bb0" Jul 23 10:02:52.554008 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:52.553983 2576 scope.go:117] "RemoveContainer" containerID="a42c2d7433bdabe8a1724ba70d83d9d299398cef9a6cf8b6ff9a21a0ac202bb0" Jul 23 10:02:52.554273 ip-10-0-135-145 kubenswrapper[2576]: E0723 10:02:52.554233 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a42c2d7433bdabe8a1724ba70d83d9d299398cef9a6cf8b6ff9a21a0ac202bb0\": container with ID starting with a42c2d7433bdabe8a1724ba70d83d9d299398cef9a6cf8b6ff9a21a0ac202bb0 not found: ID does not exist" containerID="a42c2d7433bdabe8a1724ba70d83d9d299398cef9a6cf8b6ff9a21a0ac202bb0" Jul 23 10:02:52.554339 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:52.554283 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a42c2d7433bdabe8a1724ba70d83d9d299398cef9a6cf8b6ff9a21a0ac202bb0"} err="failed to get container status \"a42c2d7433bdabe8a1724ba70d83d9d299398cef9a6cf8b6ff9a21a0ac202bb0\": rpc error: code = NotFound desc = could not find container \"a42c2d7433bdabe8a1724ba70d83d9d299398cef9a6cf8b6ff9a21a0ac202bb0\": container with ID starting with a42c2d7433bdabe8a1724ba70d83d9d299398cef9a6cf8b6ff9a21a0ac202bb0 not found: ID does not exist" Jul 23 10:02:52.566282 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:52.566252 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-0dc985/rate-test-model-0dc985-kserve-bd7857d7f-j6z5s"] Jul 23 10:02:52.572304 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:52.572278 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-0dc985/rate-test-model-0dc985-kserve-bd7857d7f-j6z5s"] Jul 23 10:02:52.678403 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:52.678372 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7614-140371019593744" Jul 23 10:02:52.699057 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:52.699027 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7614-140371019593744_c8bac22b-94d9-483d-86e3-598af9f0f7c3/test-curl-7614-140371019593744/0.log" Jul 23 10:02:52.723516 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:52.723487 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7614-140371019593744"] Jul 23 10:02:52.725535 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:52.725512 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7614-140371019593744"] Jul 23 10:02:52.767436 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:52.767403 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7hprm\" (UniqueName: \"kubernetes.io/projected/c8bac22b-94d9-483d-86e3-598af9f0f7c3-kube-api-access-7hprm\") pod \"c8bac22b-94d9-483d-86e3-598af9f0f7c3\" (UID: \"c8bac22b-94d9-483d-86e3-598af9f0f7c3\") " Jul 23 10:02:52.769565 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:52.769544 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8bac22b-94d9-483d-86e3-598af9f0f7c3-kube-api-access-7hprm" (OuterVolumeSpecName: "kube-api-access-7hprm") pod "c8bac22b-94d9-483d-86e3-598af9f0f7c3" (UID: "c8bac22b-94d9-483d-86e3-598af9f0f7c3"). InnerVolumeSpecName "kube-api-access-7hprm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 10:02:52.852667 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:52.852586 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7614-140371019593744"] Jul 23 10:02:52.852992 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:52.852979 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b33d295e-fd2b-404b-81b6-11d12c07e750" containerName="main" Jul 23 10:02:52.853044 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:52.852993 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b33d295e-fd2b-404b-81b6-11d12c07e750" containerName="main" Jul 23 10:02:52.853044 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:52.853003 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ed575ba-c014-4973-b971-443a3b5ef256" containerName="main" Jul 23 10:02:52.853044 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:52.853009 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ed575ba-c014-4973-b971-443a3b5ef256" containerName="main" Jul 23 10:02:52.853044 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:52.853016 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8bac22b-94d9-483d-86e3-598af9f0f7c3" containerName="test-curl-7614-140371019593744" Jul 23 10:02:52.853044 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:52.853023 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8bac22b-94d9-483d-86e3-598af9f0f7c3" containerName="test-curl-7614-140371019593744" Jul 23 10:02:52.853198 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:52.853073 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b33d295e-fd2b-404b-81b6-11d12c07e750" containerName="main" Jul 23 10:02:52.853198 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:52.853081 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8bac22b-94d9-483d-86e3-598af9f0f7c3" containerName="test-curl-7614-140371019593744" Jul 23 10:02:52.853198 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:52.853090 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ed575ba-c014-4973-b971-443a3b5ef256" containerName="main" Jul 23 10:02:52.856188 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:52.856171 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7614-140371019593744" Jul 23 10:02:52.868263 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:52.868213 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hprm\" (UniqueName: \"kubernetes.io/projected/c8bac22b-94d9-483d-86e3-598af9f0f7c3-kube-api-access-7hprm\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 10:02:52.868574 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:52.868548 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7614-140371019593744"] Jul 23 10:02:52.969293 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:52.969228 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfm47\" (UniqueName: \"kubernetes.io/projected/43ad2209-b338-47ba-bbe6-98edd5c8f762-kube-api-access-qfm47\") pod \"test-curl-7614-140371019593744\" (UID: \"43ad2209-b338-47ba-bbe6-98edd5c8f762\") " pod="opendatahub/test-curl-7614-140371019593744" Jul 23 10:02:52.986910 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:52.986877 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ed575ba-c014-4973-b971-443a3b5ef256" path="/var/lib/kubelet/pods/7ed575ba-c014-4973-b971-443a3b5ef256/volumes" Jul 23 10:02:52.987350 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:52.987335 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b33d295e-fd2b-404b-81b6-11d12c07e750" path="/var/lib/kubelet/pods/b33d295e-fd2b-404b-81b6-11d12c07e750/volumes" Jul 23 10:02:52.987689 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:52.987676 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8bac22b-94d9-483d-86e3-598af9f0f7c3" path="/var/lib/kubelet/pods/c8bac22b-94d9-483d-86e3-598af9f0f7c3/volumes" Jul 23 10:02:53.070064 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:53.070023 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfm47\" (UniqueName: \"kubernetes.io/projected/43ad2209-b338-47ba-bbe6-98edd5c8f762-kube-api-access-qfm47\") pod \"test-curl-7614-140371019593744\" (UID: \"43ad2209-b338-47ba-bbe6-98edd5c8f762\") " pod="opendatahub/test-curl-7614-140371019593744" Jul 23 10:02:53.078741 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:53.078710 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfm47\" (UniqueName: \"kubernetes.io/projected/43ad2209-b338-47ba-bbe6-98edd5c8f762-kube-api-access-qfm47\") pod \"test-curl-7614-140371019593744\" (UID: \"43ad2209-b338-47ba-bbe6-98edd5c8f762\") " pod="opendatahub/test-curl-7614-140371019593744" Jul 23 10:02:53.165002 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:53.164962 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7614-140371019593744" Jul 23 10:02:53.298069 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:53.298043 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7614-140371019593744"] Jul 23 10:02:53.299671 ip-10-0-135-145 kubenswrapper[2576]: W0723 10:02:53.299645 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod43ad2209_b338_47ba_bbe6_98edd5c8f762.slice/crio-5d1fc11e6890021049cab075fd189eb9a53f7c01e31878fd9fe7681261829e84 WatchSource:0}: Error finding container 5d1fc11e6890021049cab075fd189eb9a53f7c01e31878fd9fe7681261829e84: Status 404 returned error can't find the container with id 5d1fc11e6890021049cab075fd189eb9a53f7c01e31878fd9fe7681261829e84 Jul 23 10:02:53.551094 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:53.551016 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7614-140371019593744" event={"ID":"43ad2209-b338-47ba-bbe6-98edd5c8f762","Type":"ContainerStarted","Data":"5d1fc11e6890021049cab075fd189eb9a53f7c01e31878fd9fe7681261829e84"} Jul 23 10:02:53.552957 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:53.552939 2576 scope.go:117] "RemoveContainer" containerID="464c5a058b0a875a5c3bfd094251b739911d4621e3793611d8b6de40dd9b4ad7" Jul 23 10:02:53.553074 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:53.552967 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7614-140371019593744" Jul 23 10:02:54.561264 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:54.561202 2576 generic.go:356] "Generic (PLEG): container finished" podID="43ad2209-b338-47ba-bbe6-98edd5c8f762" containerID="b7dff9515abc0f81dacec7152601aa94454eb2297c0bb711213c1597d5583199" exitCode=0 Jul 23 10:02:54.561660 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:54.561285 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7614-140371019593744" event={"ID":"43ad2209-b338-47ba-bbe6-98edd5c8f762","Type":"ContainerDied","Data":"b7dff9515abc0f81dacec7152601aa94454eb2297c0bb711213c1597d5583199"} Jul 23 10:02:55.699706 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:55.699676 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7614-140371019593744" Jul 23 10:02:55.725397 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:55.725364 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7614-140371019593744_43ad2209-b338-47ba-bbe6-98edd5c8f762/test-curl-7614-140371019593744/0.log" Jul 23 10:02:55.750829 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:55.750798 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7614-140371019593744"] Jul 23 10:02:55.754383 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:55.754356 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7614-140371019593744"] Jul 23 10:02:55.797667 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:55.797636 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qfm47\" (UniqueName: \"kubernetes.io/projected/43ad2209-b338-47ba-bbe6-98edd5c8f762-kube-api-access-qfm47\") pod \"43ad2209-b338-47ba-bbe6-98edd5c8f762\" (UID: \"43ad2209-b338-47ba-bbe6-98edd5c8f762\") " Jul 23 10:02:55.799772 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:55.799736 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43ad2209-b338-47ba-bbe6-98edd5c8f762-kube-api-access-qfm47" (OuterVolumeSpecName: "kube-api-access-qfm47") pod "43ad2209-b338-47ba-bbe6-98edd5c8f762" (UID: "43ad2209-b338-47ba-bbe6-98edd5c8f762"). InnerVolumeSpecName "kube-api-access-qfm47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 10:02:55.899209 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:55.899163 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfm47\" (UniqueName: \"kubernetes.io/projected/43ad2209-b338-47ba-bbe6-98edd5c8f762-kube-api-access-qfm47\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 10:02:55.948268 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:55.948220 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-efdc11-5cc8l"] Jul 23 10:02:55.948613 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:55.948596 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43ad2209-b338-47ba-bbe6-98edd5c8f762" containerName="test-curl-7614-140371019593744" Jul 23 10:02:55.948655 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:55.948617 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="43ad2209-b338-47ba-bbe6-98edd5c8f762" containerName="test-curl-7614-140371019593744" Jul 23 10:02:55.948703 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:55.948690 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="43ad2209-b338-47ba-bbe6-98edd5c8f762" containerName="test-curl-7614-140371019593744" Jul 23 10:02:55.951763 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:55.951744 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-efdc11-5cc8l" Jul 23 10:02:55.954994 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:55.954971 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 10:02:55.955210 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:55.955196 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n4hmw\"" Jul 23 10:02:55.962169 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:55.962141 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-efdc11-5cc8l"] Jul 23 10:02:55.999759 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:55.999723 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4249fad-eadd-469f-a1ec-c4e36de4633c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-efdc11-5cc8l\" (UID: \"d4249fad-eadd-469f-a1ec-c4e36de4633c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-efdc11-5cc8l" Jul 23 10:02:56.100655 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:56.100616 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4249fad-eadd-469f-a1ec-c4e36de4633c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-efdc11-5cc8l\" (UID: \"d4249fad-eadd-469f-a1ec-c4e36de4633c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-efdc11-5cc8l" Jul 23 10:02:56.101287 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:56.101231 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4249fad-eadd-469f-a1ec-c4e36de4633c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-efdc11-5cc8l\" (UID: \"d4249fad-eadd-469f-a1ec-c4e36de4633c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-efdc11-5cc8l" Jul 23 10:02:56.263513 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:56.263420 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-efdc11-5cc8l" Jul 23 10:02:56.407022 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:56.406987 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-efdc11-5cc8l"] Jul 23 10:02:56.410450 ip-10-0-135-145 kubenswrapper[2576]: W0723 10:02:56.410425 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4249fad_eadd_469f_a1ec_c4e36de4633c.slice/crio-59a84edbf054ab0f7eff4ffc707d8c5f3dafeb46ff41e78d8de131b8852ab69a WatchSource:0}: Error finding container 59a84edbf054ab0f7eff4ffc707d8c5f3dafeb46ff41e78d8de131b8852ab69a: Status 404 returned error can't find the container with id 59a84edbf054ab0f7eff4ffc707d8c5f3dafeb46ff41e78d8de131b8852ab69a Jul 23 10:02:56.575299 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:56.575196 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7614-140371019593744" Jul 23 10:02:56.575299 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:56.575234 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d1fc11e6890021049cab075fd189eb9a53f7c01e31878fd9fe7681261829e84" Jul 23 10:02:56.576579 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:56.576544 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-efdc11-5cc8l" event={"ID":"d4249fad-eadd-469f-a1ec-c4e36de4633c","Type":"ContainerStarted","Data":"59a84edbf054ab0f7eff4ffc707d8c5f3dafeb46ff41e78d8de131b8852ab69a"} Jul 23 10:02:56.585083 ip-10-0-135-145 kubenswrapper[2576]: E0723 10:02:56.585051 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7614-140371019593744\" is forbidden: User \"system:node:ip-10-0-135-145.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-145.ec2.internal' and this object" podUID="43ad2209-b338-47ba-bbe6-98edd5c8f762" pod="opendatahub/test-curl-7614-140371019593744" Jul 23 10:02:56.662566 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:56.662520 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 10:02:56.759490 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:56.759456 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 10:02:56.986255 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:56.986201 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43ad2209-b338-47ba-bbe6-98edd5c8f762" path="/var/lib/kubelet/pods/43ad2209-b338-47ba-bbe6-98edd5c8f762/volumes" Jul 23 10:02:57.584075 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:57.584038 2576 generic.go:356] "Generic (PLEG): container finished" podID="d4249fad-eadd-469f-a1ec-c4e36de4633c" containerID="255de1dbdfb967649843f91689c3b12a06e031e0a2464fbbd8e6e6d506624f40" exitCode=0 Jul 23 10:02:57.584286 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:57.584111 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-efdc11-5cc8l" event={"ID":"d4249fad-eadd-469f-a1ec-c4e36de4633c","Type":"ContainerDied","Data":"255de1dbdfb967649843f91689c3b12a06e031e0a2464fbbd8e6e6d506624f40"} Jul 23 10:02:58.721565 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:58.721538 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-efdc11-5cc8l" Jul 23 10:02:58.825222 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:58.825186 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4249fad-eadd-469f-a1ec-c4e36de4633c-maas-api-service-ca\") pod \"d4249fad-eadd-469f-a1ec-c4e36de4633c\" (UID: \"d4249fad-eadd-469f-a1ec-c4e36de4633c\") " Jul 23 10:02:58.825575 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:58.825551 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4249fad-eadd-469f-a1ec-c4e36de4633c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d4249fad-eadd-469f-a1ec-c4e36de4633c" (UID: "d4249fad-eadd-469f-a1ec-c4e36de4633c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 10:02:58.926652 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:58.926611 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4249fad-eadd-469f-a1ec-c4e36de4633c-maas-api-service-ca\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 10:02:59.597311 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:59.597283 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-efdc11-5cc8l" Jul 23 10:02:59.597464 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:59.597278 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-efdc11-5cc8l" event={"ID":"d4249fad-eadd-469f-a1ec-c4e36de4633c","Type":"ContainerDied","Data":"59a84edbf054ab0f7eff4ffc707d8c5f3dafeb46ff41e78d8de131b8852ab69a"} Jul 23 10:02:59.597464 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:02:59.597435 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59a84edbf054ab0f7eff4ffc707d8c5f3dafeb46ff41e78d8de131b8852ab69a" Jul 23 10:03:04.942450 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:03:04.942415 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 10:03:06.429976 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:03:06.429940 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 10:03:09.598685 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:03:09.598644 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 10:03:11.056222 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:03:11.056178 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 10:03:11.074647 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:03:11.074607 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-efdc11-5cc8l"] Jul 23 10:03:11.078552 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:03:11.078525 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-efdc11-5cc8l"] Jul 23 10:03:11.493795 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:03:11.493760 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 10:03:12.985253 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:03:12.985206 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4249fad-eadd-469f-a1ec-c4e36de4633c" path="/var/lib/kubelet/pods/d4249fad-eadd-469f-a1ec-c4e36de4633c/volumes" Jul 23 10:03:17.534659 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:03:17.534578 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 10:03:17.606505 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:03:17.606471 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 10:03:17.842671 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:03:17.842574 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 10:03:18.056318 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:03:18.056287 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 10:03:18.178788 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:03:18.178753 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-0dc985-22knm"] Jul 23 10:03:18.179124 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:03:18.179112 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4249fad-eadd-469f-a1ec-c4e36de4633c" containerName="revoke-keys" Jul 23 10:03:18.179169 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:03:18.179125 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4249fad-eadd-469f-a1ec-c4e36de4633c" containerName="revoke-keys" Jul 23 10:03:18.179203 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:03:18.179192 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4249fad-eadd-469f-a1ec-c4e36de4633c" containerName="revoke-keys" Jul 23 10:03:18.184418 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:03:18.184400 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-0dc985-22knm" Jul 23 10:03:18.186748 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:03:18.186722 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 10:03:18.186865 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:03:18.186759 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n4hmw\"" Jul 23 10:03:18.191104 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:03:18.191076 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-0dc985-22knm"] Jul 23 10:03:18.311173 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:03:18.311127 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63616cb6-727c-4bed-bd50-018a0e615e2e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-0dc985-22knm\" (UID: \"63616cb6-727c-4bed-bd50-018a0e615e2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-0dc985-22knm" Jul 23 10:03:18.411836 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:03:18.411797 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63616cb6-727c-4bed-bd50-018a0e615e2e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-0dc985-22knm\" (UID: \"63616cb6-727c-4bed-bd50-018a0e615e2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-0dc985-22knm" Jul 23 10:03:18.412532 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:03:18.412505 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63616cb6-727c-4bed-bd50-018a0e615e2e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-0dc985-22knm\" (UID: \"63616cb6-727c-4bed-bd50-018a0e615e2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-0dc985-22knm" Jul 23 10:03:18.496348 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:03:18.496233 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-0dc985-22knm" Jul 23 10:03:18.643845 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:03:18.637790 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-0dc985-22knm"] Jul 23 10:03:18.702627 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:03:18.702588 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-0dc985-22knm" event={"ID":"63616cb6-727c-4bed-bd50-018a0e615e2e","Type":"ContainerStarted","Data":"ccbc9c6be929fff9f3541c74c90edf6579693b553195d3d8be88a71f35f94147"} Jul 23 10:03:18.913882 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:03:18.913837 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 10:03:19.009221 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:03:19.009185 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 10:03:19.709438 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:03:19.709398 2576 generic.go:356] "Generic (PLEG): container finished" podID="63616cb6-727c-4bed-bd50-018a0e615e2e" containerID="86c154eaf05aa9e32dcb8b06e360cb81ada065935fcc3949012af2a6eba3b4bc" exitCode=0 Jul 23 10:03:19.709895 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:03:19.709477 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-0dc985-22knm" event={"ID":"63616cb6-727c-4bed-bd50-018a0e615e2e","Type":"ContainerDied","Data":"86c154eaf05aa9e32dcb8b06e360cb81ada065935fcc3949012af2a6eba3b4bc"} Jul 23 10:03:20.844898 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:03:20.844871 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-0dc985-22knm" Jul 23 10:03:20.932929 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:03:20.932901 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63616cb6-727c-4bed-bd50-018a0e615e2e-maas-api-service-ca\") pod \"63616cb6-727c-4bed-bd50-018a0e615e2e\" (UID: \"63616cb6-727c-4bed-bd50-018a0e615e2e\") " Jul 23 10:03:20.933337 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:03:20.933310 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63616cb6-727c-4bed-bd50-018a0e615e2e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "63616cb6-727c-4bed-bd50-018a0e615e2e" (UID: "63616cb6-727c-4bed-bd50-018a0e615e2e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 10:03:21.034379 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:03:21.034334 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63616cb6-727c-4bed-bd50-018a0e615e2e-maas-api-service-ca\") on node \"ip-10-0-135-145.ec2.internal\" DevicePath \"\"" Jul 23 10:03:21.722047 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:03:21.722017 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-0dc985-22knm" Jul 23 10:03:21.722047 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:03:21.722040 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-0dc985-22knm" event={"ID":"63616cb6-727c-4bed-bd50-018a0e615e2e","Type":"ContainerDied","Data":"ccbc9c6be929fff9f3541c74c90edf6579693b553195d3d8be88a71f35f94147"} Jul 23 10:03:21.722283 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:03:21.722081 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ccbc9c6be929fff9f3541c74c90edf6579693b553195d3d8be88a71f35f94147" Jul 23 10:03:28.788595 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:03:28.788554 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 10:03:31.591051 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:03:31.591006 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5mq5"] Jul 23 10:03:33.270044 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:03:33.270001 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 10:03:33.276411 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:03:33.276379 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-0dc985-22knm"] Jul 23 10:03:33.278674 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:03:33.278652 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-0dc985-22knm"] Jul 23 10:03:33.677786 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:03:33.677742 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 10:03:34.985402 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:03:34.985372 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63616cb6-727c-4bed-bd50-018a0e615e2e" path="/var/lib/kubelet/pods/63616cb6-727c-4bed-bd50-018a0e615e2e/volumes" Jul 23 10:03:39.690601 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:03:39.690547 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 10:03:39.790857 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:03:39.790818 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 10:03:39.966408 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:03:39.966303 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 10:03:40.201697 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:03:40.201660 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 10:03:40.646837 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:03:40.646790 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5876bc94d4-gs5hp_c662fa1f-8cb7-434e-990a-31cdea707782/authorino/0.log" Jul 23 10:03:47.094412 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:03:47.094381 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-64d6699bc9-xvzrs_01d0eb90-24fa-47a5-8019-bf35aaba7e1d/copy-manifests/0.log" Jul 23 10:03:47.101870 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:03:47.101841 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-64d6699bc9-xvzrs_01d0eb90-24fa-47a5-8019-bf35aaba7e1d/manager/0.log" Jul 23 10:03:48.795827 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:03:48.795782 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595h986_51e149f3-3d54-46ad-86c9-01c83eb781f7/util/0.log" Jul 23 10:03:48.804601 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:03:48.804575 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595h986_51e149f3-3d54-46ad-86c9-01c83eb781f7/pull/0.log" Jul 23 10:03:48.810872 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:03:48.810849 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595h986_51e149f3-3d54-46ad-86c9-01c83eb781f7/extract/0.log" Jul 23 10:03:48.926364 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:03:48.926332 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wrs5q_f4dae8b7-1352-4f26-93c6-c5ffd5f8369f/util/0.log" Jul 23 10:03:48.932816 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:03:48.932786 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wrs5q_f4dae8b7-1352-4f26-93c6-c5ffd5f8369f/pull/0.log" Jul 23 10:03:48.939048 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:03:48.939026 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wrs5q_f4dae8b7-1352-4f26-93c6-c5ffd5f8369f/extract/0.log" Jul 23 10:03:49.059227 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:03:49.059142 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qrfl7_ce7494cc-693b-4499-bb8a-6646ec33f004/util/0.log" Jul 23 10:03:49.065744 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:03:49.065718 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qrfl7_ce7494cc-693b-4499-bb8a-6646ec33f004/pull/0.log" Jul 23 10:03:49.071627 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:03:49.071603 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qrfl7_ce7494cc-693b-4499-bb8a-6646ec33f004/extract/0.log" Jul 23 10:03:49.185698 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:03:49.185665 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cz5gk_c13198e7-f244-465c-82ff-199bde275f68/util/0.log" Jul 23 10:03:49.192525 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:03:49.192497 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cz5gk_c13198e7-f244-465c-82ff-199bde275f68/pull/0.log" Jul 23 10:03:49.198815 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:03:49.198794 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cz5gk_c13198e7-f244-465c-82ff-199bde275f68/extract/0.log" Jul 23 10:03:49.328069 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:03:49.327974 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5876bc94d4-gs5hp_c662fa1f-8cb7-434e-990a-31cdea707782/authorino/0.log" Jul 23 10:03:50.075700 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:03:50.075671 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-q5mq5_07acbf04-6129-4a76-ac35-87e8e8f6c19c/limitador/0.log" Jul 23 10:03:50.793746 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:03:50.793711 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5dccdf9c4-2fgq7_dc7f1999-2e2f-41be-ae43-298dbd611e14/kube-auth-proxy/0.log" Jul 23 10:03:51.432418 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:03:51.432385 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7494768586-7px6v_6bddaeac-9272-4031-b216-a31fbe58c89d/router/0.log" Jul 23 10:03:52.415418 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:03:52.415384 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-6ccj9_03bcb88b-3597-43a5-a063-869d306aebca/main/0.log" Jul 23 10:03:58.180943 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:03:58.180894 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 10:03:58.284192 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:03:58.284155 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" limit=1024 Jul 23 10:04:01.095813 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:01.095784 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-l6fgq_33072424-e286-4580-a3b5-0ccf6b4cbcf8/global-pull-secret-syncer/0.log" Jul 23 10:04:01.150691 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:01.150662 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-czm9w_c4122a2d-66ca-4ec5-969b-14b89809eaa0/konnectivity-agent/0.log" Jul 23 10:04:01.305217 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:01.305189 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-145.ec2.internal_6e4c6298602a592029f63da426ffeb44/haproxy/0.log" Jul 23 10:04:06.275097 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:06.275062 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595h986_51e149f3-3d54-46ad-86c9-01c83eb781f7/extract/0.log" Jul 23 10:04:06.304216 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:06.304181 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595h986_51e149f3-3d54-46ad-86c9-01c83eb781f7/util/0.log" Jul 23 10:04:06.328284 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:06.328235 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595h986_51e149f3-3d54-46ad-86c9-01c83eb781f7/pull/0.log" Jul 23 10:04:06.375846 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:06.375812 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wrs5q_f4dae8b7-1352-4f26-93c6-c5ffd5f8369f/extract/0.log" Jul 23 10:04:06.397331 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:06.397297 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wrs5q_f4dae8b7-1352-4f26-93c6-c5ffd5f8369f/util/0.log" Jul 23 10:04:06.419669 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:06.419642 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wrs5q_f4dae8b7-1352-4f26-93c6-c5ffd5f8369f/pull/0.log" Jul 23 10:04:06.448664 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:06.448636 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qrfl7_ce7494cc-693b-4499-bb8a-6646ec33f004/extract/0.log" Jul 23 10:04:06.469959 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:06.469923 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qrfl7_ce7494cc-693b-4499-bb8a-6646ec33f004/util/0.log" Jul 23 10:04:06.491318 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:06.491287 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qrfl7_ce7494cc-693b-4499-bb8a-6646ec33f004/pull/0.log" Jul 23 10:04:06.517583 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:06.517547 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cz5gk_c13198e7-f244-465c-82ff-199bde275f68/extract/0.log" Jul 23 10:04:06.539301 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:06.539203 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cz5gk_c13198e7-f244-465c-82ff-199bde275f68/util/0.log" Jul 23 10:04:06.561820 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:06.561791 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cz5gk_c13198e7-f244-465c-82ff-199bde275f68/pull/0.log" Jul 23 10:04:07.186703 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:07.186668 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5876bc94d4-gs5hp_c662fa1f-8cb7-434e-990a-31cdea707782/authorino/0.log" Jul 23 10:04:07.569847 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:07.569757 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-q5mq5_07acbf04-6129-4a76-ac35-87e8e8f6c19c/limitador/0.log" Jul 23 10:04:09.400457 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:09.400415 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_89311462-b7ab-4bc4-bd39-49da1d67d18d/alertmanager/0.log" Jul 23 10:04:09.425164 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:09.425136 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_89311462-b7ab-4bc4-bd39-49da1d67d18d/config-reloader/0.log" Jul 23 10:04:09.445926 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:09.445899 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_89311462-b7ab-4bc4-bd39-49da1d67d18d/kube-rbac-proxy-web/0.log" Jul 23 10:04:09.468295 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:09.468264 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_89311462-b7ab-4bc4-bd39-49da1d67d18d/kube-rbac-proxy/0.log" Jul 23 10:04:09.491991 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:09.491965 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_89311462-b7ab-4bc4-bd39-49da1d67d18d/kube-rbac-proxy-metric/0.log" Jul 23 10:04:09.512793 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:09.512772 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_89311462-b7ab-4bc4-bd39-49da1d67d18d/prom-label-proxy/0.log" Jul 23 10:04:09.536865 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:09.536842 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_89311462-b7ab-4bc4-bd39-49da1d67d18d/init-config-reloader/0.log" Jul 23 10:04:09.590277 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:09.590224 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-8zbdn_59ee07ce-4747-496f-9876-81f2c2358981/cluster-monitoring-operator/0.log" Jul 23 10:04:09.691915 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:09.691830 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-84c8fd6f77-4txkh_74395092-8e7f-4941-8d00-ac34b51838fc/metrics-server/0.log" Jul 23 10:04:09.917908 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:09.917860 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jvqzl_a40defcf-809b-41d8-bae4-72fb87268217/node-exporter/0.log" Jul 23 10:04:09.940208 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:09.940179 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jvqzl_a40defcf-809b-41d8-bae4-72fb87268217/kube-rbac-proxy/0.log" Jul 23 10:04:09.965898 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:09.965828 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jvqzl_a40defcf-809b-41d8-bae4-72fb87268217/init-textfile/0.log" Jul 23 10:04:10.309937 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:10.309853 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-9998cfcd5-b7c92_a05745d8-9561-497d-8e07-d0b9c1ee6ba3/telemeter-client/0.log" Jul 23 10:04:10.331099 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:10.331068 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-9998cfcd5-b7c92_a05745d8-9561-497d-8e07-d0b9c1ee6ba3/reload/0.log" Jul 23 10:04:10.353179 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:10.353152 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-9998cfcd5-b7c92_a05745d8-9561-497d-8e07-d0b9c1ee6ba3/kube-rbac-proxy/0.log" Jul 23 10:04:11.388910 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:11.388861 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qcmqp/perf-node-gather-daemonset-k9nzh"] Jul 23 10:04:11.389485 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:11.389465 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63616cb6-727c-4bed-bd50-018a0e615e2e" containerName="revoke-keys" Jul 23 10:04:11.389562 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:11.389489 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="63616cb6-727c-4bed-bd50-018a0e615e2e" containerName="revoke-keys" Jul 23 10:04:11.389618 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:11.389603 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="63616cb6-727c-4bed-bd50-018a0e615e2e" containerName="revoke-keys" Jul 23 10:04:11.392786 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:11.392763 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qcmqp/perf-node-gather-daemonset-k9nzh" Jul 23 10:04:11.394954 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:11.394924 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qcmqp\"/\"openshift-service-ca.crt\"" Jul 23 10:04:11.395704 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:11.395684 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-qcmqp\"/\"default-dockercfg-7c6dg\"" Jul 23 10:04:11.395803 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:11.395724 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qcmqp\"/\"kube-root-ca.crt\"" Jul 23 10:04:11.402313 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:11.402288 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qcmqp/perf-node-gather-daemonset-k9nzh"] Jul 23 10:04:11.490201 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:11.490159 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9b9bbb10-5a3a-4e5a-875a-7b301c234ca3-lib-modules\") pod \"perf-node-gather-daemonset-k9nzh\" (UID: \"9b9bbb10-5a3a-4e5a-875a-7b301c234ca3\") " pod="openshift-must-gather-qcmqp/perf-node-gather-daemonset-k9nzh" Jul 23 10:04:11.490449 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:11.490222 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9b9bbb10-5a3a-4e5a-875a-7b301c234ca3-proc\") pod \"perf-node-gather-daemonset-k9nzh\" (UID: \"9b9bbb10-5a3a-4e5a-875a-7b301c234ca3\") " pod="openshift-must-gather-qcmqp/perf-node-gather-daemonset-k9nzh" Jul 23 10:04:11.490449 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:11.490353 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9b9bbb10-5a3a-4e5a-875a-7b301c234ca3-sys\") pod \"perf-node-gather-daemonset-k9nzh\" (UID: \"9b9bbb10-5a3a-4e5a-875a-7b301c234ca3\") " pod="openshift-must-gather-qcmqp/perf-node-gather-daemonset-k9nzh" Jul 23 10:04:11.490449 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:11.490413 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc474\" (UniqueName: \"kubernetes.io/projected/9b9bbb10-5a3a-4e5a-875a-7b301c234ca3-kube-api-access-rc474\") pod \"perf-node-gather-daemonset-k9nzh\" (UID: \"9b9bbb10-5a3a-4e5a-875a-7b301c234ca3\") " pod="openshift-must-gather-qcmqp/perf-node-gather-daemonset-k9nzh" Jul 23 10:04:11.490573 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:11.490472 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9b9bbb10-5a3a-4e5a-875a-7b301c234ca3-podres\") pod \"perf-node-gather-daemonset-k9nzh\" (UID: \"9b9bbb10-5a3a-4e5a-875a-7b301c234ca3\") " pod="openshift-must-gather-qcmqp/perf-node-gather-daemonset-k9nzh" Jul 23 10:04:11.591439 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:11.591401 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9b9bbb10-5a3a-4e5a-875a-7b301c234ca3-lib-modules\") pod \"perf-node-gather-daemonset-k9nzh\" (UID: \"9b9bbb10-5a3a-4e5a-875a-7b301c234ca3\") " pod="openshift-must-gather-qcmqp/perf-node-gather-daemonset-k9nzh" Jul 23 10:04:11.591662 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:11.591452 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9b9bbb10-5a3a-4e5a-875a-7b301c234ca3-proc\") pod \"perf-node-gather-daemonset-k9nzh\" (UID: \"9b9bbb10-5a3a-4e5a-875a-7b301c234ca3\") " pod="openshift-must-gather-qcmqp/perf-node-gather-daemonset-k9nzh" Jul 23 10:04:11.591662 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:11.591490 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9b9bbb10-5a3a-4e5a-875a-7b301c234ca3-sys\") pod \"perf-node-gather-daemonset-k9nzh\" (UID: \"9b9bbb10-5a3a-4e5a-875a-7b301c234ca3\") " pod="openshift-must-gather-qcmqp/perf-node-gather-daemonset-k9nzh" Jul 23 10:04:11.591662 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:11.591518 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc474\" (UniqueName: \"kubernetes.io/projected/9b9bbb10-5a3a-4e5a-875a-7b301c234ca3-kube-api-access-rc474\") pod \"perf-node-gather-daemonset-k9nzh\" (UID: \"9b9bbb10-5a3a-4e5a-875a-7b301c234ca3\") " pod="openshift-must-gather-qcmqp/perf-node-gather-daemonset-k9nzh" Jul 23 10:04:11.591662 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:11.591554 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9b9bbb10-5a3a-4e5a-875a-7b301c234ca3-podres\") pod \"perf-node-gather-daemonset-k9nzh\" (UID: \"9b9bbb10-5a3a-4e5a-875a-7b301c234ca3\") " pod="openshift-must-gather-qcmqp/perf-node-gather-daemonset-k9nzh" Jul 23 10:04:11.591662 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:11.591601 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9b9bbb10-5a3a-4e5a-875a-7b301c234ca3-sys\") pod \"perf-node-gather-daemonset-k9nzh\" (UID: \"9b9bbb10-5a3a-4e5a-875a-7b301c234ca3\") " pod="openshift-must-gather-qcmqp/perf-node-gather-daemonset-k9nzh" Jul 23 10:04:11.591662 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:11.591608 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9b9bbb10-5a3a-4e5a-875a-7b301c234ca3-lib-modules\") pod \"perf-node-gather-daemonset-k9nzh\" (UID: \"9b9bbb10-5a3a-4e5a-875a-7b301c234ca3\") " pod="openshift-must-gather-qcmqp/perf-node-gather-daemonset-k9nzh" Jul 23 10:04:11.591662 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:11.591601 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9b9bbb10-5a3a-4e5a-875a-7b301c234ca3-proc\") pod \"perf-node-gather-daemonset-k9nzh\" (UID: \"9b9bbb10-5a3a-4e5a-875a-7b301c234ca3\") " pod="openshift-must-gather-qcmqp/perf-node-gather-daemonset-k9nzh" Jul 23 10:04:11.591907 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:11.591669 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9b9bbb10-5a3a-4e5a-875a-7b301c234ca3-podres\") pod \"perf-node-gather-daemonset-k9nzh\" (UID: \"9b9bbb10-5a3a-4e5a-875a-7b301c234ca3\") " pod="openshift-must-gather-qcmqp/perf-node-gather-daemonset-k9nzh" Jul 23 10:04:11.600044 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:11.600018 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc474\" (UniqueName: \"kubernetes.io/projected/9b9bbb10-5a3a-4e5a-875a-7b301c234ca3-kube-api-access-rc474\") pod \"perf-node-gather-daemonset-k9nzh\" (UID: \"9b9bbb10-5a3a-4e5a-875a-7b301c234ca3\") " pod="openshift-must-gather-qcmqp/perf-node-gather-daemonset-k9nzh" Jul 23 10:04:11.703656 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:11.703542 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qcmqp/perf-node-gather-daemonset-k9nzh" Jul 23 10:04:11.840964 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:11.840935 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qcmqp/perf-node-gather-daemonset-k9nzh"] Jul 23 10:04:11.842916 ip-10-0-135-145 kubenswrapper[2576]: W0723 10:04:11.842888 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9b9bbb10_5a3a_4e5a_875a_7b301c234ca3.slice/crio-0aafc34b420c8d4e3548b13a1ea25a5cefc13437a4b80aaaed32fad56bb64087 WatchSource:0}: Error finding container 0aafc34b420c8d4e3548b13a1ea25a5cefc13437a4b80aaaed32fad56bb64087: Status 404 returned error can't find the container with id 0aafc34b420c8d4e3548b13a1ea25a5cefc13437a4b80aaaed32fad56bb64087 Jul 23 10:04:11.997135 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:11.997031 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qcmqp/perf-node-gather-daemonset-k9nzh" event={"ID":"9b9bbb10-5a3a-4e5a-875a-7b301c234ca3","Type":"ContainerStarted","Data":"9e93a3bde7ad8b08ca12fd8515a8d57598b99e76a89e78b74680c802b8150582"} Jul 23 10:04:11.997135 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:11.997092 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-qcmqp/perf-node-gather-daemonset-k9nzh" Jul 23 10:04:11.997135 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:11.997114 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qcmqp/perf-node-gather-daemonset-k9nzh" event={"ID":"9b9bbb10-5a3a-4e5a-875a-7b301c234ca3","Type":"ContainerStarted","Data":"0aafc34b420c8d4e3548b13a1ea25a5cefc13437a4b80aaaed32fad56bb64087"} Jul 23 10:04:12.015856 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:12.015799 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-qcmqp/perf-node-gather-daemonset-k9nzh" podStartSLOduration=1.015780632 podStartE2EDuration="1.015780632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 10:04:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 10:04:12.014852654 +0000 UTC m=+2840.554131243" watchObservedRunningTime="2026-07-23 10:04:12.015780632 +0000 UTC m=+2840.555059208" Jul 23 10:04:12.193149 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:12.193121 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-5szxg_4908aff3-196e-44fa-b8e6-c0e81b476433/console-operator/2.log" Jul 23 10:04:12.199965 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:12.199940 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-5szxg_4908aff3-196e-44fa-b8e6-c0e81b476433/console-operator/3.log" Jul 23 10:04:12.669720 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:12.669687 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6f77c8c54b-vc7db_010072eb-4045-446e-8bbe-51f5954bc4a8/console/0.log" Jul 23 10:04:12.707821 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:12.707787 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-5bc67dcd46-28qcl_5d5c84db-7503-4aec-8ef8-81ef0c6b6a9b/download-server/0.log" Jul 23 10:04:13.279368 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:13.279338 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78cf4c6f6-k296x_e11b7a19-c8e6-4d52-8450-b86a0ca3fcd4/volume-data-source-validator/0.log" Jul 23 10:04:14.375764 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:14.375732 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mgk4m_31bba45e-487f-4c59-ac81-d8e3b90a549d/dns/0.log" Jul 23 10:04:14.395175 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:14.395143 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mgk4m_31bba45e-487f-4c59-ac81-d8e3b90a549d/kube-rbac-proxy/0.log" Jul 23 10:04:14.441971 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:14.441939 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gf7qk_72bcfd4f-fce8-46b6-b3f1-52e2ba6306fa/dns-node-resolver/0.log" Jul 23 10:04:15.041997 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:15.041968 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9r4c5_d676a11c-2214-4008-8801-efeb43be1ded/node-ca/0.log" Jul 23 10:04:16.171567 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:16.171536 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5dccdf9c4-2fgq7_dc7f1999-2e2f-41be-ae43-298dbd611e14/kube-auth-proxy/0.log" Jul 23 10:04:16.360289 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:16.360253 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7494768586-7px6v_6bddaeac-9272-4031-b216-a31fbe58c89d/router/0.log" Jul 23 10:04:17.023961 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:17.023929 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-cg599_022429e9-cac7-4ace-ae03-c39b633d6196/serve-healthcheck-canary/0.log" Jul 23 10:04:17.589963 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:17.589935 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-pxlst_989625f0-3f97-4f66-8d67-2e8f47a13a6c/insights-operator/0.log" Jul 23 10:04:17.591511 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:17.591485 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-pxlst_989625f0-3f97-4f66-8d67-2e8f47a13a6c/insights-operator/1.log" Jul 23 10:04:17.759740 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:17.759710 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kwb49_81ee471f-b215-43bd-a279-9ee79d67cd2b/kube-rbac-proxy/0.log" Jul 23 10:04:17.780209 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:17.780182 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kwb49_81ee471f-b215-43bd-a279-9ee79d67cd2b/exporter/0.log" Jul 23 10:04:17.802373 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:17.802345 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kwb49_81ee471f-b215-43bd-a279-9ee79d67cd2b/extractor/0.log" Jul 23 10:04:18.015038 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:18.015007 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-qcmqp/perf-node-gather-daemonset-k9nzh" Jul 23 10:04:20.723205 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:20.723176 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-64d6699bc9-xvzrs_01d0eb90-24fa-47a5-8019-bf35aaba7e1d/manager/0.log" Jul 23 10:04:20.744308 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:20.744286 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-64d6699bc9-xvzrs_01d0eb90-24fa-47a5-8019-bf35aaba7e1d/copy-manifests/0.log" Jul 23 10:04:22.046780 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:22.046749 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5dccdf9c4-2fgq7_dc7f1999-2e2f-41be-ae43-298dbd611e14/kube-auth-proxy/0.log" Jul 23 10:04:22.061632 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:22.061606 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6cb78dd96b-577wn_402d09a2-ca01-4485-9401-7dcd9e584fc3/manager/0.log" Jul 23 10:04:22.179630 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:22.179599 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7494768586-7px6v_6bddaeac-9272-4031-b216-a31fbe58c89d/router/0.log" Jul 23 10:04:23.602034 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:23.602004 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5dccdf9c4-2fgq7_dc7f1999-2e2f-41be-ae43-298dbd611e14/kube-auth-proxy/0.log" Jul 23 10:04:23.755316 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:23.755282 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7494768586-7px6v_6bddaeac-9272-4031-b216-a31fbe58c89d/router/0.log" Jul 23 10:04:27.643255 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:27.643209 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-nfv7r_ef3505b4-c02c-487a-8314-f6793087ad3a/kube-storage-version-migrator-operator/1.log" Jul 23 10:04:27.644975 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:27.644948 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-nfv7r_ef3505b4-c02c-487a-8314-f6793087ad3a/kube-storage-version-migrator-operator/0.log" Jul 23 10:04:28.799342 ip-10-0-135-145 kubenswrapper[2576]: I0723 10:04:28.799309 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-4gw2w_5fdaa22f-7fb2-42dd-8c10-2e422fa19df2/kube-multus/0.log"